Distributed model training method and device, program product and electronic equipment

Abstract

The embodiment of the invention relates to a distributed model training method and device, a computer program product and electronic equipment, and relates to the technical field of computers.The method comprises the steps that a machine learning model is divided into a plurality of stages based on a stage partition network, and each stage is divided into a plurality of copies; and scheduling a plurality of copies of each stage to a multi-dimensional network through a stage scheduling network so as to perform distributed training on the machine learning model. The network utilization rate can be improved, and the training effect is improved.

Description

Background Art

[0002] Training large models requires massive computing and storage resources, and cluster construction of next-generation training platforms requires expensive computing facilities.

[0003] Distributed training has become the mainstream choice in the era of large models. The existing infrastructure is constantly evolving to form a new heterogeneous IoT situation with mixed deployment of IoT and intelligent computing centers. DNN models can be divided into multiple sub-models, some of which are executed on edge devices and the other in intelligent computing centers. When using existing scheduling technologies for collective communication, it is impossible to keep all network dimensions busy in a mixed environment and to maximize the use of network bandwidth. Summary of the invention

[0004] The purpose of the present disclosure is to provide a distributed model training method, a distributed model training device, a computer program product and an electronic device, thereby at least to a certain extent overcoming the problem of being unable to maximize network bandwidth during distributed model training due to limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, a distributed model training method is provided, including: dividing a machine learning model into multiple stages based on a stage partitioning network, and dividing each stage into multiple replicas; scheduling multiple replicas of each stage to a multidimensional network through a stage scheduling network to perform distributed training on the machine learning model.

[0006] In an exemplary embodiment of the present disclosure, the machine learning model is a deep neural network model; the stage-based partitioning network divides the machine learning model into multiple stages, including: obtaining attribute features of each layer of the deep neural network model, and converting the attribute features into high-dimensional features of each layer;

[0007] The high-dimensional features are input into a stage partitioning network, and based on the optimal network parameters of the stage partitioning network, the deep neural network model is divided into multiple stages; the number of the stages is less than the number of IoT devices.

[0008] In an exemplary embodiment of the present disclosure, the deep neural network model is divided into multiple stages based on the optimal network parameters of the stage partition network, including: determining the divisible stages according to the properties of the deep neural network model, and determining the maximum load-bearing stage according to the stage partition network properties corresponding to the optimal network parameters of the stage partition network; based on the comparison results of the divisible stages and the maximum load-bearing stage, the deep neural network model is divided into multiple stages according to the divisible stages or the maximum load-bearing stage.

[0009] In an exemplary embodiment of the present disclosure, the method further includes: inputting high-dimensional features of each stage into a stage scheduling network, determining a training time according to a forward propagation time and a backward propagation time between adjacent stages; determining a reward function according to the training time; and determining optimal network parameters of the stage partitioning network and the stage scheduling network based on the reward function.

[0010] In an exemplary embodiment of the present disclosure, the multiple copies of each stage are scheduled to the multidimensional network through the stage scheduling network, including: decomposing the first communication operation into a second communication operation stage and a third communication operation stage; scheduling the training blocks of the copies of the multiple stages to the target dimension in the multidimensional network through the second communication operation and the third communication operation; wherein the second communication operation is completed before the third communication operation is started.

[0011] In an exemplary embodiment of the present disclosure, scheduling the training blocks of replicas of multiple stages to the target dimension in the multidimensional network includes: performing dynamic block scheduling through a greedy algorithm to schedule the training block of each replica to the target dimension in the multidimensional network.

[0012] In an exemplary embodiment of the present disclosure, the dynamic block scheduling is performed through a greedy algorithm to schedule the training block of each replica to a target dimension in a multidimensional network, including: allocating the training block to a reference dimension; when the training block is in the reference dimension, if the occupancy of multiple dimensions is full, determining the reference dimension as the target dimension.

[0013] According to one aspect of the present disclosure, a distributed model training device is provided, including: a model partitioning module, used to divide a machine learning model into multiple stages based on a stage partitioning network, and divide each stage into multiple copies; a scheduling module, used to schedule multiple copies of each stage to a multidimensional network through a stage scheduling network, so as to perform distributed training on the machine learning model.

[0014] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the distributed model training method described in any one of the above is implemented.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-described distributed model training methods by executing the executable instructions.

[0016] In the technical solution provided in the embodiments of the present disclosure, on the one hand, the machine learning model is divided into multiple stages through a stage partitioning network, and multiple copies of each stage are scheduled to a multi-dimensional network through a stage scheduling network, so as to achieve scheduling of each stage to a computing device, which can enable the multi-dimensional network to maintain a high bandwidth utilization rate in all dimensions, maximize the use of network bandwidth, and improve the overall resource utilization rate of heterogeneous networks. On the other hand, by dividing the machine learning model into multiple stages and scheduling the copies of each stage to the appropriate dimensions in the multi-dimensional network, the model training time can be shortened, thereby improving the model training efficiency.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A flowchart of a distributed model training method in an embodiment of the present disclosure is schematically shown.

[0020] Figure 2 A schematic diagram schematically illustrates the division of model stages in an embodiment of the present disclosure.

[0021] Figure 3 The flowchart of network scheduling is schematically shown.

[0022] Figure 4 The specific flow chart of the distributed training scheduling system based on the dual network architecture is schematically shown.

[0023] Figure 5 A block diagram of a distributed model training device in an embodiment of the present disclosure is schematically shown.

[0024] Figure 6 The block diagram of the electronic device according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] In some embodiments, large model parameters have entered the tens of billions or hundreds of billions era. Large model training requires extremely large computing and storage resources. The cluster construction of the next-generation training platform requires expensive computing facilities. In the process of building a new generation of clusters, effective connection and resource utilization with existing cluster infrastructure can greatly reduce R&D costs.

[0028] Distributed training has become the mainstream choice in the era of large models. The existing infrastructure is constantly developing and forming a new situation of heterogeneous IoT with mixed deployment of IoT and intelligent computing centers. DNN models can be divided into multiple sub-models, some of which are executed on edge devices and the other part is executed in the intelligent computing center. The current types of distributed training include data parallelism, model parallelism and pipeline parallelism. Model parallelism in a heterogeneous environment can only ensure that a certain access node has only certain stages of the model, and cannot guarantee the overall busy state of the interconnected cluster.

[0029] At the same time, in the next generation of large-scale training platforms, even within a certain intelligent computing center, NPUs will be connected through multi-dimensional networks with multiple heterogeneous bandwidths. When using existing scheduling technologies for collective communication, it is impossible to keep all network dimensions busy and maximize network bandwidth in a mixed environment.

[0030] In order to solve the above technical problems, a distributed model training method is provided in the embodiment of the present disclosure, which is used to perform distributed training on the model. It can be applied to the application scenario of distributed model training of any type of machine learning model in a wide area network. In the embodiment of the present disclosure, a distributed training collective scheduling system based on a dual network architecture is proposed, which aims to minimize the training time of model iteration and maximize the use of multi-dimensional network bandwidth. Figure 1 As shown in , the distributed model training method mainly includes the following steps:

[0031] In step S110, the machine learning model is divided into multiple stages based on the stage partition network, and each stage is divided into multiple copies;

[0032] In step S120, multiple copies of each stage are scheduled to the multi-dimensional network through the stage scheduling network to train the machine learning model.

[0033] The technical solution provided in the embodiments of the present disclosure, on the one hand, divides the machine learning model into multiple stages through a stage partitioning network, and schedules multiple copies of each stage to a multi-dimensional network through a stage scheduling network, thereby scheduling each stage to a computing device, which enables the multi-dimensional network to maintain a high bandwidth utilization rate in all dimensions, maximizes the use of network bandwidth, and improves the overall resource utilization rate of heterogeneous networks. On the other hand, by dividing the machine learning model into multiple stages and scheduling the copies of each stage to the appropriate dimensions in the multi-dimensional network, the model training time can be shortened, thereby improving the model training efficiency.

[0034] Next, refer to Figure 1 The distributed model training method in the embodiment of the present disclosure is specifically described as shown in FIG.

[0035] In step S110, the machine learning model is divided into multiple stages based on the stage partition network, and each stage is divided into multiple copies.

[0036] In the disclosed embodiments, the machine learning model may be a deep neural network model or any type of model. Here, the machine learning model is described as a deep neural network model. The deep neural network model may be a DNN (Deep Neural Network), which refers to an artificial neural network composed of multiple layers, usually including an input layer, multiple hidden layers, and an output layer. The deep neural network model DNN can automatically learn complex data representations.

[0037] During the process of training a deep neural network model, in order to minimize the training time of model iteration and maximize the utilization of heterogeneous multi-dimensional network bandwidth, a model partitioning and scheduling scheme based on policy gradient can be provided. Exemplarily, first, the deep neural network model can be divided into multiple stages. In some embodiments, based on the existing heterogeneous and dynamic Internet of Things infrastructure environment, DNN training data samples generated by the existing environment can be collected, and the optimal network parameters of the stage partitioning network and the optimal network parameters of the stage scheduling network can be learned based on the DNN training data samples.

[0038] The stage partitioning network and the stage scheduling network can form a dual-network architecture. The stage partitioning network can be a fully connected network, mainly responsible for model partitioning; the stage scheduling network can be a long short-term memory network (LSTM), mainly used to be responsible for the scheduling of the training blocks corresponding to the stages in the model.

[0039] After determining the optimal network parameters of the stage partitioning network, the deep neural network model (DNN) can be divided into multiple stages based on the optimal network parameters of the stage partitioning network. The multiple stages can be represented by {s1, s2, s3,... s s}, where s < the number of IoT devices.

[0040] When dividing the deep neural network model (DNN) into multiple stages, the division can be performed according to the deep neural network model attributes and the stage partitioning network attributes. Exemplarily, the deep neural network model attributes refer to the attributes that the DNN model itself has, including one or more of the operation type, input data volume, output data volume, and computational volume of each layer of the deep neural network model. The stage partitioning network attributes can be the attributes of the stage partitioning network. Since the stage partitioning network can be a fully connected network, the stage partitioning network attributes specifically refer to the situation of the fully connected network used to implement the partitioning of the machine learning model. The stage partitioning network attributes include one or more of the computational power, memory, and bandwidth of the stage partitioning network. It should be noted that the stage partitioning network attributes can be the attributes corresponding to the optimal network parameters of the stage partitioning network.

[0041] In some embodiments, the divisible stages of the deep neural network model can first be determined according to the deep neural network model attributes, and the maximum load-bearing stage can be further determined according to the stage partition network attributes. The maximum load-bearing stage can be the same as or different from the divisible stage. The divisible stages determined according to the deep neural network model attributes are screened based on the maximum load-bearing stage to determine the number of multiple stages into which the deep neural network model is divided. Exemplarily, if the comparison result is that the divisible stage determined according to the deep neural network model attributes is greater than the maximum load-bearing stage, the deep neural network model can be divided according to the number of the maximum load-bearing stage to divide the deep neural network model into multiple stages. If the comparison result is that the divisible stage determined according to the deep neural network model attributes is less than or equal to the maximum load-bearing stage, the deep neural network model can be divided according to the number of divisible stages to divide the deep neural network model into multiple stages.

[0042] After the deep neural network model is divided into multiple stages, in order to speed up the training process of the deep neural network model DNN, each stage s i Divide into multiple copies, the number of copies can be represented as i, and the number of copies divided in each stage can be the same or different. Based on this, the deep neural network model is divided into stages s i It can be expressed as {s i1 ,s i2 ,s i3 ,...s ii After each stage is divided into multiple replicas, each replica can be divided into multiple training blocks, thereby realizing the scheduling of training blocks. Among them, the granularity of training blocks is much smaller than that of replicas.

[0043] Figure 2 The schematic diagram shows the stage division process of the deep neural network model DNN, refer to Figure 2 As shown in , the deep neural network model DNN can be divided into four stages, which can be represented as s1, s2, s3, and s4. On this basis, each stage can be divided into multiple copies, such as reference Figure 2 As shown in , the replicas of stage s1 include s 10 ; The copy of stage s2 includes s 20 、s 21 、s 22 ; The copy of stage s3 includes s 30 ; The copy of stage s4 includes s 40 .

[0044] Next, in step S120, the replicas corresponding to the multiple stages are scheduled to the multi-dimensional network through the stage scheduling network to perform distributed training on the machine learning model.

[0045] In the disclosed embodiment, the stage scheduling network may be a long short-term memory network LSTM, which is mainly used to schedule the stages and replicas obtained by dividing the deep neural network model to a multi-dimensional network composed of IoT devices. The number of IoT devices may be multiple, and may be multiple heterogeneous devices in a wide area network. Multiple IoT devices may form a multi-dimensional network.

[0046] In some embodiments, the high-dimensional features of each stage can be input into the stage scheduling network to minimize the training time of each iteration, so as to realize the training block scheduling through the stage scheduling network. The communication time between each stage can be calculated with the help of the LSTM network, so as to facilitate the optimization of the network parameters of the dual network architecture in the subsequent process, so as to determine the optimal stage division method. The LSTM network includes two stages: forward propagation and back propagation. In the forward propagation stage, the LSTM network processes the sequence data step by time through the gate mechanism; in the back propagation stage, the LSTM network adjusts the weights and biases through the gradient descent method.

[0047] Among them, the high-dimensional features can be attribute features of each layer of the deep neural network model in each stage. The attribute features may include one or more of the operation type, input data volume, output data volume, and calculation amount. Exemplarily, one or more of the operation type, input data volume, output data volume, and calculation amount of each layer of the deep neural network model can be extracted, and these attribute features can be converted into high-dimensional features of each layer and input into the stage scheduling network of the dual network architecture to achieve model partitioning and model scheduling.

[0048] It should be noted that the training time of the deep neural network model can be the sum of the processing time within the deep neural network model and the communication time between the stages of the deep neural network model. The communication time between the stages of the deep neural network model is determined according to the forward propagation time and the backward propagation time between adjacent stages, and can be specifically the sum of the forward propagation time and the backward propagation time between adjacent stages. i and stages i+1 In terms of stage i To stage i+1 The forward propagation time can be expressed as formula (1), stage s i To stage i+1 The backward propagation time can be expressed as formula (2):

[0049]

[0050] in, is the computational effort of the forward propagation, B represents the available bandwidth between the two stages, is the amount of computation required for back propagation.

[0051] Furthermore, a convolutional feature encoding branch consisting of four independent layers can be constructed. After obtaining the training time of the deep neural network model, a reward function can be designed according to the iterative training time, and the network parameters of the stage partition network and the stage scheduling network in the dual network architecture can be updated based on the reward function. Exemplarily, after determining the reward function, the optimal dual network model can be learned with the help of a gradient learning algorithm, and the network parameters of the two modules, the stage partition network and the stage scheduling network, can be updated to obtain the optimal network parameters. The network parameters can be determined based on the network parameters at the previous moment, the learning rate, and the gradient of the reward function under the network parameters at the previous moment. Exemplarily, the network parameters can be determined based on the network parameters at the previous moment, and the sum of the product of the learning rate and the gradient of the reward function under the network parameters at the previous moment. The network parameters can be expressed as shown in formula (3):

[0052]

[0053] Among them, θ represents the network parameters, η represents the learning rate, and J(θ t ) refers to the network parameter θ t The reward function under Refers to the gradient of the reward function with respect to the network parameters.

[0054] After obtaining the optimal network parameters of the stage partitioning network and the stage scheduling network in the dual network architecture, the deep neural network model can be optimally divided into stages based on the stage partitioning network to obtain multiple stages. The multiple stages and corresponding copies can then be scheduled to the target dimension in the multi-dimensional network composed of different IoT devices through the stage scheduling network.

[0055] In the embodiments of the present disclosure, when performing policy scheduling, a multi-dimensional network bandwidth perception module, an All-Reduce decomposition scheduling module, and a policy scheduling module may be included. The new generation of intelligent computing centers includes multiple IoT devices, and multiple IoT devices include multiple network dimensions. The NPU is constantly expanding, and each network dimension shows its own uniqueness. In some embodiments, by adding a multi-dimensional network bandwidth perception module, the link with the smallest bandwidth can be prevented from becoming a scheduling bottleneck. Among them, the multi-dimensional network bandwidth perception module can be used to perceive the bandwidth of the network and other situations.

[0056] Further, the first communication operation can be decomposed into a second communication operation and a third communication operation based on the All-Reduce decomposition scheduling module. Among them, the first communication operation can be an All-Reduce operation. The All-Reduce algorithm is a special communication protocol that performs reduction operations (such as summing, taking the maximum value, etc.) on the data on all nodes and returns the result to the root node. In this way, the All-Reduce algorithm realizes data synchronization and reduction between nodes, thereby accelerating the convergence speed of the model. The All-Reduce algorithm usually uses a tree structure for data transmission and reduction operations to reduce communication overhead and improve efficiency. In distributed training, each node sends its own parameter data to the parent node of the previous level, which is then summarized by the parent node and sent to the parent node of a higher level until the root node is reached. After the root node performs a reduction operation on all parameter data, it broadcasts the result to all nodes for updating. The All-Reduce operation adds the data on each node or performs other types of reduction operations (such as summing, averaging, and maximum value) between all nodes. In the end, each node will get the same result, which is a summary of the data of all nodes.

[0057] The second communication operation may be Reduce-Scatter. Reduce-Scatter is a communication operation used in distributed computing, which is mainly used to aggregate data in multiple processes through an All-Reduce operation, and then distribute the aggregated results to each process through a Scatter scattering operation.

[0058] The third communication operation can be All-Gather. All-Gather is used in distributed computing. Its basic function is to collect all the data in a group of processes and distribute the collected data to all processes. Specifically, the All-Gather operation involves two main steps: first, all processes send their local data to a central node or all other processes; then, this central node or each process distributes the collected data back to all other processes. In this way, each process will eventually receive a copy of the data of all other processes.

[0059] In the disclosed embodiment, the collective communication algorithm uses layering to decompose the first communication operation All-Reduce into the second communication operation Reduce-Scatter and the third communication operation All-Gather. In the actual process of training the deep neural network model, the second communication operation Reduce-Scatter and the third communication operation All-Gather do not have a strict execution order. It is only required to complete the second communication operation Reduce-Scatter stage before starting the third communication operation All-Gather. After the first communication operation is decomposed into the second communication operation and the third communication operation, in each stage obtained by dividing the deep neural network model, it can be dynamically scheduled in a block scheduling manner based on the second communication operation and the third communication operation to maximize the use of each dimension of the network.

[0060] In some embodiments, when scheduling the training blocks of each stage, the training block scheduling can be implemented by a greedy algorithm. The greedy algorithm means that when solving a problem, the best choice is made at the moment.

[0061] For example, a homogeneous or heterogeneous data center contains multiple IoT devices composed of computers. IoT devices can be computers, for example. The GPUs in the same computer may use nvlink links, which belong to one bandwidth type; different computers may use ib links or nvswitch links, and the bandwidth belongs to another bandwidth type. Since multiple IoT devices include different bandwidth types, they can be considered to include multiple networks of different dimensions, where the dimensions are set according to the network conditions.

[0062] Since multiple IoT devices may include multiple dimensions, the scheduling order between each dimension may be determined according to actual needs. For example, the first dimension may be scheduled first, and then the second dimension, or the second dimension may be scheduled first, and then the first dimension. Exemplarily, dynamic scheduling may be achieved according to the dimensional occupancy of the multiple-dimensional network by the training blocks corresponding to the multiple stages of the deep neural network model. In some embodiments, the target dimension for placement of each training block may be calculated by a greedy algorithm, wherein the occupancy of multiple dimensions may be full when the training block is placed in the target dimension. Based on this, the training block may be assigned to a reference dimension; when the training block is in the reference dimension, if the occupancy of multiple dimensions is full, the reference dimension is determined as the target dimension. The reference dimension may be any dimension in the multidimensional network. If the occupancy of multiple dimensions is full, the utilization rate is the highest. The fact that all dimensions are full means that all bandwidths are full. If the training block is in the reference dimension, if the occupancy of multiple dimensions is not full, the reference dimension does not need to be determined as the target dimension, but the target dimension needs to be reassigned to the training block. When scheduling, if both the high-dimensional and low-dimensional dimensions are idle, the dimension with high bandwidth can be scheduled first. If both the high-dimensional and low-dimensional dimensions are not idle, there is no need to schedule the dimension with high bandwidth first.

[0063] In some embodiments, a 2-dimensional network architecture composed of multiple IoT devices is used as an example for explanation. Block scheduling of training blocks can be achieved through a greedy algorithm. RS scheduling in the second dimension can also precede RS scheduling in the first dimension. Specifically, assuming a two-dimensional network architecture, the first dimension (dim1) is interconnected by Nvlink high-speed interconnection; the second dimension (dim2) is expanded outward through Ethernet or InfiniBand network card, and the network bandwidth of the first dimension is twice the network bandwidth of the second dimension. Based on the above implementation, a dynamic algorithm can be used to calculate which dimension a certain training block is placed in to fill multiple dimensions, and then the dimension can be determined as the target dimension of the training block. For example, if training block 1 is placed in the first dimension to fill multiple dimensions, training block 1 is scheduled to the first dimension; if training block 2 is placed in the second dimension to fill multiple dimensions, training block 2 can be scheduled to the second dimension. If training block 3 is placed in the second dimension to fill multiple dimensions, training block 3 can be scheduled to the second dimension.

[0064] In the disclosed embodiment, the training blocks in each stage can be dynamically scheduled according to the relationship between the dimension where the training blocks are located and the occupancy state of the multi-dimensional network, so that the reasonable and accurate scheduling of the training blocks can be achieved. By scheduling the training blocks to the appropriate dimensions, the training time of the model iteration can be minimized and the multi-dimensional network can be utilized to the greatest extent.

[0065] Figure 3 A flowchart of network scheduling is schematically shown in FIG. Figure 3 As shown in , it mainly includes a model input module, a parameter initialization module, a stage partition network module, a stage scheduling network module, a multi-dimensional network bandwidth perception module, an All-Reduce decomposition scheduling module, and a scheduling strategy module. Among them, the stage partition network module and the stage scheduling network module can form a dual network architecture, which is used to divide the machine learning model into stages, and then divide each stage into multiple copies to obtain a training block corresponding to each copy. The multi-dimensional network bandwidth perception module is used to perceive the bandwidth of the network. The All-Reduce decomposition scheduling module is used to decompose the first communication operation into the second communication operation and the third communication operation. The scheduling strategy module is used to determine which dimension each training block is scheduled to. It should be noted that the stage partition network module and the stage scheduling network module can be trained according to the policy gradient learning module. The policy gradient learning module may include a feature extraction module, a reward function module, and a parameter update module. The reward function module is used to define the reward function according to the training time. The training time can be the sum of the processing time in the deep neural network model and the communication time between the stages of the deep neural network model. The communication time between the stages of the deep neural network model can specifically be the sum of the forward propagation time and the backward propagation time. The parameter update module is used to update the network parameters of the stage partition module and the stage scheduling module according to the network parameters, learning rate and gradient of the reward function under the network parameters at the previous moment.

[0066] Figure 4 The specific flow chart of the distributed training scheduling system based on the dual network architecture is schematically shown. Figure 4 As shown in , it mainly includes the following steps:

[0067] In step S401, an Internet of Things environment is obtained, which may be a wide area network environment, which may include intelligent computing center 1, intelligent computing center 2 and a new intelligent computing center n.

[0068] In step S402, a deep neural network model is input.

[0069] In step S403, the MLP phase partition network is obtained.

[0070] In step S404, the deep neural network model is divided into multiple stages based on the MLP stage partition network, such as stage 1, stage 2, and stage 3, where each stage may include a different number of replicas.

[0071] In step S405, the MLP stage scheduling network is obtained.

[0072] In step S406, the intelligent computing center m includes device 1, device 2, and device 3, forming a multi-dimensional network.

[0073] In step S407, a network bandwidth-aware collective scheduling strategy is provided through the LSTM stage scheduling network to schedule the training blocks of each stage to the target dimension of the multidimensional network. Exemplarily, each training block can be assigned to a reference dimension; when the training block is in the reference dimension, if the occupancy of multiple dimensions is full, that is, multiple dimensions are full and multiple bandwidths are full, the reference dimension can be determined as the target dimension. The reference dimension can be any dimension in the multidimensional network.

[0074] In the disclosed embodiment, a dual network architecture is proposed to solve the problem of DNN model partitioning and scheduling in a heterogeneous Internet of Things environment. The four features of operation type, input data volume, output data volume and calculation amount of each layer of the DNN model are extracted, and these features are converted into high-dimensional features of each layer and fed back to the dual network to partition and schedule the model. Since a policy gradient algorithm is proposed to learn the optimal dual network model, a reward function is designed to adaptively partition and effectively schedule the DNN model, which can minimize the model training time and improve the model training efficiency. In addition, the training blocks can be dynamically and evenly scheduled on network bandwidths of different dimensions, making full use of the bandwidths of different dimensions within the intelligent computing center and improving the overall bandwidth utilization. The disclosed embodiment provides a distributed training collective scheduling system based on a dual network architecture. The system has designed a model partitioning and scheduling scheme based on policy gradient, which adaptively divides the DNN model training into multiple stages and effectively schedules each stage to a multi-dimensional network composed of computing devices. Based on this architecture, a network-aware collective scheduling scheme is constructed for independent computing clusters in local area networks in heterogeneous Internet of Things. The multi-dimensional network that can balance heterogeneous bandwidth can maintain high bandwidth utilization in all network dimensions, further improving the overall resource utilization of heterogeneous Internet of Things computing centers.

[0075] The present disclosure also provides a distributed model training device. Figure 5 As shown, the distributed model training device 500 mainly includes the following modules:

[0076] A model partitioning module 501 is used to divide the machine learning model into multiple stages based on the stage partitioning network, and divide each stage into multiple copies;

[0077] The scheduling module 502 is used to schedule multiple copies of each stage to the multi-dimensional network through the stage scheduling network to perform distributed training on the machine learning model.

[0078] In an exemplary embodiment of the present disclosure, the machine learning model is a deep neural network model; the stage-based partitioning network divides the machine learning model into multiple stages, including: obtaining attribute features of each layer of the deep neural network model, and converting the attribute features into high-dimensional features of each layer;

[0079] The high-dimensional features are input into a stage partitioning network, and based on the optimal network parameters of the stage partitioning network, the deep neural network model is divided into multiple stages; the number of the stages is less than the number of IoT devices.

[0080] In an exemplary embodiment of the present disclosure, the deep neural network model is divided into multiple stages based on the optimal network parameters of the stage partition network, including: determining the divisible stages according to the properties of the deep neural network model, and determining the maximum load-bearing stage according to the stage partition network properties corresponding to the optimal network parameters of the stage partition network; based on the comparison results of the divisible stages and the maximum load-bearing stage, the deep neural network model is divided into multiple stages according to the divisible stages or the maximum load-bearing stage.

[0081] In an exemplary embodiment of the present disclosure, the method further includes: inputting high-dimensional features of each stage into a stage scheduling network, determining a training time according to a forward propagation time and a backward propagation time between adjacent stages; determining a reward function according to the training time; and determining optimal network parameters of the stage partitioning network and the stage scheduling network based on the reward function.

[0082] In an exemplary embodiment of the present disclosure, the multiple copies of each stage are scheduled to the multidimensional network through the stage scheduling network, including: decomposing the first communication operation into a second communication operation stage and a third communication operation stage; scheduling the training blocks of the copies of the multiple stages to the target dimension in the multidimensional network through the second communication operation and the third communication operation; wherein the second communication operation is completed before the third communication operation is started.

[0083] In an exemplary embodiment of the present disclosure, scheduling the training blocks of replicas of multiple stages to the target dimension in the multidimensional network includes: performing dynamic block scheduling through a greedy algorithm to schedule the training block of each replica to the target dimension in the multidimensional network.

[0084] In an exemplary embodiment of the present disclosure, the dynamic block scheduling is performed through a greedy algorithm to schedule the training block of each replica to a target dimension in a multidimensional network, including: allocating the training block to a reference dimension; when the training block is in the reference dimension, if the occupancy of multiple dimensions is full, determining the reference dimension as the target dimension.

[0085] It should be noted that the specific details of each module in the above-mentioned distributed model training device have been described in detail in the corresponding distributed model training method, so they will not be repeated here.

[0086] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0087] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0088] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0089] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0090] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0092] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0093] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0094] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0095] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0096] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0097] It should be noted that, in some embodiments of the present disclosure, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method is implemented.

[0098] In one embodiment, a computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, a computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), and the like. In one embodiment, a computer program product may be an intangible product containing a computer program. Exemplarily, a computer program product may be implemented as a virtual digital product, such as a digital file such as an executable file, an installation package, etc. storing a computer program.

[0099] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0100] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure.

[0101] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0102] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0103] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0104] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0105] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A distributed model training method, characterized in that: include: Divide the machine learning model into multiple stages based on the stage partition network, and divide each stage into multiple copies; Through the stage scheduling network, multiple copies of each stage are scheduled to the multi-dimensional network to perform distributed training on the machine learning model.

2. The distributed model training method according to claim 1, characterized in that: The machine learning model is a deep neural network model; the stage-based partitioning network divides the machine learning model into multiple stages, including: Obtaining attribute features of each layer of the deep neural network model, and converting the attribute features into high-dimensional features of each layer; The high-dimensional features are input into a stage partitioning network, and based on the optimal network parameters of the stage partitioning network, the deep neural network model is divided into multiple stages; the number of the stages is less than the number of IoT devices.

3. The distributed model training method according to claim 2, characterized in that: The method of dividing the deep neural network model into multiple stages based on the optimal network parameters of the stage partitioning network includes: Determine the divisible stages according to the properties of the deep neural network model, and determine the maximum number of stages that can be carried according to the properties of the stage partition network corresponding to the optimal network parameters of the stage partition network; Based on the comparison results of the divisible stages and the maximum load-bearing stages, the deep neural network model is divided into multiple stages according to the divisible stages or the maximum load-bearing stages.

4. The distributed model training method according to claim 1, characterized in that: The method further comprises: The high-dimensional features of each stage are input into the stage scheduling network, and the training time is determined according to the forward propagation time and backward propagation time between adjacent stages; determining a reward function based on the training time; Optimal network parameters of the stage partitioning network and the stage scheduling network are determined based on the reward function.

5. The distributed model training method according to claim 1, characterized in that: The method of scheduling multiple copies of each stage to the multi-dimensional network through the stage scheduling network includes: Decomposing the first communication operation into a second communication operation phase and a third communication operation phase; Scheduling the training blocks of the replicas of the multiple stages to the target dimension in the multi-dimensional network through the second communication operation and the third communication operation; Wherein, the second communication operation is completed before the third communication operation is started.

6. The distributed model training method according to claim 5, characterized in that: The step of scheduling the training blocks of the replicas of the multiple stages to the target dimension in the multi-dimensional network includes: Dynamic block scheduling is performed through a greedy algorithm to schedule the training blocks of each replica to the target dimension in the multi-dimensional network.

7. The distributed model training method according to claim 6, characterized in that: The dynamic block scheduling is performed by the greedy algorithm to schedule the training block of each replica to the target dimension in the multi-dimensional network, including: assigning the training blocks to reference dimensions; In the case where the training block is in the reference dimension, if the occupancy status of multiple dimensions is full, the reference dimension is determined as the target dimension.

8. A distributed model training device, characterized in that: include: A model partitioning module, which is used to divide the machine learning model into multiple stages based on the stage partitioning network, and divide each stage into multiple copies; The scheduling module is used to schedule multiple copies of each stage to a multi-dimensional network through a stage scheduling network to perform distributed training on the machine learning model.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the distributed model training method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the distributed model training method described in any one of claims 1-7 by executing the executable instructions.

Images