A neural network model partitioning method, system, device and medium for parallel training

Abstract

The present application discloses a neural network model partitioning method, system, device and medium for parallel training, which obtains the neural network model to be trained, training data and cluster configuration; determines the communication group according to the link type in the cluster configuration; and trains the neural network model to be trained based on the communication group and the training data until a trained neural network model is obtained. For the respective communication processes of sharded data parallelism and data parallelism, different parameters of the model can use the corresponding links within the computing node and between the nodes to communicate simultaneously, making use of the originally idle links, reducing the communication volume on the original links, thereby reducing the total communication overhead and improving the model training speed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a neural network model partitioning method, system, device and medium for parallel training. Background Art

[0002] In order to improve performance and enhance the generalization ability of models, the scale of large-scale deep learning models and the size of required training datasets continue to grow. This makes it difficult for the storage capacity of a single computing hardware to meet the current model training needs. To solve this problem, a variety of mature and effective data parallel and sharding parallel training strategies have been developed. Data parallel methods implement model training by distributing datasets across multiple computing nodes, while sharding parallelism simultaneously splits model parameters and datasets so that training tasks can be distributed across multiple devices, thereby reducing the storage burden of a single computing node.

[0003] However, in existing data parallel and sharding parallel methods, communication costs have become the main bottleneck restricting training efficiency. These parallel methods require frequent and large-scale global communication during the training process. However, the existing methods divide model parameters in a coarse-grained manner and can only use a single link between devices for communication at a time, leaving other links idle, resulting in huge communication overhead. Therefore, model training often becomes very time-consuming. Summary of the invention

[0004] Based on the above problems, the present application provides a neural network model partitioning method, system, device and medium for parallel training, which are used to reduce communication overhead and thus reduce the time consumption of model training.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] The first aspect of the present application provides a neural network model partitioning method for parallel training, comprising:

[0007] Acquire cluster configuration, wherein the cluster configuration includes information used to characterize link connection status of the cluster, link bandwidth status of the cluster, number of computing devices, and type of computing nodes;

[0008] Determine, according to the link type in the cluster configuration, a division method of each parameter and a computing device corresponding to each link;

[0009] A communication group is determined based on the division method of each parameter and the computing device corresponding to each link, and the communication group includes parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes.

[0010] In a possible implementation, determining the division method of each parameter and the computing device corresponding to each link according to the link type in the cluster configuration includes:

[0011] According to the first type of links and the second type of links in the cluster configuration, the division method of each parameter and the computing device corresponding to each link are determined, the first type of links are bandwidth links connecting computing devices within the computing node, and the second type of links are bandwidth links connecting computing devices between computing nodes. The first type of links and the second type of links are divided according to bandwidth and are links that do not affect each other during communication.

[0012] In a possible implementation, the method further includes:

[0013] According to the links corresponding to the communication groups and the parameters included, each communication group is started concurrently, so that each communication group can realize concurrent execution of communication.

[0014] In a possible implementation, the method further includes:

[0015] Obtain the neural network model to be trained and the training data;

[0016] Based on the communication group and the training data, the neural network model to be trained is trained until a trained neural network model is obtained.

[0017] In a possible implementation, the training of the neural network model to be trained based on the communication group and the training data until a trained neural network model is obtained includes:

[0018] Based on the communication group and the training data, performing forward propagation of a neural network model in each computing node;

[0019] Perform back propagation of the neural network model in each computing node and perform gradient aggregation within the node;

[0020] Perform global gradient updates between computing nodes to obtain a trained neural network model.

[0021] In a possible implementation, performing back propagation of the neural network model in each computing node and performing gradient aggregation in the node includes:

[0022] Starting from the output layer, the gradient of the loss function is propagated back along the network, and the gradient value of each layer is calculated layer by layer;

[0023] After calculating the gradient value of each computing device, the gradients from all computing devices are aggregated using the reduced scatter communication primitive, and the entire gradient is divided among all computing devices to obtain the gradient results corresponding to the parameter slices on each computing device.

[0024] In a possible implementation, performing global gradient update among computing nodes to obtain a trained neural network model includes:

[0025] The full protocol communication primitive is used to standardize the gradient results of each computing device, perform global gradient updates, and obtain the trained neural network model.

[0026] The second aspect of the present application provides a neural network model partitioning system for parallel training, comprising:

[0027] An acquisition unit, used to acquire a cluster configuration, wherein the cluster configuration includes information used to characterize a link connection condition of the cluster, a link bandwidth condition of the cluster, a number of computing devices, and a type of computing nodes;

[0028] A first determining unit, configured to determine a division method of each parameter and a computing device corresponding to each link according to a type of link in the cluster configuration;

[0029] The second determining unit is used to determine a communication group based on the division method of each parameter and the computing device corresponding to each link, wherein the communication group includes parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes.

[0030] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the neural network model partitioning method for parallel training described in the first aspect is implemented.

[0031] The fourth aspect of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes the neural network model partitioning method for parallel training as described in the first aspect above.

[0032] Compared with the prior art, this application has the following beneficial effects:

[0033] By obtaining the cluster configuration, according to the link type in the cluster configuration, determine the division method of each parameter and the computing device corresponding to each link; based on the division method of each parameter and the computing device corresponding to each link, determine the communication group, the communication group includes parameters that use the corresponding link in the computing node and the corresponding link between the computing nodes to communicate simultaneously. That is, specify a separate division method for each parameter of the model, so that during the training process, each parameter can communicate on its separate link. For the respective communication processes of sharded data parallelism and data parallelism, different parameters of the model can use the corresponding links in the computing node and between nodes to communicate simultaneously, making use of the originally idle links, reducing the communication volume on the original links, thereby reducing the total communication overhead and improving the model training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flow chart of a model partitioning method provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a parameter division method provided in an embodiment of the present application;

[0037] Figure 3 A flowchart of another model partitioning method provided in an embodiment of the present application;

[0038] Figure 4 A structural diagram of a model partitioning system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0041] As mentioned above, the current method to improve sharded data parallelism is to limit the scope of model partitioning to one computing node (generally including multiple computing devices), and only perform sharded data parallelism within the computing node. Only divide the data set between multiple computing nodes, perform data parallelism, and distribute the copies of each parameter partition on other devices.

[0042] For example, if there are 2 computing nodes, each of which contains 4 computing devices, a total of 2×4=8 computing devices. Then, on the 4 computing devices in each computing node, the model and data set are divided, and sharded data parallelism is used. Only data parallelism is performed between the corresponding 2 computing nodes. Since the main communication is carried out between the computing devices in the computing nodes, high-bandwidth link communication can be used to reduce the communication overhead of training.

[0043] However, this approach has the following drawbacks:

[0044] First, in the sharded data parallel approach, although the inter-device communication within the computing node utilizes high-bandwidth links, a large amount of communication is still required during the entire training process. This is mainly because:

[0045] In distributed training, the model parameters on each computing node (or device) need to be synchronized regularly to ensure that the models on all nodes remain consistent. As the model size increases, the number of parameters that need to be synchronized also increases significantly, resulting in a sharp increase in communication overhead. In addition, although data is mainly processed in parallel in shards within the computing nodes, cross-node data exchange (such as gradient aggregation, model parameter updates, etc.) is still necessary, and this part of the communication overhead cannot be ignored. For example, in the training process of a large model using this sharded data parallel method, communication time accounts for up to 45% of the entire model training process.

[0046] Secondly, the existing sharded data parallel method first uses the links within the computing node for communication, and then uses the links between computing nodes for communication. This sequentiality results in only some links being used at a certain moment, while other links are idle. The model parameters are uniformly and coarsely divided to different computing devices, resulting in all parameter updates needing to be transmitted through the same link. In addition, the communication strategy is not dynamically adjusted according to the actual network topology and device performance, so link resources cannot be optimally utilized.

[0047] Therefore, in order to solve the above problems, the embodiment of the present application provides a neural network model partitioning method for parallel training. According to the given model and cluster configuration, the corresponding link and computing device are specified for each parameter in the model, so as to make full use of the different link connections in the cluster and provide a basis for the communication overlap of the training process. In the training stage, an iterative training method is adopted until the model reaches the target accuracy. In each iteration, a sharded data parallel data set partitioning method is used in the computing node, and a data parallel data set partitioning method is used between the computing nodes. Forward propagation and backward propagation of the neural network model are performed in each computing node, and gradient aggregation is performed within the node. After that, a global gradient update is performed between the computing nodes to complete the training.

[0048] Therefore, the method provided by the embodiment of the present application specifies separate partitioning methods and communication links for different parameters of the model on the basis of traditional sharding data parallelism and data parallelism. Different parameters perform sharding data parallelism in different groups, and perform data parallelism in the corresponding groups. Parameters on different links can communicate concurrently without affecting each other. Different parameters of the model can make full use of idle links, thereby reducing the communication overhead of model training and increasing the speed of model training. In addition, since communication is avoided from becoming a performance bottleneck, this method can effectively improve the expansion efficiency of large-scale neural network model training.

[0049] It should be noted that the neural network model partitioning method, system, device and medium for parallel training provided in this application can be applied to the field of computer technology. The above is only an example and does not limit the application field of the model partitioning method, system, device and medium provided in this application. In addition, the embodiment of the present application may not limit the execution subject of the model partitioning. For example, the model partitioning method of the embodiment of the present application can be applied to data processing devices such as terminal devices or servers. Among them, the terminal device can be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cloud server, or a cluster server composed of multiple servers.

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The following is an example of an embodiment to illustrate a neural network model partitioning method for parallel training provided by the present application. Figure 1 ,Should Figure 1 A flow chart of a model partitioning method provided in an embodiment of the present application, the method comprising:

[0052] S101. Obtain cluster configuration.

[0053] The cluster configuration includes information for characterizing the link connection status of the cluster, the link bandwidth status of the cluster, the number of computing devices and the type of computing nodes.

[0054] For example, assume that the total number of computing nodes is 2, the number of computing devices contained in each computing node is 2, and the total number of computing devices is 2×2=4. Assume that there are two types of links with different bandwidths in the cluster, one is a high-bandwidth link connecting computing devices within a computing node, and the other is a low-bandwidth link connecting computing devices between computing nodes. The two links can communicate simultaneously without affecting each other.

[0055] S102: Determine a division method for each parameter and a computing device corresponding to each link according to the link type in the cluster configuration.

[0056] According to the link type of the cluster, the possible parallel partitioning method of the sharded data for each parameter in the model is determined, and the computing device corresponding to each link is determined.

[0057] In a possible implementation, determining the division method of each parameter and the computing device corresponding to each link according to the link type in the cluster configuration includes:

[0058] According to the first type of links and the second type of links in the cluster configuration, a division method of each parameter and a computing device corresponding to each link are determined.

[0059] That is, if there are two types of links with different bandwidths in the cluster, one is a high-bandwidth link connecting computing devices within a computing node, and the other is a low-bandwidth link connecting computing devices between computing nodes, the two links can communicate simultaneously without affecting each other. The first type of link is a bandwidth link connecting computing devices within a computing node, and the second type of link is a bandwidth link connecting computing devices between computing nodes. The first type of link and the second type of link are divided according to bandwidth and do not affect each other when communicating simultaneously.

[0060] S103: Determine a communication group based on the division method of each parameter and the computing device corresponding to each link.

[0061] The communication group includes parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes.

[0062] like Figure 2 As shown, Figure 2 A schematic diagram of the parameter partitioning method provided for the embodiment of the present application, wherein the two computing devices in the horizontal direction belong to the same computing node, and the two computing devices in the vertical direction belong to two different computing nodes. Since there are two types of links in total, each parameter of the model has two ways of partitioning, namely, partitioning on two computing devices in the horizontal direction within the computing node, or partitioning on two computing devices in the vertical direction between computing nodes. Taking two parameters P and Q as an example, the dotted box represents the communication group of each parameter. For parameter P, the partition is selected on the link within the computing node, and P is divided into two parts, P0 and P1, in the horizontal direction; and for parameter Q, the partition is selected on the link between the computing nodes, and Q is divided into two parts, Q0 and Q1, in the vertical direction. Therefore, different parameters P and Q each have different communication groups, and their communication will be started on independent links to achieve concurrent execution of communication.

[0063] In a possible implementation, different parameters P and Q each have a different communication group, and their communications can be started on independent links, thereby achieving concurrent execution of communications.

[0064] Therefore, the embodiment of the present application makes full use of the idle links in the cluster and enables the communication of different parameters of the model to be overlapped and executed concurrently, which can effectively reduce the communication overhead and improve the training speed of large-scale neural network models. And when expanding to more nodes, it can maintain linear expansion.

[0065] The following is an example of an embodiment to illustrate a neural network model partitioning method for parallel training provided by the present application. Figure 3 ,Should Figure 3 A flowchart of another model partitioning method provided in an embodiment of the present application, the method comprising:

[0066] S201. Obtain a neural network model to be trained, training data, and cluster configuration.

[0067] The cluster configuration may include computing node type, number of computing devices, link connection status of the cluster, link bandwidth status of the cluster, etc.

[0068] S202: Determine a communication group according to the link type of the cluster.

[0069] The communication group is determined based on the division method of each parameter and the computing device corresponding to each link, and the communication group includes parameters for communicating simultaneously using corresponding links within a computing node and corresponding links between computing nodes. Figure 2In the figure, the parameters P and Q, the dotted boxes represent the communication groups of each parameter. For parameter P, we choose to divide it on the links within the computing node, and P is divided into two parts P0 and P1 along the horizontal direction; while for parameter Q, we choose to divide it on the links between computing nodes, and Q is divided into two parts Q0 and Q1 along the vertical direction. Therefore, different parameters P and Q each have different communication groups, and each communication group can be started on an independent link to achieve concurrent execution of communication.

[0070] That is, the corresponding partitioning links and computing devices can be specified for each parameter in advance according to the preset partitioning strategy. This includes determining whether the parameters are partitioned within the computing node or between the computing nodes, and on which computing devices the partitioned parameters should be placed. When executing this step, the partitioning method of each parameter and the computing device corresponding to each link are determined according to the link type in the cluster configuration. Then, the communication group is determined based on the partitioning method of each parameter and the computing device corresponding to each link.

[0071] S203: Based on the communication group and the training data, the neural network model to be trained is trained until a trained neural network model is obtained.

[0072] After determining the division method for each parameter, an iterative training method can be used until the model reaches the target accuracy. In each iteration, the data set division method of sharding data parallelism is used within the computing node, and the data set division method of data parallelism is used between computing nodes. The forward propagation and backward propagation of the neural network model are performed in each computing node, and the gradient aggregation within the node is performed. After that, the global gradient update is performed between computing nodes to complete the training.

[0073] In the actual application scenario, after completing the model parameter partitioning method specified in S202, the training steps can be determined, including the forward propagation within the node, the reverse propagation within the node, and the gradient update between the nodes. That is, the forward propagation of the neural network model is performed in each computing node, and then the reverse propagation of the neural network model is performed in each computing node, and the gradient aggregation within the node is performed. Finally, the global gradient update is performed between the computing nodes to complete the training.

[0074] First, the forward propagation process within a node is explained.

[0075] S301. Aggregate corresponding model parameters based on the full aggregation communication primitive to obtain complete model parameters.

[0076] During the forward propagation process, each layer requires a corresponding part of the complete model parameters before performing tensor calculations. Therefore, the AllGather communication primitive is used to aggregate the corresponding model parameters.

[0077] S302: Perform tensor calculation based on the complete model parameters obtained through aggregation to obtain intermediate activation values.

[0078] S303. After obtaining the calculation results, retain some of the parameters obtained by division until the next tensor calculation.

[0079] After each calculation result is obtained, the computing device releases the complete parameters just aggregated and only retains the original divided parameters until the next tensor calculation. Finally, the forward propagation obtains the prediction result of the model and only retains a part of the model parameters. Figure 1 As shown in the figure, during the forward propagation process, the number of communications is very frequent, and the parameters of each communication (taking P and Q parameters as an example) are fully aggregated and communicated simultaneously through different links, which effectively reduces the communication overhead during the forward propagation process.

[0080] S304, starting from the output layer, the gradient of the loss function is propagated back along the network, and the gradient value of each layer is calculated layer by layer.

[0081] Then the back propagation process of model training is carried out. The back propagation process starts from the output layer, propagates the gradient of the loss function backward along the network, and calculates the gradient value of each layer layer by layer.

[0082] Before each calculation, the corresponding part of the model also needs to be fully modeled. Therefore, the full aggregation communication primitive is also used to aggregate the corresponding model parameters, and the aggregated parameters are released after the calculation is completed.

[0083] S305. After calculating the gradient of each computing device, use the reduced scatter communication primitive to aggregate the gradients from all devices, and divide the entire gradient among all devices to obtain the gradient results corresponding to the parameter slices on each computing device.

[0084] Finally, the gradient update communication process is carried out. In each computing node, after each computing device calculates the gradient, sharded data parallelism will use the specification scatter communication primitive to aggregate the gradients from all devices and divide the entire gradient among all devices, thereby obtaining the gradient results corresponding to the parameter shards on each computing device.

[0085] S306. Use the full protocol communication primitive to standardize the gradient results of each computing device and perform a global gradient update.

[0086] After obtaining the gradient results corresponding to the parameter slices on each computing device, data parallelism uses the AllReduce communication primitive to reduce the gradient results of each computing device and perform global gradient updates, thus completing a complete and efficient training iteration.

[0087] Therefore, the model training process provided by the embodiment of the present application has the following beneficial effects:

[0088] During the forward propagation and back propagation processes, the communication of different parameters is carried out simultaneously on different links, which effectively reduces the communication overhead.

[0089] By using AllGather, ReduceScatter, and AllReduce communication primitives, the efficiency of the overall training process is improved.

[0090] This approach allows computing devices within each computing node to communicate simultaneously on different links, thereby reducing the overall communication overhead and increasing the speed of model training.

[0091] In summary, the method provided by this application effectively reduces communication overhead and improves the training speed of large-scale neural network models by making full use of idle links in the cluster and overlapping and concurrently executing the communications of different model parameters. And when expanding to more nodes, it can almost maintain linear expansion.

[0092] This method proposes a new parameter-level fine-grained model partitioning method. On the basis of traditional sharding data parallelism and data parallelism, separate partitioning methods and communication links are specified for different parameters of the model. Different parameters are sharded in different groups and data parallelized in the corresponding groups. Parameters on different links can communicate concurrently without affecting each other.

[0093] The originally huge sharded data parallelism and communication in data parallelism can be hidden to a certain extent through idle links, avoiding the performance bottleneck problem caused by communication overhead in the training process of large-scale neural network models. It can be effectively applied to the training process of large-scale neural network models, and can be used together with other training methods to improve the efficiency of parallel training.

[0094] In addition, since communication is avoided as a performance bottleneck, this method can effectively improve the scalability efficiency of large-scale neural network model training.

[0095] The above are some specific implementations of the model partitioning method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding system for model partitioning. The system provided in the embodiment of the present application will be introduced from the perspective of functional modularization. Figure 4 A structural diagram of a model partitioning system provided in an embodiment of the present application.

[0096] The system comprises:

[0097] An acquisition unit 110 is used to acquire a cluster configuration, where the cluster configuration includes information used to characterize a link connection status of the cluster, a link bandwidth status of the cluster, a number of computing devices, and a type of computing nodes;

[0098] A first determining unit 111 is used to determine a division method of each parameter and a computing device corresponding to each link according to the link type in the cluster configuration;

[0099] The second determining unit 112 is used to determine a communication group based on the division method of each parameter and the computing device corresponding to each link, wherein the communication group includes parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes.

[0100] In a possible implementation manner, the first determining unit is specifically configured to:

[0101] According to the first type of links and the second type of links in the cluster configuration, the division method of each parameter and the computing device corresponding to each link are determined, the first type of links are bandwidth links connecting computing devices within the computing node, and the second type of links are bandwidth links connecting computing devices between computing nodes. The first type of links and the second type of links are divided according to bandwidth and are links that do not affect each other during communication.

[0102] In a possible implementation, the system further includes:

[0103] The concurrent starting unit is used to concurrently start each communication group according to the links corresponding to the communication groups and the parameters included, so that each communication group can realize concurrent execution of communication.

[0104] In a possible implementation, the system further includes:

[0105] A data acquisition unit, used to acquire a neural network model to be trained and training data;

[0106] A training unit is used to train the neural network model to be trained based on the communication group and the training data until a trained neural network model is obtained.

[0107] In a possible implementation manner, the training unit is specifically used to:

[0108] Based on the communication group and the training data, performing forward propagation of a neural network model in each computing node;

[0109] Perform back propagation of the neural network model in each computing node and perform gradient aggregation within the node;

[0110] Perform global gradient updates between computing nodes to obtain a trained neural network model.

[0111] In a possible implementation, performing back propagation of the neural network model in each computing node and performing gradient aggregation in the node includes:

[0112] Starting from the output layer, the gradient of the loss function is propagated back along the network, and the gradient value of each layer is calculated layer by layer;

[0113] After calculating the gradient value of each computing device, the gradients from all computing devices are aggregated using the reduced scatter communication primitive, and the entire gradient is divided among all computing devices to obtain the gradient results corresponding to the parameter slices on each computing device.

[0114] In a possible implementation, performing global gradient update among computing nodes to obtain a trained neural network model includes:

[0115] The full protocol communication primitive is used to standardize the gradient results of each computing device, perform global gradient updates, and obtain the trained neural network model.

[0116] The embodiment of the present application also provides corresponding devices and computer storage media for implementing the model partitioning method provided in the embodiment of the present application.

[0117] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the model partitioning method described in any embodiment of the present application.

[0118] The computer storage medium stores codes, and when the codes are executed, the device executing the codes implements the model partitioning method described in any embodiment of the present application.

[0119] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0120] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0121] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0122] It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0123] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0124] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0125] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A neural network model partitioning method for parallel training, characterized in that: include: Acquire cluster configuration, wherein the cluster configuration includes information used to characterize link connection status of the cluster, link bandwidth status of the cluster, number of computing devices, and type of computing nodes; Determine, according to the link type in the cluster configuration, a division method of each parameter and a computing device corresponding to each link; Determining a communication group based on the division method of each parameter and the computing device corresponding to each link, the communication group including parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes; The determining, according to the link type in the cluster configuration, a division method of each parameter and a computing device corresponding to each link includes: According to the first type of links and the second type of links in the cluster configuration, the division method of each parameter and the computing device corresponding to each link are determined, the first type of links are bandwidth links connecting computing devices within the computing node, and the second type of links are bandwidth links connecting computing devices between computing nodes. The first type of links and the second type of links are divided according to bandwidth and are links that do not affect each other during communication.

2. The method according to claim 1, characterized in that The method further comprises: According to the links corresponding to the communication groups and the parameters included, each communication group is started concurrently, so that each communication group can realize concurrent execution of communication.

3. The method according to claim 1, characterized in that The method further comprises: Obtain the neural network model to be trained and the training data; Based on the communication group and the training data, the neural network model to be trained is trained until a trained neural network model is obtained.

4. The method according to claim 3, characterized in that The step of training the neural network model to be trained based on the communication group and the training data until a trained neural network model is obtained comprises: Based on the communication group and the training data, performing forward propagation of a neural network model in each computing node; Perform back propagation of the neural network model in each computing node and perform gradient aggregation within the node; Perform global gradient updates between computing nodes to obtain a trained neural network model.

5. The method according to claim 4, characterized in that The back propagation of the neural network model is performed in each computing node, and the gradient aggregation in the node is performed, including: Starting from the output layer, the gradient of the loss function is propagated back along the network, and the gradient value of each layer is calculated layer by layer; After calculating the gradient value of each computing device, the gradients from all computing devices are aggregated using the reduced scatter communication primitive, and the entire gradient is divided among all computing devices to obtain the gradient results corresponding to the parameter slices on each computing device.

6. The method according to claim 4, characterized in that The global gradient update is performed between the computing nodes to obtain a trained neural network model, including: The full protocol communication primitive is used to standardize the gradient results of each computing device, perform global gradient updates, and obtain the trained neural network model.

7. A neural network model partitioning system for parallel training, characterized in that: The system comprises: An acquisition unit, used to acquire a cluster configuration, wherein the cluster configuration includes information used to characterize a link connection condition of the cluster, a link bandwidth condition of the cluster, a number of computing devices, and a type of computing nodes; A first determining unit, configured to determine a division method of each parameter and a computing device corresponding to each link according to a type of link in the cluster configuration; A second determining unit, configured to determine a communication group based on the division manner of each parameter and the computing device corresponding to each link, wherein the communication group includes parameters for simultaneously communicating using corresponding links within a computing node and corresponding links between computing nodes; The determining, according to the link type in the cluster configuration, a division method of each parameter and a computing device corresponding to each link includes: According to the first type of links and the second type of links in the cluster configuration, the division method of each parameter and the computing device corresponding to each link are determined, the first type of links are bandwidth links connecting computing devices within the computing node, and the second type of links are bandwidth links connecting computing devices between computing nodes. The first type of links and the second type of links are divided according to bandwidth and are links that do not affect each other during communication.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the neural network model partitioning method for parallel training as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the neural network model partitioning method for parallel training as described in any one of claims 1 to 6.

Images