Hybrid expert model parallel training expansion method and hybrid expert model training cluster

Abstract

The embodiment of the invention discloses a hybrid expert model parallel training expansion method and a hybrid expert model training cluster, and the method comprises the steps: obtaining a current model input sequence of a hybrid expert model and cluster architecture association configuration information of a current model training cluster; wherein the current model training cluster is used for performing distributed parallel training on the hybrid expert model; determining training strategy association information of a current parallel training strategy of the hybrid expert model; determining a cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy association information and the cluster architecture association configuration information; and expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster. According to the technical scheme, the rationality of parallel training expansion of the hybrid expert model can be improved, and then the training efficiency of the hybrid expert model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a hybrid expert model parallel training expansion method and a hybrid expert model training cluster. Background Art

[0002] The Mixture of Expert (MoE) model is a sparse neural network that can greatly increase the number of model parameters without increasing the amount of model calculations, thereby improving the model's capabilities. At the same time, due to the sparsity of the model, usually 1-2 experts are selected for processing for each token (the smallest unit of input data used by the model), and different data are processed by different experts.

[0003] The hybrid expert model is a large model. Multimodal information such as text, images, videos, and voice can be converted into tokens (the smallest unit of input data used by the model) through the tokenizer, so that the large model has the ability to understand the world, which will further expand the context length. The entire training process of the hybrid expert model involves multiple training processes with context lengths ranging from 4K to 128K, or even 1M tokens.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: the cluster used to train the hybrid expert model often needs to be expanded to meet the distributed parallel training requirements of the hybrid expert model. However, there is currently no effective expansion strategy for clusters used for hybrid expert model parallel training. Summary of the invention

[0005] The embodiments of the present invention provide a hybrid expert model parallel training expansion method and a hybrid expert model training cluster, which can improve the rationality of hybrid expert model parallel training expansion and further improve the training efficiency of the hybrid expert model.

[0006] According to one aspect of the present invention, a hybrid expert model parallel training expansion method is provided, comprising:

[0007] Acquire the cluster architecture association configuration information of the current model input sequence of the hybrid expert model and the current model training cluster; wherein the current model training cluster is used to perform distributed parallel training on the hybrid expert model;

[0008] Determining training strategy association information of a current parallel training strategy of the hybrid expert model;

[0009] Determine a cluster expansion strategy for the current model training cluster according to the current model input sequence, the training strategy association information, and the cluster architecture association configuration information;

[0010] The current model training cluster is expanded according to the cluster expansion strategy of the current model training cluster.

[0011] According to another aspect of the present invention, there is provided a hybrid expert model training cluster, comprising a plurality of nodes, wherein the hybrid expert model parallel training cluster is used to perform distributed parallel training on the hybrid expert model;

[0012] The hybrid expert model parallel training cluster is expanded according to the hybrid expert model parallel training expansion method described in any embodiment of the present invention.

[0013] The embodiment of the present invention obtains the current model input sequence of the hybrid expert model and the cluster architecture associated configuration information of the current model training cluster, and determines the training strategy associated information of the current parallel training strategy of the hybrid expert model, thereby determining the cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy associated information and the cluster architecture associated configuration information, and then expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster. The expanded current model training cluster can be used for distributed parallel training of the hybrid expert model. The above technical scheme can solve the problem of unreasonable expansion of existing hybrid expert model parallel training, improve the rationality of hybrid expert model parallel training expansion, and thus improve the training efficiency of the hybrid expert model.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a flow chart of a hybrid expert model parallel training expansion method provided in Embodiment 1 of the present invention;

[0017] Figure 2 It is a flowchart of a long text training step of a hybrid expert model provided in the first embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the effect of parallel execution of multiple distributed strategies in a single module of a MoE model provided by the first embodiment of the present invention;

[0019] Figure 4 is a flow chart of a hybrid expert model parallel training expansion method provided by Embodiment 2 of the present invention;

[0020] Figure 5 It is a flowchart of a hybrid expert model parallel training expansion method provided in the second embodiment of the present invention;

[0021] Figure 6 It is a schematic diagram of the effect of horizontally expanding the current model training cluster for training a hybrid expert model provided by the second embodiment of the present invention;

[0022] Figure 7 It is a schematic diagram of the effect of vertically expanding the current model training cluster for training a hybrid expert model provided by the second embodiment of the present invention;

[0023] Figure 8 It is a structural diagram of a hybrid expert model training cluster provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1is a flow chart of a hybrid expert model parallel training expansion method provided in the first embodiment of the present invention. This embodiment is applicable to the case where a cluster for distributed training of a hybrid expert model is reasonably expanded according to cluster configuration information and training strategy related information. Figure 1 As shown, the method includes the following operations:

[0028] S110, obtaining the cluster architecture association configuration information of the current model input sequence of the hybrid expert model and the current model training cluster; wherein the current model training cluster is used to perform distributed parallel training on the hybrid expert model.

[0029] Among them, the current model input sequence can be a data sequence that needs to be input to the hybrid expert model, that is, sequence. It can be understood that the current model input sequence can include at least one token. The current model training cluster can be a cluster currently used to train the hybrid expert model, and the cluster has a certain number of cluster device nodes (hereinafter referred to as nodes). Optionally, the hybrid expert model can be trained in a distributed parallel manner based on the current model training cluster. The cluster architecture associated configuration information can be the relevant architecture configuration information of the current model training cluster, for example, it can include but is not limited to the number of cluster device nodes included and the number of training cards configured for each cluster device node. The training card, that is, the core hardware of the cluster device node for training the hybrid expert model, can be, for example, a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit) or other chips. The embodiment of the present invention does not limit the specific hardware type of the training card.

[0030] Before performing distributed parallel training on the hybrid expert model, the current model training cluster used for distributed parallel training on the hybrid expert model can be determined first. Further, after determining the current model training cluster, the cluster architecture associated configuration information of the current model training cluster can be obtained.

[0031] It should be noted that if the input sequence provided to the hybrid expert model is a long text type sequence, then for the long text sequence, different training stages need to be used to execute distributed parallel training processes respectively. Figure 2 is a flow chart of a hybrid expert model long text training step provided in the first embodiment of the present invention. In a specific example, Figure 2As shown, the original hybrid expert model is used as the basic model. If the original length of the input text is 1M, the basic model needs to start from training an input sequence of 32K (initial text sequence length) and gradually expand to an input sequence of 1M. That is, in the first training stage, the text sequence length of the training data set is 32K, and the 32k input sequence can be directly used as the current model input sequence to perform distributed parallel training on the basic model of the hybrid expert model to obtain the model after the first stage training. In the second training stage, the text length of the training data set is 128K, and the 128K text can be segmented, and the number of segments is the value of the context parallelism. The context parallelism is the ratio of the current input text sequence length to the initial text sequence length: 128 / 32=4, that is, the 128k text sequence is divided into 4 32k input sequences, and the 4 32k input sequences are used as the current model input sequence to perform distributed parallel training on the model after the first stage training to obtain the model after the second stage training. And so on, until the last stage, 1M is divided into 32 32K input sequences, and the 32 32K input sequences are used as the current model input sequence to perform distributed parallel training on the model trained in the fourth stage, and the final trained hybrid expert model is obtained.

[0032] In summary, since the length of the current model input sequence of the hybrid expert model may be different in different distributed parallel training stages, it is also necessary to determine the current model input sequence used for the hybrid expert model. For example, in the first stage of long text training, the length of the current model input sequence can be 32K and the number is 1; in the third stage of long text training, the length of the current model input sequence can be 256K and the number is 8, etc.

[0033] In an optional embodiment of the present invention, the current model training cluster can adopt at least one parallel strategy among the tensor parallel (TP) strategy, sequence parallel (SP) strategy, expert parallel (EP) strategy, context parallel (CP) strategy, pipeline parallel (PP) strategy and data parallel (DP) strategy to perform distributed parallel training on the hybrid expert model.

[0034] Among them, the tensor parallel strategy is to divide the model weights into several parts along a specific dimension, and calculate the sub-matrices of the matrix multiplication on different devices. After the calculation is completed, these sub-results need to be merged through collective communication operations to restore the complete result. The sequence parallel strategy is to divide the model input sequence into multiple sub-sequences and calculate them in parallel on different devices. By dividing the sequence, the amount of sequence state that each device needs to store is reduced, and the memory usage is reduced. The expert parallel strategy is to distribute the expert layer of the model to multiple computing devices. Each expert layer is a small sub-model that focuses on processing different aspects or sub-tasks of the input data. It can effectively utilize the computing power of multiple devices and reduce the memory load of a single device. The context parallel strategy is to split the sequence length (sequence length) and calculate and aggregate it in parallel to reduce the demand for video memory size, which is particularly suitable for processing ultra-long sequences. The pipeline parallel strategy is to divide the model into several blocks by layer and assign them to different devices. In the forward pass, after each device processes a micro-batch, the next device continues to process this micro-batch. At the same time, the device processes the next micro-batch, thereby realizing pipeline processing. The data parallel strategy is to copy the same model weights to multiple devices (such as multiple GPUs), and then divide the training data into multiple subsets, with each device processing a portion of the data. In this way, multiple devices can perform forward propagation, backpropagation, and parameter updates simultaneously, thereby speeding up training.

[0035] Figure 3 1 is a schematic diagram of the effect of parallel execution of multiple distributed strategies in a single module of a MoE model provided by the first embodiment of the present invention. Figure 3 As shown, in the process of using the current model training cluster to perform distributed parallel training on the hybrid expert model, one or more of the above parallel strategies can be sampled to perform a distributed parallel training process.

[0036] S120: Determine training strategy association information of the current parallel training strategy of the hybrid expert model.

[0037] The current parallel training strategy may include the current distributed parallel training strategy. The current distributed parallel training strategy may be the distributed parallel training strategy currently adopted by the current model training cluster. The training strategy associated information may be related information of the current parallel training strategy and the long text training step strategy.

[0038] Exemplarily, when the EP parallel strategy is adopted to train the hybrid expert model, the training strategy associated information may include but is not limited to the EP parallelism, etc. When the current long text training stage is determined to be the third stage according to the long text training step strategy, the training strategy associated information may also include the length of the training data set specified in the third stage of the current long text training stage, etc.

[0039] S130. Determine a cluster expansion strategy for the current model training cluster according to the current model input sequence, the training strategy association information, and the cluster architecture association configuration information.

[0040] The cluster expansion strategy may be a strategy for expanding the current model training cluster.

[0041] Due to the different strategies of the long text training steps, the configuration of the current model input sequence is also different. At the same time, the current distributed parallel training strategy adopted by the hybrid expert model is different, and the expansion method of the current model training cluster is also different. Therefore, the training strategy association information of the current parallel training strategy of the hybrid expert model can also be determined. Furthermore, the cluster expansion strategy applicable to the current model training cluster can be determined based on the obtained current model input sequence and cluster architecture association configuration information combined with the training strategy association information.

[0042] S140: Expand the current model training cluster according to the cluster expansion strategy of the current model training cluster.

[0043] It is understandable that the training pressure borne by the current model training cluster varies with different training tasks. Therefore, after determining the current model input sequence and training strategy association information, the cluster expansion strategy of the current model training cluster can be determined based on the cluster architecture association configuration information of the current model training cluster, and the cluster architecture of the current model training cluster can be expanded according to the cluster expansion strategy of the current model training cluster.

[0044] Exemplarily, the method of expanding the current model training cluster may include, for example, expansion in the longitudinal direction and / or expansion in the lateral direction. Among them, longitudinal expansion can be understood as the EP group direction and the CP group direction are in the same direction, and the EP group processes different data blocks of the input sequence. When the input sequence length is very long, the longitudinal expansion of the EP group meets the needs of processing the input sequence. Horizontal expansion can be understood as the EP group direction and the CP group direction are orthogonal, and the EP group processes the same data block of different batches of the input sequence. When the input sequence length is short, the horizontal expansion of the EP group meets the needs of processing the input sequence.

[0045] The embodiment of the present invention obtains the current model input sequence of the hybrid expert model and the cluster architecture associated configuration information of the current model training cluster, and determines the training strategy associated information of the current parallel training strategy of the hybrid expert model, thereby determining the cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy associated information and the cluster architecture associated configuration information, and then expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster. The expanded current model training cluster can be used for distributed parallel training of the hybrid expert model. The above technical scheme can solve the problem of unreasonable expansion of existing hybrid expert model parallel training, improve the rationality of hybrid expert model parallel training expansion, and thus improve the training efficiency of the hybrid expert model.

[0046] Embodiment 2

[0047] Figure 4 is a flow chart of a hybrid expert model parallel training expansion method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods for determining the cluster expansion strategy of the current model training cluster and expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster are provided. Figure 4 As shown, the method of this embodiment may include:

[0048] S210, obtaining the cluster architecture association configuration information of the current model input sequence of the hybrid expert model and the current model training cluster; wherein the current model training cluster is used to perform distributed parallel training on the hybrid expert model.

[0049] S220: Determine training strategy association information of the current parallel training strategy of the hybrid expert model.

[0050] In an embodiment of the present invention, optionally, before determining the cluster expansion strategy of the current model training cluster, the optimal distributed parallel strategy suitable for the hybrid expert model under the condition of the initial text input sequence length when cp_world_size (CP parallelism) is 1 can be first determined based on the computing and network configuration of the current model training cluster and the structure of the hybrid expert model.

[0051] Exemplarily, multiple distributed training strategies for distributed training of the hybrid expert model can be determined, and the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained are calculated. According to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, each distributed training strategy is screened to obtain the optimal distributed parallel strategy suitable for the hybrid expert model.

[0052] Exemplarily, when screening each distributed training strategy based on the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model, each distributed training strategy can be screened for the first time based on the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model and the cluster storage resources to obtain the first candidate distributed training strategy. Furthermore, the communication bandwidth requirements of each first candidate distributed training strategy are calculated based on the communication volume and computing volume of each first candidate distributed training strategy, and then the communication bandwidth requirements of each first candidate distributed training strategy and the cluster bandwidth resources are screened for the second time to obtain the second candidate distributed training strategy. Finally, the optimal distributed parallel strategy suitable for the hybrid expert model is determined based on the second candidate distributed training strategy.

[0053] Exemplarily, when determining the optimal distributed parallel strategy suitable for the hybrid expert model according to the second candidate distributed training strategy, the training performance index of each second candidate distributed training strategy can be specifically calculated; wherein the training performance index includes at least one of throughput, latency, and E2E training duration. Further, according to the training performance index of each second candidate distributed training strategy, the optimal distributed training strategy is selected from each second candidate distributed training strategy.

[0054] Accordingly, the optimal distributed training strategy is selected as the current distributed parallel training strategy of the hybrid expert model.

[0055] S230: Calculate a maximum processing sequence of a single expert group according to the cluster architecture association configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model.

[0056] The maximum processing sequence of a single expert group is also the maximum length of the sequence that a single expert parallel group (Expert Parallel_Group, EP_Group) can process. The expert parallel group can be an expert group composed of expert network weights stored in the corresponding training cards of a certain number of nodes. Each computing device of the expert parallel group includes a part of the network weights of multiple different expert networks. Different distributed training strategies can be configured with corresponding parallel groups.

[0057] In a specific example, assuming that each node (Node) is composed of 8 GPUs, the weight of each expert network is divided into 8 parts, and each weight of an expert network is placed in a node. A training card of a node can store a weight of an expert network, and 8 nodes can store the weights of 8 expert networks. Each node corresponding card can form an EP_Group. For example, the weights of the expert networks stored by GPU1 of node 1, GPU1 of node 2, GPU1 of node 3, GPU1 of node 4, GPU1 of node 5, GPU1 of node 6, GPU1 of node 7, and GPU1 of node 8 can constitute an expert parallel group. It can be seen that multiple expert parallel groups can realize the expert parallel computing process.

[0058] It is understandable that before determining the cluster expansion strategy of the current model training cluster, it is necessary to clarify the maximum length sequence that an expert parallel group can process, that is, the maximum processing sequence of a single expert group. The maximum processing sequence of a single expert group is different, and the corresponding specific cluster expansion strategy is also different. Specifically, the maximum processing sequence of a single expert group can be calculated based on the cluster architecture association configuration information of the current model training cluster and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model.

[0059] In an optional embodiment of the present invention, the calculation of the maximum processing sequence of a single expert group based on the cluster architecture associated configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model may include: determining the single card processing sequence length of the current model training cluster based on the cluster architecture associated configuration information and the training strategy associated information; determining the expert parallelism of the expert parallel strategy adopted by the hybrid expert model based on the training strategy associated information; and calculating the product of the single card processing sequence length and the expert parallelism to obtain the maximum processing sequence of the single expert group.

[0060] The single-card processing sequence length may be the length of a sequence that can be processed by a single training card in the cluster.

[0061] Specifically, the length of the single-card processing sequence of the current model training cluster can be first determined according to the cluster architecture associated configuration information and the training strategy associated information. Exemplarily, after the optimal distributed training strategy selected according to the cluster architecture associated configuration information is used as the current distributed parallel training strategy of the hybrid expert model, the length of the processing sequence of the training card defined by the current distributed parallel training strategy can be used as the length of the single-card processing sequence of the current model training cluster. Furthermore, the expert parallelism of the expert parallel strategy adopted by the hybrid expert model can be determined according to the training strategy associated information, and then the product of the single-card processing sequence length and the expert parallelism can be calculated to obtain the maximum processing sequence of a single expert group.

[0062] S240: Determine a cluster expansion strategy for the current model training cluster according to a size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group.

[0063] In an embodiment of the present invention, specifically, the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of a single expert group can be judged, thereby determining the cluster expansion strategy applicable to the current model training cluster based on the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of a single expert group.

[0064] In an optional embodiment of the present invention, determining the cluster expansion strategy of the current model training cluster based on the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group may include: when it is determined that the length of the current model input sequence is less than the length of the maximum processing sequence of the single expert group, and when it is determined that the cluster network switch meets the horizontal expansion conditions, using the expert group horizontal expansion strategy as the cluster expansion strategy of the current model training cluster; when it is determined that the length of the current model input sequence is greater than or equal to the length of the maximum processing sequence of the single expert group, using the expert group vertical expansion strategy as the cluster expansion strategy of the current model training cluster.

[0065] Among them, the horizontal expansion condition can be the condition that the current model training cluster of the current model training cluster can be horizontally expanded. Exemplarily, the horizontal expansion condition can be that the number of interfaces of the cluster network switch is greater than or equal to the value of ep_world_size*cp_world_size. Among them, ep_world_size is the EP parallelism, and cp_world_size is the CP parallelism. The expert group horizontal expansion strategy is an expansion strategy for expanding a certain number of expert parallel groups in the horizontal direction. The expert group vertical expansion strategy is an expansion strategy for expanding a certain number of expert parallel groups in the vertical direction.

[0066] Specifically, if the length of the current model input sequence is less than the length of the maximum processing sequence of a single expert group, that is, Input Sequence < ep_world_size * single-card processing sequence length, where Input Sequence is the length of the current model input sequence, and it is determined that the cluster network switch meets the horizontal expansion condition, that is, the cluster network switch meets ep_world_size * cp_world_size, then horizontal expansion can be used, that is, the horizontal expansion strategy of the expert group is used as the cluster expansion strategy of the current model training cluster. If it is determined that the length of the current model input sequence is greater than or equal to the length of the maximum processing sequence of a single expert group, that is, Input Sequence >= ep_world_size * single-card processing sequence length, then vertical expansion can be used, and the vertical expansion strategy of the expert group is used as the cluster expansion strategy of the current model training cluster.

[0067] S250. Determine the cluster expansion direction and the number of node expansions of the current model training cluster according to the cluster expansion strategy of the current model training cluster.

[0068] S260. Expand the current model training cluster according to the cluster expansion direction and the number of node expansions.

[0069] Correspondingly, after determining the cluster expansion strategy of the current model training cluster, the cluster expansion strategy of the current model training cluster can be parsed to determine the cluster expansion direction and the number of node expansions of the current model training cluster, so as to expand the current model training cluster in the cluster expansion direction according to the number of node expansions. It can be understood that the expanded model training cluster can meet the current distributed parallel training requirements of the mixture of experts model.

[0070] In an optional embodiment of the present invention, determining the cluster expansion direction and the number of node expansion of the current model training cluster according to the cluster expansion strategy of the current model training cluster may include: when it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, taking the horizontal expansion direction as the cluster expansion direction of the current model training cluster; calculating the quotient between the length of the current model input sequence and the length of the single card processing sequence as the target horizontal expansion value; determining the number of horizontal expansion nodes of the current model training cluster according to the target horizontal expansion value and the number of expert parallel groups included in the current model training cluster; when it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, taking the vertical expansion direction as the cluster expansion direction of the current model training cluster; calculating the quotient between the length of the current model input sequence and the length of the maximum processing sequence of a single expert group as the target vertical expansion value; determining the number of vertical expansion nodes of the current model training cluster according to the target vertical expansion value and the number of expert parallel groups included in the current model training cluster.

[0071] The target horizontal expansion value may be the total number of expert parallel groups that are ultimately expanded in the horizontal direction calculated for the current model training cluster. The node horizontal expansion number may be the number of expert parallel groups that need to be expanded and supplemented in the horizontal direction calculated for the current model training cluster. The target vertical expansion value may be the total number of expert parallel groups that are ultimately expanded in the vertical direction calculated for the current model training cluster. The node vertical expansion number may be the number of expert parallel groups that need to be supplemented in the vertical direction calculated for the current model training cluster.

[0072] Specifically, if it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, the horizontal expansion direction can be used as the cluster expansion direction of the current model training cluster. If it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, the vertical expansion direction can be used as the cluster expansion direction of the current model training cluster. After determining that the cluster expansion direction is the horizontal expansion direction, the quotient between the length of the current model input sequence and the length of the single card processing sequence can be further calculated as the target horizontal expansion value. The target horizontal expansion value can be used to determine the scale of the final model training cluster after the current model training cluster is horizontally expanded. Correspondingly, the difference between the target horizontal expansion value and the number of expert parallel groups included in the current model training cluster is calculated to obtain the number of horizontal expansion nodes of the current model training cluster. After determining that the cluster expansion direction is the vertical expansion direction, the quotient between the length of the current model input sequence and the length of the maximum processing sequence of a single expert group can be further calculated as the target vertical expansion value. The target vertical expansion value can be used to determine the scale of the final model training cluster after the current model training cluster is vertically expanded. Correspondingly, the difference between the target vertical expansion value and the number of expert parallel groups included in the current model training cluster is calculated to obtain the number of vertical expansion nodes of the current model training cluster.

[0073] For example, assuming that the length of the current model input sequence is 1M, and the length of the maximum processing sequence of a single expert group is 256k, if it is the vertical expansion direction, the target vertical expansion value is 1M / 256k=4. If the number of expert parallel groups included in the current model training cluster is 2, it can be calculated that the number of vertical expansion nodes of the current model training cluster is 2.

[0074] In an optional embodiment of the present invention, the expanding the current model training cluster according to the cluster expansion direction and the node expansion number may include: when it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, expanding the current model training cluster according to the horizontal expansion direction to increase the expert group nodes of the target horizontal expansion value; when it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, expanding the current model training cluster according to the vertical expansion direction to increase the expert group nodes of the target vertical expansion value.

[0075] The expert group node may be a plurality of nodes that can constitute an expert parallel group. For example, when an expert parallel group includes 8 training cards, the expert group node may include 8 nodes.

[0076] Correspondingly, after calculating the number of node expansions of the current model training cluster, the calculated number of node expansions can be used as a benchmark, and the current model training cluster can be expanded with reference to the determined cluster expansion strategy. Specifically, if the cluster expansion strategy of the current model training cluster is determined to be the expert group horizontal expansion strategy, the current model training cluster can be expanded in the horizontal expansion direction to increase the expert group nodes of the target horizontal expansion value; if the cluster expansion strategy of the current model training cluster is determined to be the expert group vertical expansion strategy, the current model training cluster can be expanded in the vertical expansion direction to increase the expert group nodes of the target vertical expansion value.

[0077] In an optional embodiment of the present invention, the hybrid expert model parallel training expansion method may also include: when it is determined that the current model input sequence and / or the cluster architecture associated configuration information have changed, re-obtaining the changed current model input sequence, cluster architecture associated configuration information and training strategy associated information; and re-determining the cluster expansion strategy of the current model training cluster based on the changed current model input sequence, cluster architecture associated configuration information and training strategy associated information.

[0078] It is understandable that in the process of hybrid expert model training, as the training stage evolves, when entering different training stages, it can be determined according to the text training step strategy that the current model input sequence may change. At the same time, when the current model training cluster is expanded at each stage, the cluster architecture associated configuration information of the current model training cluster will also change accordingly. Therefore, if it is determined that the current model input sequence and / or cluster architecture associated configuration information has changed, the changed current model input sequence, cluster architecture associated configuration information and training strategy associated information can be re-acquired to re-determine the cluster expansion strategy of the current model training cluster based on the changed current model input sequence, cluster architecture associated configuration information and training strategy associated information, thereby ensuring the rationality and effectiveness of the cluster expansion strategy of the current model training cluster.

[0079] In a specific example, Figure 5 is a flow chart of a hybrid expert model parallel training expansion method provided by the second embodiment of the present invention, such as Figure 5As shown in the figure, in the multi-stage training process of the hybrid expert model for long texts, first, according to the computing and network configuration of the cluster and the structure of the hybrid expert model, the optimal distributed parallel training strategy adapted to the hybrid expert model under the initial text sequence length when the CP parallelism is 1 can be determined. Furthermore, the text lengths of different input sequences are calculated according to different training stages of the long text, and the CP parallelism is calculated. Specifically, the CP parallelism is the ratio of the current input text sequence length to the initial text sequence length. Finally, according to the network topology between cluster machines, the CP parallelism and the EP parallelism, a reasonable cluster expansion strategy is selected for expansion.

[0080] In a specific example, Figure 6 1 is a schematic diagram of the effect of horizontally expanding the current model training cluster for training a hybrid expert model provided by the second embodiment of the present invention. Figure 7 1 is a schematic diagram of the effect of vertically expanding the current model training cluster for training a hybrid expert model provided by the second embodiment of the present invention. Figure 6 and Figure 7 As shown in the figure, the MoE model 8x70B is taken as an example, where "8x70B" means that the MoE model has 8 experts and the total parameter scale reaches 70 billion. Assuming that the input sequence is a long text of 1M tokens, each node in the current model training cluster consists of 8 GPUs, and the current model training cluster uses distributed training strategies such as TP, SP, EP, CP, PP and DP for distributed parallel training. Among them, TP and SP are usually used at the same time.

[0081] Among them, tensor parallel group (Tensor Parallel_Group, TP_Group) can be achieved through Figure 6 and Figure 7 The blue dotted box in the figure indicates that tp_world_size (tensor parallelism) is 8, 8 training cards are tensor parallel, and each training card stores 1 / 8 of the weight of the non-expert network module in the hybrid expert model and 1 / 8 of the weight of an expert network module, such as Indicates the 0th expert weight 0 to 7 splits, node 7 is Indicates the 0 to 7 cuts of the 7th expert weight, etc. Figure 6In the horizontal expansion strategy, each node processes a 32K-length sequence of a fragment chunk (chunk) of a minibatch (a small batch or batch, in deep learning, a batch refers to a collection of data samples processed in a single forward propagation and backpropagation process), and each training card processes the same 32K data. The 32K-length sequence can be further divided into smaller fragments such as 4K through sequence parallelism, and the SP_Group (Sequence Parallel_Group) is consistent with the TP_Group. Figure 6 and Figure 7 [G0~G7] in the TP_Group.

[0082] Expert parallel groups can be Figure 6 and Figure 7 The red box in the figure indicates that the expert parallelism is 8. The weight of each expert network is divided into 8 parts and placed in one node. 8 nodes can store 8 expert network weights. Each node corresponds to a card to form an EP_Group, such as Figure 6 and Figure 7 [G0, G8, G16, G24, G32, G40, G48, G56] form an EP_Group.

[0083] Context Parallel_Group (CP_Group) can be Figure 6 and Figure 7 The green box in the figure indicates that the ContextParallel parallel group is different under different expansion strategies. Figure 6 In the horizontal expansion strategy, the context parallelism is 32, the 1M input sequence is divided into 32 fragment blocks, the text sequence length of a fragment block is 32k, each node processes a text sequence of 32K, and a total of 32 nodes are required to process a text sequence of 1M. Figure 6 [G0, G64~G1984] in the CP_Group constitutes a CP Group. Figure 7 In the vertical expansion strategy, the 1M sequence is split according to the EP_Group direction, that is, each node in each EP_Group processes 32K fragments of the same 1M sequence, and 4 EP_Groups complete the processing of the 1M sequence. Figure 7 [G0, G8~G248] form a CP_Group, and EP_Group is a subset of CP_Group.

[0084] Pipeline Parallel Group (PP_Group): Figure 6 and Figure 7It shows a stage (pipe) of the PP_Group processing the MoE model. When the MoE model is relatively large, the MoE model needs to be divided into approximately equal numbers of block layers. That is, the MoE model is divided into different stages, and each stage is assigned to a different node. Each node processes its own stage and collaborates with other nodes in sequence to complete the training of the entire model. For example, the first pipe group processes layers 0-9 of the MoE model, the second pipe group processes layers 10-19 of the MoE model, and so on.

[0085] Data Parallel_Group (DP_Group): Figure 6 and Figure 7 In, each CP_Group processes a minibatch. The different colors marked by the squares corresponding to different nodes in the figure represent different minibatch data. 8 experts process a minibatch of 8, that is, 8 sequences of 1M. Figure 6 In horizontal scaling, the minibatch is equal to the EP parallelism. When the PP parallelism is 1, the smallest unit of a DP_Group that processes 1M sequences consists of 256 nodes and 2048 cards, and processes a minibatch of 8. The training dataset is divided into multiple subsets, and the entire MoE model is replicated on each DP_Group node. Each DP_Group node independently performs forward propagation and backward propagation. After each batch is processed, the DP_Group nodes exchange gradients and average the gradients to ensure that the model parameters on all nodes are consistent. For example, if the Global Batch (the size of the total sample set for one round of training in the cluster) is 64, then 8 DP_Group nodes are required, that is, 2048 * 8 = 16384 GPUs.

[0086] Correspondingly, when the length of the current model input sequence (i.e., the Input Sequence length) is less than the maximum sequence length that one EP_Group can process (such as 256k), that is, Input Sequence < ep_world_size * the sequence length processed by a single card, and the cluster network switch satisfies ep_world_size * cp_world_size, the expert group horizontal scaling strategy can be used, that is, expand one or more expert parallel groups in the horizontal direction. Specifically, as Figure 6 shown, assume that the length of the current model input sequence is 1M, one node can process 32k data, and one EP-Group can process 32k data. When expanding the current model training cluster, one group of EP-Group (such as 8 nodes) can be added in the horizontal direction each time until the requirements are met. In Figure 6In the scenario shown, the current model training cluster can be expanded to 32 EP-Groups in total, that is, after the current model training cluster is expanded, there are 32 EP-Groups in total horizontally, with a total of 256 nodes. Figure 6 As shown in the figure, in the horizontal expansion strategy, CP_Group and EP_Group are in an orthogonal topological relationship. When CP is horizontally expanded, microbatch is equal to EP parallelism. The above cluster expansion method can improve the utilization rate of a single node.

[0087] Correspondingly, when the length of the current model input sequence is greater than or equal to the length that can be processed by one EP_Group (such as 256k), that is, when Input Sequence>=ep_world_size*single card processing sequence length, the expert group vertical expansion strategy can be used, that is, one or more expert parallel groups can be expanded vertically, and the cluster can be flexibly configured according to the minimum expansion granularity to meet the training of Global Batch. Specifically, Figure 7 As shown in the figure, when a sequence greater than 256K in length needs to be processed and the Global Batch is 4, the vertical expansion is 128 nodes and 1024 training cards. If the horizontal expansion is performed, the minimum number of cards required to process the Global Batch is 2048. Specifically, Figure 7 As shown in the figure, assuming that the length of the current model input sequence is 1M, one node can process 32k data, and one EP-Group can process 32*8=256k data, then when expanding the current model training cluster, one group of EP-Groups (such as 8 nodes) can be added vertically each time until the demand is met. Figure 7 In the scenario shown, a total of 4 EP-Groups can be expanded, that is, there are a total of 4 EP-Groups in the horizontal direction, with a total of 32 nodes.

[0088] It can be seen that the tensor parallel group and expert parallel group in the vertical expansion are the same as the horizontal expansion strategy, and the difference is the CP parallel group. In the vertical expansion strategy, CP_Group and EP_Group are in the same direction. The microbatch of a single card in CP vertical expansion is 1. In CP vertical expansion, the network topology of EP and CP is in the same dimension, and in CP horizontal expansion, the network topology of EP and CP is orthogonal. Expert parallel and context parallel communication use inter-machine communication resources, usually northbound NIC or southbound Scale out to achieve cluster expansion. In CP horizontal expansion, EP and CP are orthogonal, and a larger switch network needs to be formed to meet the communication requirements of EP and CP at the same time, that is, the scale is cp_world_size*ep_world_size. In CP vertical expansion, CP and EP are in the same dimension, and the networking scale is, max(ep_world_size, cp_world_size).

[0089] The above technical solution proposes a hybrid expert model parallel training expansion method, which can generate an expert model distributed parallel expansion strategy according to the storage capacity, computing power, network topology, model structure and text length of the cluster, thereby improving the training efficiency of the hybrid expert model. Specifically, different expert parallel expansion methods are selected according to the length of the training Context. Vertical expansion is applicable to expert parallel expansion under long texts, and horizontal expansion is applicable to expert parallel expansion under short texts. At the same time, combined with different cluster network configurations and communication hardware architectures, as well as different stages of long text training, different distributed parallel expansion schemes are reasonably set to improve the computing speed during hybrid expert model training, thereby improving the utilization of hardware resources and the execution efficiency of AI models, and can be widely applied to various types of AI (Artificial Intelligence) training clusters.

[0090] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0091] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0092] Embodiment 3

[0093] Figure 8 is a schematic diagram of the structure of a hybrid expert model training cluster provided by the third embodiment of the present invention, such as Figure 8 As shown, the structure of the hybrid expert model training cluster includes multiple nodes, and the hybrid expert model parallel training cluster is used to perform distributed parallel training on the hybrid expert model.

[0094] The hybrid expert model parallel training cluster is expanded according to the hybrid expert model parallel training expansion method described in any embodiment of the present invention.

[0095] Optionally, the hybrid expert model parallel training cluster can be expanded according to the following hybrid expert model parallel training expansion method:

[0096] Acquire the cluster architecture association configuration information of the current model input sequence of the hybrid expert model and the current model training cluster; wherein the current model training cluster is used to perform distributed parallel training on the hybrid expert model;

[0097] Determining training strategy association information of a current parallel training strategy of the hybrid expert model;

[0098] Determine a cluster expansion strategy for the current model training cluster according to the current model input sequence, the training strategy association information, and the cluster architecture association configuration information;

[0099] The current model training cluster is expanded according to the cluster expansion strategy of the current model training cluster.

[0100] Optionally, determining the cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy association information and the cluster architecture association configuration information may include:

[0101] Calculate the maximum processing sequence of a single expert group according to the cluster architecture association configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model;

[0102] The cluster expansion strategy of the current model training cluster is determined according to the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group.

[0103] Optionally, the calculating the maximum processing sequence of a single expert group according to the cluster architecture association configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model may include:

[0104] Determine the single card processing sequence length of the current model training cluster according to the cluster architecture associated configuration information and the training strategy associated information;

[0105] Determining the expert parallelism of the expert parallel strategy adopted by the hybrid expert model according to the training strategy association information;

[0106] The product of the single card processing sequence length and the expert parallelism is calculated to obtain the maximum processing sequence of the single expert group.

[0107] Optionally, determining the cluster expansion strategy of the current model training cluster according to the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group may include:

[0108] When it is determined that the length of the current model input sequence is less than the length of the maximum processing sequence of the single expert group, and when it is determined that the cluster network switch meets the horizontal expansion condition, the expert group horizontal expansion strategy is used as the cluster expansion strategy of the current model training cluster;

[0109] When it is determined that the length of the current model input sequence is greater than or equal to the length of the maximum processing sequence of the single expert group, the expert group vertical expansion strategy is used as the cluster expansion strategy of the current model training cluster.

[0110] Optionally, the expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster may include:

[0111] Determine the cluster expansion direction and node expansion number of the current model training cluster according to the cluster expansion strategy of the current model training cluster;

[0112] The current model training cluster is expanded according to the cluster expansion direction and the node expansion quantity.

[0113] Optionally, determining the cluster expansion direction and the number of node expansion of the current model training cluster according to the cluster expansion strategy of the current model training cluster may include:

[0114] In the case where it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, the horizontal expansion direction is used as the cluster expansion direction of the current model training cluster;

[0115] Calculating the quotient between the length of the current model input sequence and the length of the single card processing sequence as the target horizontal expansion value;

[0116] Determine the number of nodes for horizontal expansion of the current model training cluster according to the target horizontal expansion value and the number of expert parallel groups included in the current model training cluster;

[0117] In the case where it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, taking the vertical expansion direction as the cluster expansion direction of the current model training cluster;

[0118] Calculate the quotient between the length of the current model input sequence and the length of the maximum processed sequence of a single expert group as the target longitudinal expansion value;

[0119] The number of vertical expansion nodes of the current model training cluster is determined according to the target vertical expansion value and the number of expert parallel groups included in the current model training cluster.

[0120] Optionally, the expanding the current model training cluster according to the cluster expansion direction and the node expansion quantity may include:

[0121] When it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, the expert group nodes of the current model training cluster are expanded according to the horizontal expansion direction to increase the target horizontal expansion value;

[0122] When it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, the expert group nodes of the current model training cluster are expanded in the vertical expansion direction to increase the target vertical expansion value.

[0123] Optionally, the above method may further include:

[0124] When it is determined that the current model input sequence and / or the cluster architecture associated configuration information have changed, reacquiring the changed current model input sequence, cluster architecture associated configuration information, and training strategy associated information;

[0125] The cluster expansion strategy of the current model training cluster is re-determined according to the changed current model input sequence, cluster architecture associated configuration information, and training strategy associated information.

[0126] Optionally, the current model training cluster adopts at least one parallel strategy among tensor parallel strategy, sequence parallel strategy, expert parallel strategy, context parallel strategy, pipeline parallel strategy and data parallel strategy to perform distributed parallel training on the hybrid expert model.

[0127] The embodiment of the present invention obtains the current model input sequence of the hybrid expert model and the cluster architecture associated configuration information of the current model training cluster, and determines the training strategy associated information of the current parallel training strategy of the hybrid expert model, thereby determining the cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy associated information and the cluster architecture associated configuration information, and then expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster. The expanded current model training cluster can be used for distributed parallel training of the hybrid expert model. The above technical scheme can solve the problem of unreasonable expansion of existing hybrid expert model parallel training, improve the rationality of hybrid expert model parallel training expansion, and thus improve the training efficiency of the hybrid expert model.

[0128] The hybrid expert model training cluster in the embodiment of the present invention can be expanded by using the hybrid expert model parallel training expansion method provided in any embodiment of the present invention. For technical details not described in detail in this embodiment, please refer to the hybrid expert model parallel training expansion method provided in any embodiment of the present invention.

Claims

1. A hybrid expert model parallel training expansion method, characterized in that: include: Acquire the cluster architecture association configuration information of the current model input sequence of the hybrid expert model and the current model training cluster; wherein the current model training cluster is used to perform distributed parallel training on the hybrid expert model; Determining training strategy association information of a current parallel training strategy of the hybrid expert model; Determine a cluster expansion strategy for the current model training cluster according to the current model input sequence, the training strategy association information, and the cluster architecture association configuration information; The current model training cluster is expanded according to the cluster expansion strategy of the current model training cluster.

2. The method according to claim 1, characterized in that: The step of determining the cluster expansion strategy of the current model training cluster according to the current model input sequence, the training strategy association information, and the cluster architecture association configuration information includes: Calculate the maximum processing sequence of a single expert group according to the cluster architecture association configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model; The cluster expansion strategy of the current model training cluster is determined according to the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group.

3. The method according to claim 2, characterized in that The calculating the maximum processing sequence of a single expert group according to the cluster architecture association configuration information and the expert parallelism of the expert parallel strategy adopted by the hybrid expert model includes: Determine the single card processing sequence length of the current model training cluster according to the cluster architecture associated configuration information and the training strategy associated information; Determining the expert parallelism of the expert parallel strategy adopted by the hybrid expert model according to the training strategy association information; The product of the single card processing sequence length and the expert parallelism is calculated to obtain the maximum processing sequence of the single expert group.

4. The method according to claim 2, characterized in that: The step of determining the cluster expansion strategy of the current model training cluster according to the size relationship between the length of the current model input sequence and the length of the maximum processing sequence of the single expert group includes: When it is determined that the length of the current model input sequence is less than the length of the maximum processing sequence of the single expert group, and when it is determined that the cluster network switch meets the horizontal expansion condition, the expert group horizontal expansion strategy is used as the cluster expansion strategy of the current model training cluster; When it is determined that the length of the current model input sequence is greater than or equal to the length of the maximum processing sequence of the single expert group, the expert group vertical expansion strategy is used as the cluster expansion strategy of the current model training cluster.

5. The method according to claim 1, characterized in that: The step of expanding the current model training cluster according to the cluster expansion strategy of the current model training cluster includes: Determine the cluster expansion direction and node expansion number of the current model training cluster according to the cluster expansion strategy of the current model training cluster; The current model training cluster is expanded according to the cluster expansion direction and the node expansion quantity.

6. The method according to claim 5, characterized in that The determining the cluster expansion direction and the number of node expansion of the current model training cluster according to the cluster expansion strategy of the current model training cluster includes: In the case where it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, the horizontal expansion direction is used as the cluster expansion direction of the current model training cluster; Calculating the quotient between the length of the current model input sequence and the length of the single card processing sequence as the target horizontal expansion value; Determine the number of nodes for horizontal expansion of the current model training cluster according to the target horizontal expansion value and the number of expert parallel groups included in the current model training cluster; In the case where it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, taking the vertical expansion direction as the cluster expansion direction of the current model training cluster; Calculate the quotient between the length of the current model input sequence and the length of the maximum processed sequence of a single expert group as the target longitudinal expansion value; The number of vertical expansion nodes of the current model training cluster is determined according to the target vertical expansion value and the number of expert parallel groups included in the current model training cluster.

7. The method according to claim 6, characterized in that The step of expanding the current model training cluster according to the cluster expansion direction and the node expansion quantity includes: When it is determined that the cluster expansion strategy of the current model training cluster is the expert group horizontal expansion strategy, the expert group nodes of the current model training cluster are expanded according to the horizontal expansion direction to increase the target horizontal expansion value; When it is determined that the cluster expansion strategy of the current model training cluster is the expert group vertical expansion strategy, the expert group nodes of the current model training cluster are expanded in the vertical expansion direction to increase the target vertical expansion value.

8. The method according to claim 1, characterized in that Also includes: When it is determined that the current model input sequence and / or the cluster architecture associated configuration information have changed, reacquiring the changed current model input sequence, cluster architecture associated configuration information, and training strategy associated information; The cluster expansion strategy of the current model training cluster is re-determined according to the changed current model input sequence, cluster architecture associated configuration information, and training strategy associated information.

9. The method according to any one of claims 1 to 8, characterized in that: The current model training cluster adopts at least one parallel strategy among tensor parallel strategy, sequence parallel strategy, expert parallel strategy, context parallel strategy, pipeline parallel strategy and data parallel strategy to perform distributed parallel training on the hybrid expert model.

10. A hybrid expert model training cluster, characterized in that: Comprising a plurality of nodes, the hybrid expert model parallel training cluster is used for performing distributed parallel training on the hybrid expert model; The hybrid expert model parallel training cluster is expanded according to the hybrid expert model parallel training expansion method according to any one of claims 1 to 9.

Images