Distributed training method, device, equipment, medium and program for hybrid expert model

Abstract

The embodiment of the present invention discloses a method, device, equipment, medium and program for distributed training of a hybrid expert model, the method comprising: determining multiple distributed training strategies for distributed training of the hybrid expert model; calculating the storage resources occupied by each of the distributed training strategies when the hybrid expert model is distributed trained; screening each of the distributed training strategies according to the storage resources occupied by each of the distributed training strategies when the hybrid expert model is distributed trained, and obtaining a target distributed training strategy for distributed training of the hybrid expert model. The technical solution of the embodiment of the present invention can improve the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and the execution efficiency of the model when the hybrid expert model is distributed trained.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a hybrid expert model distributed training method, device, electronic device, storage medium and program product. Background Art

[0002] The Mixture of Expert (MoE) model is a sparse neural network that can greatly increase the number of model parameters without increasing the amount of model calculations, thereby improving the model's capabilities. At the same time, due to the sparsity of the model, usually 1-2 experts are selected for processing for each token (the smallest unit of input data used by the model), and different data are processed by different experts.

[0003] At present, the existing MoE training framework involves a distributed training method of 6D parallel training. The so-called 6D parallel training means using EP (Expert Parallel), DP (Data Parallel), PP (Pipeline Parallel), SP (Sequence Parallel), CP (ContextParallel) and TP (Tensor Parallel) at the same time.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: Due to the high complexity of 6D parallelism, the parallel group size is usually set manually. Once improperly set, it will lead to a high communication ratio, which seriously affects the chip computing power utilization. Therefore, in the process of distributed training of hybrid expert models, there is currently a lack of an effective method for screening hybrid expert distributed parallel training strategies. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, electronic device, storage medium and program product for distributed training of a hybrid expert model, which can improve the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and the execution efficiency of the model during distributed training of the hybrid expert model.

[0006] According to one aspect of the present invention, a hybrid expert model distributed training method is provided, comprising:

[0007] determining a plurality of distributed training strategies for distributed training of the mixture of experts model;

[0008] Calculating the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model;

[0009] According to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributedly trained, each distributed training strategy is screened to obtain a target distributed training strategy;

[0010] The target distributed training strategy is used to perform distributed training on the hybrid expert model.

[0011] According to another aspect of the present invention, a hybrid expert model distributed training device is provided, comprising:

[0012] A distributed training strategy determination module, used to determine a plurality of distributed training strategies for performing distributed training on the hybrid expert model;

[0013] A storage resource calculation module, used to calculate the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model;

[0014] A target distributed training strategy screening module is used to screen each of the distributed training strategies according to the storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a target distributed training strategy;

[0015] The target distributed training strategy is used to perform distributed training on the hybrid expert model.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model distributed training method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model distributed training method described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the hybrid expert model distributed training method described in any embodiment of the present invention.

[0022] The embodiment of the present invention determines multiple distributed training strategies for distributed training of the hybrid expert model, and calculates the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, thereby screening each distributed training strategy according to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, and obtains the target distributed training strategy for distributed training of the hybrid expert model. Accordingly, the screened target distributed training strategy can be used for distributed training of the hybrid expert model. The above technical scheme can solve the problem of low training performance when the hybrid expert model is trained using a distributed training method in the prior art, and can improve the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and the execution efficiency of the model when the hybrid expert model is distributed trained.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a hybrid expert model distributed training method provided in Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a hybrid expert model distributed training method provided in Embodiment 2 of the present invention;

[0027] Figure 3 It is a flow chart of a hybrid expert model distributed training method provided in the second embodiment of the present invention;

[0028] Figure 4 It is a structural diagram of a non-MoE module and a MoE module in a hybrid expert model provided in Embodiment 2 of the present invention;

[0029] Figure 5 This is a schematic diagram of the effect of parallel execution of multiple distributed strategies in a single module of a MoE model provided by the second embodiment of the present invention;

[0030] Figure 6 is a schematic diagram of a hybrid expert model distributed training device provided in Embodiment 3 of the present invention;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second" and "target" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 This is a flowchart of a hybrid expert model distributed training method provided by the first embodiment of the present invention. This embodiment can be applied to the case where the storage resources occupied by the hybrid expert model during distributed training are selected according to the distributed training strategy to train the hybrid expert model. The method can be executed by a hybrid expert model distributed training device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can execute the hybrid expert model distributed training method, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0036] S110. Determine multiple distributed training strategies for performing distributed training on the hybrid expert model.

[0037] Among them, the distributed training strategy can be a strategy for distributed training of the hybrid expert model.

[0038] Distributed training refers to the process of training machine learning or deep learning model training tasks in parallel on multiple computing devices. Distributed training methods use multiple computing nodes, which usually include multiple CPUs (Central Processing Units), GPUs (Graphic Processing Units), and TPUs (Tensor Processing Units). Through parallel computing and data management, the model training process is significantly accelerated, and larger data sets and models can be processed than a single node can manage. Since single-machine training is limited by hardware resources, distributed training is usually a better choice for handling large-scale training tasks.

[0039] It is understandable that when performing distributed training on the hybrid expert model, a variety of optional distributed training strategies can be adopted. Before screening the distributed training strategy adapted to the hybrid expert model, multiple distributed training strategies that can be used for distributed training of the hybrid expert model can be first determined. Exemplarily, multiple distributed training strategies for distributed training of the hybrid expert model may include but are not limited to tensor parallel strategy (TP), sequence parallel strategy (SP), expert parallel strategy (EP), context parallel strategy (CP), pipeline parallel strategy (PP) and data parallel strategy (DP). The above strategies all belong to distributed parallel training strategies, and the hybrid expert model can be distributed and parallel trained. Among them, the tensor parallel strategy is to divide the model weights into several parts along a specific dimension, and calculate the sub-matrices of matrix multiplication on different devices respectively. After the calculation is completed, these sub-results need to be merged through collective communication operations to restore the complete result. The sequence parallel strategy is to divide the model input sequence into multiple sub-sequences and calculate them in parallel on different devices. By dividing the sequence, the amount of sequence state that needs to be stored on each device is reduced, and the memory occupancy is reduced. The expert parallel strategy is to distribute the expert layer of the model to multiple computing devices. Each expert layer is a small sub-model that focuses on processing different aspects or sub-tasks of the input data. It can effectively utilize the computing power of multiple devices and reduce the memory load of a single device. The context parallel strategy is to split the sequence length (sequence length) and calculate and summarize it in parallel to reduce the demand for video memory size, which is particularly suitable for processing ultra-long sequences. The pipeline parallel strategy is to divide the model into several blocks by layer and assign them to different devices. During the forward pass, after each device processes a micro-batch, the next device continues to process this micro-batch. At the same time, the device processes the next micro-batch, thereby realizing pipeline processing. The data parallel strategy is to copy the same model weights to multiple devices (such as multiple GPUs), and then divide the training data into multiple subsets, with each device processing a part of the data. In this way, multiple devices can perform forward propagation, backpropagation, and parameter updates at the same time, thereby speeding up training.

[0040] In the actual training process, a single parallel strategy can be used to perform distributed training on the hybrid expert model, or multiple hybrid parallel strategies can be used to perform distributed training on the hybrid expert model at the same time. The choice of distributed training strategy depends on the scale of the model, available hardware resources, and the requirements for training efficiency. Different distributed training strategies have their own advantages and disadvantages, and need to be selected and adjusted according to specific circumstances.

[0041] S120, calculating the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model.

[0042] It is understandable that different distributed training strategies may also cause different storage resources occupied by the current cluster device when performing distributed training on the hybrid expert model. The current cluster device may be a cluster device used for distributed training of the hybrid expert model.

[0043] S130. Screen each of the distributed training strategies according to the storage resources occupied when the hybrid expert model is distributedly trained, and obtain a target distributed training strategy, wherein the target distributed training strategy is used to perform distributed training on the hybrid expert model.

[0044] Among them, the target distributed training strategy can be obtained after screening various distributed training strategies, and is suitable for the distributed training strategy adopted by the current cluster device for the hybrid expert model, and can be applicable to the current cluster device for distributed training of the hybrid expert model.

[0045] For cluster devices that train hybrid expert models, the storage capacity of the cluster is one of the main factors that restrict whether the distributed training strategy can be successfully executed. Therefore, after determining multiple distributed training strategies for distributed training of the hybrid expert model, the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model can be first calculated, and then the target distributed training strategy that meets the storage resource requirements can be screened out from each distributed training strategy based on the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model, so as to perform distributed training on the hybrid expert model through the screened target distributed training strategy.

[0046] Exemplarily, the distributed training strategy for which the storage resources required for training exceed the storage resources that can be provided by the current cluster device can be excluded from the distributed training strategies to obtain the target distributed training strategy.

[0047] It can be seen that the above-mentioned hybrid expert model distributed training method takes the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model as the basis, and screens out the target distributed training strategy that occupies storage resources to meet the distributed training needs to perform distributed training on the hybrid expert model. This can ensure that the storage resources provided by the current cluster device meet the storage resource requirements of the target distributed training strategy, so that the current cluster device can successfully complete the distributed training process of the hybrid expert model through the target distributed training strategy, avoiding the problem of training interruption or even training failure due to insufficient storage resources during the training process, accelerating the execution process of distributed parallel training of the hybrid expert model, and improving the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and model execution efficiency during distributed training of the hybrid expert model.

[0048] The embodiment of the present invention determines multiple distributed training strategies for distributed training of the hybrid expert model, and calculates the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, thereby screening each distributed training strategy according to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, and obtains the target distributed training strategy for distributed training of the hybrid expert model. Accordingly, the screened target distributed training strategy can be used for distributed training of the hybrid expert model. The above technical scheme can solve the problem of low training performance when the hybrid expert model is trained using a distributed training method in the prior art, and can improve the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and the execution efficiency of the model when the hybrid expert model is distributed trained.

[0049] Embodiment 2

[0050] Figure 2 is a flow chart of a hybrid expert model distributed training method provided in Embodiment 2 of the present invention. Figure 3 : is a flow chart of a hybrid expert model distributed training method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods are provided for calculating the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model, and for screening each distributed training strategy to obtain a target distributed training strategy. Figure 2 and Figure 3 As shown, the method of this embodiment may include:

[0051] S210: Determine multiple distributed training strategies for performing distributed training on the hybrid expert model.

[0052] In an optional embodiment of the present invention, the hybrid expert model distributed training method may further include: obtaining cluster-associated resource parameters; wherein the cluster-associated resource parameters may include cluster configuration parameters, training data sets, computing power utilization, model structure parameters, and estimated training duration; estimating the number of training cards required for distributed training of the hybrid expert model based on the cluster-associated resource parameters; wherein the number of training cards is used to calculate the values ​​of pipeline parallelism, tensor parallelism, expert parallelism, sequence parallelism, context parallelism, and data parallelism.

[0053] Among them, the cluster-associated resource parameters may be relevant parameters of the current cluster device used for distributed training of the hybrid expert model. The cluster configuration parameters may be relevant configuration parameters of the current cluster device, for example, they may be the FLOPs (Floating-point Operations Per Second) of the GPU card computing power in all devices in the current cluster device. The training data set may include the total number of tokens. The model structure parameter may be the model parameter quantity. The estimated training duration may be the estimated training duration for the hybrid expert model, for example, it may be 2 weeks, one month or two months, etc. The embodiment of the present invention does not limit the specific duration value of the estimated training duration.

[0054] In an embodiment of the present invention, before screening a distributed training strategy for distributed training of a hybrid expert model, the number of training cards required for distributed training of the hybrid expert model can be estimated first according to the cluster-associated resource parameters of the current cluster device. The number of training cards is also the number of training cards for training the hybrid expert model, and the training card can be, for example, a chip such as a GPU. The estimated number of training cards can be used to calculate the value of the degree of parallelism required for each distributed training strategy.

[0055] In a specific example, the following expression can be used to estimate the number of training cards required for distributed training of the mixed expert model:

[0056] Estimated training time = total computing power required for the training task / total computing power of the cluster GPU;

[0057] The total computing power required for the training task = A*model parameter number*total number of tokens in the training data set;

[0058] The total GPU computing power of the cluster = the total number of GPUs in the cluster * the FLOPs of a single GPU card * the GPU computing power utilization.

[0059] Among them, the model parameter quantity belongs to the model structure parameter, the total number of tokens in the training data set belongs to the training data set, the total number of GPUs in the cluster is the number of training cards, and the total number of GPUs in the cluster * the FLOPs of the GPU single card computing power is the cluster configuration parameter. A is a constant, which is optional, and the value of A can be 6.

[0060] Optionally, the model parameter amount may be the parameter amount of the expert model activated for each token, and may include the non-MoE module parameter amount and the MoE module parameter amount. Figure 4 Schematic diagram of the structure of a non-MoE module and a MoE module in a hybrid expert model provided by the second embodiment of the present invention. Figure 4As shown, the non-MoE module parameter amount can be the parameter amount of other network modules except the expert network in the hybrid expert model. The MoE module parameter amount can be the parameter amount involved in the expert network module in the hybrid expert model.

[0061] Figure 5 Schematic diagram of the effect of parallel execution of multiple distributed strategies in a single module of a MoE model provided by the second embodiment of the present invention. Figure 5 As shown in the figure, in a single module of the MoE model, multiple distributed training strategies such as SP, TP, CP and EP can be executed in parallel at the same time. The parallel execution of multiple distributed training strategies to train MoE can improve the training efficiency of MoE.

[0062] S220, calculating the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model.

[0063] In an optional embodiment of the present invention, the calculation of the storage resources occupied by each of the distributed training strategies when the hybrid expert model is distributed trained may include: calculating the actual video memory of a single card occupied by each of the distributed training strategies when the hybrid expert model is distributed trained.

[0064] The actual video memory of a single card may be the video memory actually occupied by a single training card when the distributed training strategy performs distributed training on the hybrid expert model.

[0065] In the embodiment of the present invention, optionally, the actual video memory of a single card occupied by each distributed training strategy when performing distributed training on the hybrid expert model can be calculated as the storage resource occupied by each distributed training strategy when performing distributed training on the hybrid expert model.

[0066] In an optional embodiment of the present invention, the calculation of the actual video memory of a single card occupied when each of the distributed training strategies performs distributed training on the hybrid expert model may include: calculating the video memory occupied by the weight storage of a single card when each of the distributed training strategies performs distributed training on the hybrid expert model; calculating the video memory occupied by the activation storage of a single card when each of the distributed training strategies performs distributed training on the hybrid expert model; calculating the sum of the video memory occupied by the activation storage of the single card and the video memory occupied by the weight storage of the single card to obtain the actual video memory of the single card.

[0067] The single card weight storage may be the weight parameters required to be stored for a single training card. The single card activation storage may be the activation data required to be stored for a single training card. Optionally, the activation data may include input activation and / or output activation.

[0068] In an embodiment of the present invention, the video memory occupied by the distributed training strategy during the distributed training of the hybrid expert model is mainly divided into activation storage occupancy and weight storage occupancy. Therefore, when calculating the actual video memory of a single card occupied by each distributed training strategy when the hybrid expert model is distributed trained, the video memory occupied by the single card weight storage when each distributed training strategy is distributed trained on the hybrid expert model, and the video memory occupied by the single card activation storage when each distributed training strategy is distributed trained on the hybrid expert model can be calculated separately, thereby summing the calculated video memory occupied by the single card activation storage and the video memory occupied by the single card weight storage to obtain the actual video memory of a single card occupied by the distributed training strategy when the hybrid expert model is distributed trained.

[0069] In an optional embodiment of the present invention, the calculation of the video memory occupied by the single-card weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model may include: calculating the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques; calculating the sum of the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage to obtain the video memory occupied by the single-card weight storage.

[0070] Among them, the optimization technology can be an algorithm technology for optimizing the hybrid expert model. Exemplarily, the optimization technology can include but is not limited to Data Parallel Zero (Zero Redundancy Optimizer) O1, Data Parallel Zero O2, and Data Parallel Zero O3. When the optimization technology is not used, it can be used as a baseline. A benchmark usually refers to a simple and easy-to-implement benchmark for comparison with more complex machine learning models. Non-MoE module weight storage can be the storage of weight parameters of non-MoE modules in the hybrid expert model. MoE module weight storage can be the storage of weight parameters of MoE modules in the hybrid expert model.

[0071] In an embodiment of the present invention, the video memory occupancy can be estimated according to a combination of different distributed training strategies and optimization techniques. Specifically, the single-card video memory occupied by the non-MoE module weight storage and the single-card video memory occupied by the MoE module weight storage when each distributed training strategy performs distributed training on the hybrid expert model under different optimization techniques can be calculated, and then the single-card video memory occupied by the non-MoE module weight storage and the single-card video memory occupied by the MoE module weight storage are summed to obtain the video memory occupied by the single-card weight storage.

[0072] In an optional embodiment of the present invention, the calculation of the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization technologies may include: determining single-card weight association parameters adopted by the non-MoE module and the MoE module when each of the distributed training strategies performs distributed training on the hybrid expert model; wherein the single-card weight association parameters may include the non-MoE module parameter amount and the MoE module parameter amount, and the single-card weight association parameters also include at least one of pipeline parallelism, tensor parallelism, sequence parallelism, expert parallelism and data parallelism; calculating the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization technologies according to the single-card weight association parameters.

[0073] Among them, the non-MoE module is the network module other than the expert network in the hybrid expert model. The MoE module is the network module of the expert network in the hybrid expert model. The single card weight association parameter can be a parameter related to the video memory occupied by the single card weight storage for estimating the distributed training strategy.

[0074] Specifically, the non-MoE module parameter amount and the MoE module parameter amount can be used as single card weight association parameters, and at least one of pipeline parallelism, tensor parallelism, sequence parallelism, expert parallelism and data parallelism can be used as single card weight association parameters. The above-mentioned types of parallelism can be arbitrarily combined as single card weight association parameters. It should be noted that tensor parallelism and sequence parallelism are usually represented by one parameter.

[0075] In a specific example, taking fp16 (a data type that uses 2 bytes (16 bits) for encoding and storage) training and using the adam (Adaptive Moment Estimation) optimizer as an example, the formula shown in Table 1 can be used to calculate the video memory occupied by the weight storage of a single GPU. In Table 1, p is pipeline parallelism, t is tensor parallelism, or t represents both tensor parallelism and sequence parallelism, e is expert parallelism, and d is data parallelism. P1 is the number of non-MoE module parameters, and P2 is the number of parameters for all MoE expert models. Among them, the number of training cards = t*e*d*p. Therefore, the specific values ​​of p, t, e, and d can be calculated based on the number of training cards calculated above.

[0076] Table 1 List of methods for calculating the video memory occupied by weight storage

[0077]

[0078] As shown in Table 1, under different optimization techniques, the formulas corresponding to the non-MoE module storage can be used to calculate the video memory occupied by the non-MoE module weight storage, and the formulas corresponding to the MoE module storage can be used to calculate the video memory occupied by the MoE module weight storage. The total model storage of a single GPU is the sum of the non-MoE module storage and the MoE module storage, that is, the video memory occupied by the single card weight storage. In the above formulas, the values ​​of p, t, e, and d can be configured as needed. Exemplarily, when there is only a pipeline parallel strategy, p can have a corresponding value, and the values ​​of t, e, and d can be 1. When there are pipeline parallel strategies and expert parallel strategies, p and e can have corresponding values, and the values ​​of t and d can be 1.

[0079] In an optional embodiment of the present invention, the calculation of the video memory occupied by the single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model may include: determining the single card activation associated parameters used by each of the distributed training strategies when performing distributed training on the hybrid expert model; wherein the single card activation associated parameters include microbatch, sequence length, hidden layer dimension, number of attention heads, number of single token selection experts, and number of model layers, and the single card activation associated parameters also include at least one of tensor parallelism, sequence parallelism, and expert parallelism; and calculating, according to the single card activation associated parameters, the video memory occupied by the single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques.

[0080] In a specific example, the video memory occupied by single-card activation storage can be calculated according to different distributed training strategies and activation recalculation strategies, combined with different optimization techniques, with reference to the formula shown in Table 2. As shown in Table 2, microbatch (a microbatch is a subset of a batch, usually used in pipeline parallel training to reduce the memory requirements on each computing device, and at the same time, multiple devices can process a microbatch of data in parallel.), sequence length, hidden layer dimension, number of attention heads, number of single token selection experts, and number of model layers are used as single-card activation-related parameters. At the same time, at least one of tensor parallelism, sequence parallelism, and expert parallelism can also be used as a single-card activation-related parameter. Where b is microbatch; s is the sequence length. If there is a CP training strategy, it is expressed as a sequence block length; h is hidden dim (the size of the hidden layer dimension); a is the head number; k is the number of experts selected for each token; l is the number of transformer layers, t is tensor parallelism, or it represents tensor parallelism and sequence parallelism at the same time, and e is expert parallelism, see Table 2.

[0081] Table 2 List of calculation methods for video memory occupied by activated storage

[0082]

[0083] As shown in Table 2, under different optimization techniques, the video memory occupied by single-card activation storage when each distributed training strategy performs distributed training on the hybrid expert model can be calculated based on the single-card activation associated parameters combined with the corresponding expressions. The selective activation value recalculation involved in Table 2 is a technology that optimizes the use of video memory during large model training. In large model training, it is usually necessary to retain the activation values ​​of the forward calculation for subsequent reverse gradient calculations, which will cause the number of results that need to be saved to increase linearly with the increase in the number of model layers, thereby greatly increasing the memory pressure of the chip. Selective activation value recalculation directly deletes the activation value during the forward process and loss calculation, and recalculates the activation value when needed, thereby effectively shortening the life cycle of the activation value and alleviating memory pressure.

[0084] S230. Perform a first screening of each of the distributed training strategies according to the storage resources and cluster storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a first candidate distributed training strategy.

[0085] The cluster storage resource may be a storage resource that can be provided by the current cluster device when training the hybrid expert model. The first candidate distributed training strategy may be a distributed training strategy obtained by first screening each distributed training strategy according to the storage resource.

[0086] Optionally, based on storage resources, the storage resources occupied by each distributed training strategy when performing distributed training on the hybrid expert model can be compared with the cluster storage resources that can be provided by the current cluster device, and the first candidate distributed training strategy can be selected from each distributed training strategy according to the comparison result. It can be understood that the first candidate distributed training strategy can meet the storage resource requirements of the current cluster device when performing distributed training on the hybrid expert model.

[0087] In an optional embodiment of the present invention, the storage resources occupied by each distributed training strategy when the hybrid expert model is distributedly trained and the cluster storage resources are first screened to obtain a first candidate distributed training strategy, which may include: calculating the single-card available video memory of the cluster storage resources according to the single-card theoretical video memory and video memory utilization of the cluster storage resources; and performing a first screening of the single-card actual video memory of each distributed training strategy according to the single-card available video memory of the cluster storage resources to obtain the first candidate distributed training strategy.

[0088] The single card theoretical video memory may be the video memory resources that each training card in the current cluster device can theoretically provide, and the single card available video memory may be the video memory resources that each training card in the current cluster device can actually provide.

[0089] It is understandable that although the cluster device includes a large number of training cards, the video memory resources of each training card are limited, so the training tasks undertaken by a single card are also limited. Therefore, when the distributed training strategies are screened for the first time, the video memory of a single card can be used as the basis for screening. Specifically, the available video memory of a single card of the cluster storage resources can be first calculated based on the theoretical video memory of a single card of the cluster storage resources and the video memory utilization rate, so as to compare the actual video memory of a single card of each distributed training strategy with the available video memory of a single card of the cluster storage resources, and screen the distributed training strategy corresponding to the actual video memory of a single card that does not exceed the available video memory of a single card of the cluster storage resources as the first candidate distributed training strategy.

[0090] Specifically, the available video memory of a single card of the cluster storage resources can be the product of the theoretical video memory of a single card of the cluster storage resources and the video memory utilization, where the video memory utilization is a known quantity. The distributed training strategy can only be screened as the first candidate distributed training strategy if the actual video memory of a single card is less than or equal to the available video memory of a single card of the cluster storage resources. Exemplarily, the single GPU storage capacity of the current cluster device is obtained by calculation. When the actual video memory of a single card of the distributed training strategy exceeds the single GPU storage capacity, it is considered that the distributed training strategy does not meet the cluster resource allocation.

[0091] S240: Calculate the communication bandwidth requirement of each of the first candidate distributed training strategies according to the communication volume and computing volume of each of the first candidate distributed training strategies.

[0092] It is understandable that for cluster devices that train hybrid expert models, communication bandwidth is also one of the main factors that restrict whether the distributed training strategy can be successfully executed. Therefore, after completing the first round of screening of each distributed training strategy based on storage resources, the first candidate distributed training strategy obtained by screening can be screened for the second time based on the communication bandwidth. Before the second screening, the communication volume and calculation amount of each first candidate distributed training strategy can be calculated, so as to calculate the communication bandwidth requirements of each first candidate distributed training strategy based on the communication volume and calculation amount of the first candidate distributed training strategy.

[0093] In an optional embodiment of the present invention, calculating the communication bandwidth requirement of each of the first alternative distributed training strategies based on the communication volume and computing amount of each of the first alternative distributed training strategies may include: determining the expected communication computing ratio and the target unit communication volume of each of the first alternative distributed training strategies; calculating the target unit expected communication time of each of the first alternative distributed training strategies based on the expected communication computing ratio and the target unit computing time of each of the first alternative distributed training strategies; calculating the ratio between the target unit communication volume and the target unit expected communication time of each of the first alternative distributed training strategies to obtain the communication bandwidth requirement of each of the first alternative distributed training strategies.

[0094] Among them, the expected communication computing ratio can be the expected communication computing ratio configured before the distributed training of the hybrid expert model, which can be specifically the ratio of the expected communication time to the total data processing time, that is, the communication time overhead, and the smaller the value, the better. The communication computing ratio is an important metric that reflects the performance of parallel programs. It involves the comparison of two aspects: computing and communication. In parallel computing, computing refers to the time when the processor performs computing tasks, while communication involves the time when data is transmitted between processors. The communication computing ratio evaluates the efficiency of parallel programs by comparing the time of these two aspects. Specifically, this ratio can determine the relationship between the computing ratio and the communication ratio in the parallel computing process, thereby optimizing the program to improve the overall performance. The target unit communication volume can be the communication volume required by the corresponding distributed training strategy under the target unit data volume. The target unit computing time can be the time required to process the target unit data volume. The target unit expected communication time can be the communication time required to process the target unit data volume.

[0095] Specifically, before calculating the communication bandwidth requirements of each first alternative distributed training strategy, we can first determine the expected communication computing ratio and the target unit communication volume of each first alternative distributed training strategy, and further calculate the product value between the expected communication computing ratio and the target unit computing time of each first alternative distributed training strategy to obtain the target unit expected communication time of each first alternative distributed training strategy. Finally, we can calculate the ratio between the target unit communication volume and the target unit expected communication time of each first alternative distributed training strategy to obtain the communication bandwidth requirements of each first alternative distributed training strategy.

[0096] In a specific example, for the tensor parallel strategy, the communication bandwidth requirement of the tensor parallel strategy can be calculated by the expression: single microbatch tensor parallel communication volume / (single microbatch computing time*expected communication computing ratio). Among them, the single microbatch tensor parallel communication volume represents the target unit communication volume of the tensor parallel strategy, and the single microbatch computing time represents the target unit computing time of the tensor parallel strategy.

[0097] For the pipeline parallel strategy, the communication bandwidth requirement of the pipeline parallel strategy can be calculated by the expression: single microbatch pipeline parallel communication volume / (single microbatch computing time*expected communication computing ratio). Among them, the single microbatch pipeline parallel communication volume represents the target unit communication volume of the pipeline parallel strategy, and the single microbatch computing time represents the target unit computing time of the pipeline parallel strategy.

[0098] For data parallel strategies, the communication bandwidth requirement of data parallel strategies can be calculated by the expression: single iteration (iteration refers to the process of forward propagation, loss calculation, backpropagation and weight update of a small batch of data during training) data parallel communication volume / (single iteration computing time*expected communication computing ratio). Among them, single iteration data parallel communication volume represents the target unit communication volume of data parallel strategies, and single iteration computing time represents the target unit computing time of data parallel strategies.

[0099] For the context parallel strategy, the communication bandwidth requirement of the context parallel strategy can be calculated by the expression: single microbatch context parallel communication volume / (single microbatch computing time*expected communication computing ratio). Among them, the single microbatch context parallel communication volume represents the target unit communication volume of the context parallel strategy, and the single microbatch computing time represents the target unit computing time of the context parallel strategy.

[0100] For the expert parallel strategy, the communication bandwidth requirement of the expert parallel strategy can be calculated by the expression: single microbatch expert parallel communication volume / (single microbatch computing time*expected communication computing ratio). Among them, the single microbatch expert parallel communication volume represents the target unit communication volume of the expert parallel strategy, and the single microbatch computing time represents the target unit computing time of the expert parallel strategy. It should be noted that the communication bandwidth requirement of the sequence parallel strategy belongs to the communication bandwidth requirement of the tensor parallel strategy. That is, the communication bandwidth requirement of the tensor parallel strategy calculated by the above expression also includes the communication bandwidth requirement of the sequence parallel strategy.

[0101] S250: Perform a second screening on each of the first candidate distributed training strategies according to the communication bandwidth requirements and cluster bandwidth resources of each of the first candidate distributed training strategies to obtain a second candidate distributed training strategy.

[0102] The cluster bandwidth resource may be the available bandwidth resource that the current cluster device can provide. The second candidate distributed training strategy may be a distributed training strategy obtained by second screening the first candidate distributed training strategy based on the communication bandwidth resource. It can be understood that the second candidate distributed training strategy can satisfy both the storage resources of the current cluster device and the communication bandwidth resources of the current cluster device.

[0103] S260: Determine the target distributed training strategy according to the second candidate distributed training strategy.

[0104] In an embodiment of the present invention, after completing the first screening of each distributed training strategy, each first candidate distributed training strategy can be further screened for a second time based on the cluster bandwidth resources, thereby obtaining a second candidate distributed training strategy. Specifically, referring to the communication bandwidth requirements calculated by each first candidate distributed training strategy, the combination of distributed parallel strategies that exceeds the cluster bandwidth resources of the current cluster device can be excluded from the first candidate distributed training strategy, thereby screening out the second candidate distributed training strategy. Accordingly, after the second candidate distributed training strategy is screened out, the second candidate distributed training strategy can be further evaluated, and finally the target distributed training strategy can be screened out from the second candidate distributed training strategy.

[0105] In an optional embodiment of the present invention, determining the target distributed training strategy based on the second alternative distributed training strategies may include: calculating the training performance indicators of each of the second alternative distributed training strategies; wherein the training performance indicators may include at least one of throughput, latency and E2E (End to End, end-to-end communication protection) training duration; and screening the target distributed training strategy from each of the second alternative distributed training strategies based on the training performance indicators of each of the second alternative distributed training strategies.

[0106] Among them, the training performance indicator can be a relevant indicator for evaluating the training performance and effect of the distributed training strategy.

[0107] Specifically, when determining the target distributed training strategy based on the second candidate distributed training strategy, the training performance indicators such as throughput, latency, and E2E training duration of each second candidate distributed training strategy can be calculated. After obtaining the training performance indicators of each second candidate distributed training strategy, the training performance indicators of different second candidate distributed training strategies can be compared, and the second candidate distributed training strategy with the best training performance indicator can be selected as the target distributed training strategy, such as selecting the second candidate distributed training strategy with the smallest latency and / or the largest throughput as the target distributed training strategy. The screened target distributed training strategy can be the optimal distributed training strategy for the current cluster device.

[0108] In order to continuously optimize the distributed training effect of the hybrid expert model, if new model optimization technologies and distributed training strategies emerge during the training process, they can be output to the optimization strategy pool, that is, the contents of multiple distributed training strategies used for distributed training of the hybrid expert model are updated, and the above process is iteratively executed.

[0109] S270: Perform distributed training on the hybrid expert model according to a target distributed training strategy.

[0110] The above technical solution proposes a method for screening out the optimal distributed training strategy suitable for the distributed training of the hybrid expert model according to the cluster storage capacity, computing power, network topology and model structure, so as to perform distributed training on the hybrid expert model according to the screened optimal distributed training strategy. The entire method process can traverse different distributed training strategies according to the expert parameter size, communication bandwidth and storage limit, combine different optimization techniques, and evaluate the training performance indicators to select the optimal distributed training scheme and obtain the optimal distributed training strategy. At the same time, the distributed training strategy can also be iterated according to the model optimization technology and communication optimization strategy. The whole process can reasonably set different distributed parallel schemes according to different cluster network configurations and communication hardware architectures, which can be widely applied to different types of AI (Artificial Intelligence) training clusters, and can improve the distributed training performance of each AI training cluster for the hybrid expert model, thereby improving the hardware resource utilization and model execution efficiency during the distributed training of the hybrid expert model.

[0111] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0112] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0113] Embodiment 3

[0114] Figure 6 is a schematic diagram of a hybrid expert model distributed training device provided in Embodiment 3 of the present invention, such as Figure 6 As shown, the device includes: a distributed training strategy determination module 310, a storage resource calculation module 320 and a target distributed training strategy screening module 330, wherein:

[0115] A distributed training strategy determination module 310, used to determine a plurality of distributed training strategies for performing distributed training on the hybrid expert model;

[0116] A storage resource calculation module 320 is used to calculate the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model;

[0117] A target distributed training strategy screening module 330 is used to screen each of the distributed training strategies according to the storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a target distributed training strategy;

[0118] The target distributed training strategy is used to perform distributed training on the hybrid expert model.

[0119] The embodiment of the present invention determines multiple distributed training strategies for distributed training of the hybrid expert model, and calculates the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, thereby screening each distributed training strategy according to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributed trained, and obtains the target distributed training strategy for distributed training of the hybrid expert model. Accordingly, the screened target distributed training strategy can be used for distributed training of the hybrid expert model. The above technical scheme can solve the problem of low training performance when the hybrid expert model is trained using a distributed training method in the prior art, and can improve the distributed training performance of the hybrid expert model, thereby improving the hardware resource utilization and the execution efficiency of the model when the hybrid expert model is distributed trained.

[0120] Optionally, the target distributed training strategy screening module 330 is also used to: perform a first screening of each of the distributed training strategies according to the storage resources and cluster storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a first candidate distributed training strategy; calculate the communication bandwidth requirement of each of the first candidate distributed training strategies according to the communication volume and computing amount of each of the first candidate distributed training strategies; perform a second screening of each of the first candidate distributed training strategies according to the communication bandwidth requirement and cluster bandwidth resources of each of the first candidate distributed training strategies to obtain a second candidate distributed training strategy; determine the target distributed training strategy based on the second candidate distributed training strategy.

[0121] Optionally, the storage resource calculation module 320 is also used to: calculate the actual video memory of a single card occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model; the target distributed training strategy screening module 330 is also used to: calculate the single-card available video memory of the cluster storage resources based on the single-card theoretical video memory and video memory utilization of the cluster storage resources; perform a first screening of the single-card actual video memory of each of the distributed training strategies based on the single-card available video memory of the cluster storage resources to obtain the first alternative distributed training strategy.

[0122] Optionally, the target distributed training strategy screening module 330 is also used to: calculate the video memory occupied by the single card weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model; calculate the video memory occupied by the single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model; calculate the sum of the video memory occupied by the single card activation storage and the video memory occupied by the single card weight storage to obtain the actual video memory of the single card.

[0123] Optionally, the target distributed training strategy screening module 330 is also used to: calculate the single-card video memory occupied by the non-MoE module weight storage and the single-card video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization technologies; calculate the sum of the single-card video memory occupied by the non-MoE module weight storage and the single-card video memory occupied by the MoE module weight storage to obtain the video memory occupied by the single-card weight storage.

[0124] Optionally, the target distributed training strategy screening module 330 is also used to: determine the single-card weight association parameters adopted by the non-MoE module and the MoE module when each of the distributed training strategies performs distributed training on the hybrid expert model; wherein the single-card weight association parameters include the non-MoE module parameter amount and the MoE module parameter amount, and the single-card weight association parameters also include at least one of pipeline parallelism, tensor parallelism, sequence parallelism, expert parallelism and data parallelism; calculate the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization technologies according to the single-card weight association parameters.

[0125] Optionally, the target distributed training strategy screening module 330 is also used to: determine the single card activation associated parameters adopted by each of the distributed training strategies when performing distributed training on the hybrid expert model; wherein the single card activation associated parameters include microbatch, sequence length, hidden layer dimension, number of attention heads, number of single token selection experts and number of model layers, and the single card activation associated parameters also include at least one of tensor parallelism, sequence parallelism and expert parallelism; calculate the video memory occupied by single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques according to the single card activation associated parameters.

[0126] Optionally, the above-mentioned device also includes a training card number estimation module, which is used to: obtain cluster-associated resource parameters; wherein the cluster-associated resource parameters include cluster configuration parameters, training data sets, computing power utilization, model structure parameters and estimated training time; estimate the number of training cards required for distributed training of the hybrid expert model based on the cluster-associated resource parameters; wherein the number of training cards is used to calculate the values ​​of pipeline parallelism, tensor parallelism, expert parallelism, sequence parallelism, context parallelism and data parallelism.

[0127] Optionally, the target distributed training strategy screening module 330 is also used to: determine the expected communication computing ratio and target unit communication volume of each of the first alternative distributed training strategies; calculate the target unit expected communication time of each of the first alternative distributed training strategies based on the expected communication computing ratio and target unit computing time of each of the first alternative distributed training strategies; calculate the ratio between the target unit communication volume and the target unit expected communication time of each of the first alternative distributed training strategies to obtain the communication bandwidth requirement of each of the first alternative distributed training strategies.

[0128] Optionally, the target distributed training strategy screening module 330 is also used to: calculate the training performance indicators of each of the second candidate distributed training strategies; wherein the training performance indicators include at least one of throughput, latency and E2E training duration; and screen the target distributed training strategy from each of the second candidate distributed training strategies according to the training performance indicators of each of the second candidate distributed training strategies.

[0129] The above-mentioned hybrid expert model distributed training device can execute the hybrid expert model distributed training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the hybrid expert model distributed training method provided by any embodiment of the present invention.

[0130] Since the hybrid expert model distributed training device introduced above is a device that can execute the hybrid expert model distributed training method in the embodiment of the present invention, based on the hybrid expert model distributed training method introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the hybrid expert model distributed training device of the present embodiment and its various variations, so how the hybrid expert model distributed training device implements the hybrid expert model distributed training method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel of the field implement the device adopted by the hybrid expert model distributed training method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0131] Embodiment 4

[0132] Figure 7A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0133] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0134] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the hybrid expert model distributed training method.

[0136] Optionally, the distributed training method of the hybrid expert model may include: determining multiple distributed training strategies for distributed training of the hybrid expert model; calculating the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model; screening each of the distributed training strategies according to the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model to obtain a target distributed training strategy; wherein the target distributed training strategy is used for distributed training of the hybrid expert model.

[0137] In some embodiments, the hybrid expert model distributed training method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the hybrid expert model distributed training method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the hybrid expert model distributed training method in any other appropriate manner (e.g., by means of firmware).

[0138] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0140] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0143] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A hybrid expert model distributed training method, characterized in that: include: determining a plurality of distributed training strategies for distributed training of the mixture of experts model; Calculating the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model; According to the storage resources occupied by each distributed training strategy when the hybrid expert model is distributedly trained, each distributed training strategy is screened to obtain a target distributed training strategy; Wherein, the target distributed training strategy is used to perform distributed training on the hybrid expert model; The calculating of the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model includes: Calculate the actual video memory of a single card occupied when each of the distributed training strategies performs distributed training on the hybrid expert model; The calculating the actual video memory of a single card occupied when each of the distributed training strategies performs distributed training on the hybrid expert model includes: Calculate the video memory occupied by the single card weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model; Calculate the video memory occupied by the single card activated storage when each of the distributed training strategies performs distributed training on the hybrid expert model; Calculate the sum of the video memory occupied by the single card activation storage and the video memory occupied by the single card weight storage to obtain the actual video memory of the single card; The calculating the video memory occupied by the single card weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model includes: Calculate the single card video memory occupied by the non-MoE module weight storage and the single card video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques; Calculate the sum of the single card video memory occupied by the non-MoE module weight storage and the single card video memory occupied by the MoE module weight storage to obtain the video memory occupied by the single card weight storage; The calculating of the video memory occupied by the single card activated storage when each of the distributed training strategies performs distributed training on the hybrid expert model includes: Determine the single card activation associated parameters used by each of the distributed training strategies when performing distributed training on the hybrid expert model; wherein the single card activation associated parameters include microbatch, sequence length, hidden layer dimension, number of attention heads, number of single token selection experts, and number of model layers, and the single card activation associated parameters also include at least one of tensor parallelism, sequence parallelism, and expert parallelism; The video memory occupied by the single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model is calculated according to the single card activation associated parameters under different optimization technologies.

2. The method according to claim 1, characterized in that The method of screening each of the distributed training strategies according to the storage resources occupied when the hybrid expert model is distributedly trained to obtain a target distributed training strategy includes: According to the storage resources and cluster storage resources occupied by each distributed training strategy when the hybrid expert model is distributedly trained, each distributed training strategy is screened for the first time to obtain a first candidate distributed training strategy; Calculating the communication bandwidth requirement of each of the first candidate distributed training strategies according to the communication volume and the computational volume of each of the first candidate distributed training strategies; According to the communication bandwidth requirements and cluster bandwidth resources of each of the first candidate distributed training strategies, each of the first candidate distributed training strategies is screened for the second time to obtain a second candidate distributed training strategy; The target distributed training strategy is determined according to the second candidate distributed training strategy.

3. The method according to claim 2, characterized in that The first screening of each of the distributed training strategies according to the storage resources and cluster storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a first candidate distributed training strategy includes: Calculate the available video memory of a single card of the cluster storage resource according to the theoretical video memory of a single card and the video memory utilization rate of the cluster storage resource; The actual video memory of a single card of each of the distributed training strategies is screened for the first time according to the available video memory of a single card of the cluster storage resources to obtain the first candidate distributed training strategy.

4. The method according to claim 1, characterized in that The calculation of the video memory occupied by non-MoE module weight storage and the video memory occupied by MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques includes: Determine the single card weight association parameters used by the non-MoE module and the MoE module when each of the distributed training strategies performs distributed training on the hybrid expert model; wherein the single card weight association parameters include the non-MoE module parameter amount and the MoE module parameter amount, and the single card weight association parameters also include at least one of pipeline parallelism, tensor parallelism, sequence parallelism, expert parallelism, and data parallelism; According to the single card weight association parameter calculation under different optimization technologies, each of the distributed training strategies performs distributed training on the hybrid expert model, and the video memory occupied by the non-MoE module weight storage and the video memory occupied by the MoE module weight storage.

5. The method according to claim 1 or 4, characterized in that: Also includes: Obtain cluster-associated resource parameters; wherein the cluster-associated resource parameters include cluster configuration parameters, training data sets, computing power utilization, model structure parameters, and estimated training time; estimating the number of training cards required for distributed training of the hybrid expert model according to the cluster-associated resource parameters; The number of training cards is used to calculate the values ​​of pipeline parallelism, tensor parallelism, expert parallelism, sequence parallelism, context parallelism and data parallelism.

6. The method according to claim 2, characterized in that The calculating the communication bandwidth requirement of each of the first candidate distributed training strategies according to the communication volume and the computational volume of each of the first candidate distributed training strategies includes: Determine the expected communication computing ratio and the target unit communication volume of each of the first candidate distributed training strategies; Calculate the target unit expected communication time of each of the first candidate distributed training strategies according to the expected communication calculation ratio and the target unit calculation time of each of the first candidate distributed training strategies; The ratio between the target unit communication volume and the target unit expected communication time of each of the first candidate distributed training strategies is calculated to obtain the communication bandwidth requirement of each of the first candidate distributed training strategies.

7. The method according to claim 2, characterized in that: The determining the target distributed training strategy according to the second candidate distributed training strategy includes: Calculate the training performance index of each of the second candidate distributed training strategies; wherein the training performance index includes at least one of throughput, latency, and E2E training duration; The target distributed training strategy is selected from each of the second candidate distributed training strategies according to the training performance indicators of each of the second candidate distributed training strategies.

8. A hybrid expert model distributed training device, characterized in that: include: A distributed training strategy determination module, used to determine a plurality of distributed training strategies for performing distributed training on the hybrid expert model; A storage resource calculation module, used to calculate the storage resources occupied by each of the distributed training strategies when performing distributed training on the hybrid expert model; A target distributed training strategy screening module is used to screen each of the distributed training strategies according to the storage resources occupied when the hybrid expert model is distributedly trained by each of the distributed training strategies to obtain a target distributed training strategy; Wherein, the target distributed training strategy is used to perform distributed training on the hybrid expert model; The storage resource calculation module is also used to: calculate the actual video memory of a single card occupied when each of the distributed training strategies performs distributed training on the hybrid expert model; The target distributed training strategy screening module is also used to: calculate the video memory occupied by the single card weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model; calculate the video memory occupied by the single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model; calculate the sum of the video memory occupied by the single card activation storage and the video memory occupied by the single card weight storage to obtain the actual video memory of the single card; The target distributed training strategy screening module is also used to: calculate the single card video memory occupied by the non-MoE module weight storage and the single card video memory occupied by the MoE module weight storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization technologies; calculate the sum of the single card video memory occupied by the non-MoE module weight storage and the single card video memory occupied by the MoE module weight storage, and obtain the video memory occupied by the single card weight storage; The target distributed training strategy screening module is also used to: determine the single card activation associated parameters used by each of the distributed training strategies when performing distributed training on the hybrid expert model; wherein the single card activation associated parameters include microbatch, sequence length, hidden layer dimension, number of attention heads, number of single token selection experts, and number of model layers, and the single card activation associated parameters also include at least one of tensor parallelism, sequence parallelism, and expert parallelism; calculate the video memory occupied by single card activation storage when each of the distributed training strategies performs distributed training on the hybrid expert model under different optimization techniques according to the single card activation associated parameters.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the hybrid expert model distributed training method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hybrid expert model distributed training method described in any one of claims 1-7 when executed.

11. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the hybrid expert model distributed training method described in any one of claims 1-7 is implemented.

Images