A training method and device of a hybrid expert model, equipment and medium

Abstract

The present application relates to the technical field of large language model, in particular to a training method and device of mixed expert model, equipment and medium. In each iteration process, the local node generates original activation value for activating the sub-network on the remote node group, the present application compresses the original activation value to obtain compressed activation value, and then the local node sends the compressed activation value to the remote node group. After each iteration, the local node generates model original gradient and compresses the gradient to obtain compressed gradient, and then sends the compressed gradient to the remote node group. The remote node group continues to iteratively train the mixed expert model based on the compressed activation value and the compressed gradient. The present application communicates the compressed gradient and the compressed activation value between the local node and the remote node group, rather than the model original gradient and the original activation value. The compressed gradient and the compressed activation value reduce the number of communications compared to the model original gradient and the original activation value, thereby improving the training speed of the mixed expert model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a training method and device of hybrid expert model, equipment and medium. BACKGROUND

[0002] A plurality of sub-networks (sub-networks are expert models) are respectively deployed on different nodes, and the expert models on different nodes constitute a hybrid expert model. The task is divided into a plurality of sub-tasks, and the expert model on each node processes a sub-task, so that the expert models on a plurality of nodes interact to complete the processing of the task. When training the hybrid expert model, the plurality of nodes need to communicate with each other to transmit gradients and activation values. The prior art uses All-Reduce communication to synchronize gradients between nodes and uses All-to-All communication to transmit activation values between nodes. Because All-Reduce communication and All-to-All communication expand the data to be communicated between nodes to the order of trillions, directly using All-Reduce communication and All-to-All communication will face a communication bottleneck, thereby reducing the training speed of the hybrid expert model.

[0003] In summary, the prior art reduces the training speed of the hybrid expert model.

[0004] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0005] To solve the above technical problems, the present application provides a training method, device, equipment and medium of hybrid expert model, which solves the problem that the prior art reduces the training speed of the hybrid expert model.

[0006] To achieve the above purpose, the present application adopts the following technical scheme:

[0007] In a first aspect, the present application provides a training method of hybrid expert model, comprising:

[0008] When iteratively training the hybrid expert model, the original gradient of the model on the local node is obtained, and the original gradient is compressed to obtain a compressed gradient;

[0009] When iteratively training the hybrid expert model, the original activation value generated on the local node is obtained, and the original activation value is compressed to obtain a compressed activation value;

[0010] The compressed activation value is sent to a remote node group, and the compressed gradient is sent to the remote node group, and the remote node group is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0011] In one implementation, during iterative training of the hybrid expert model, the original gradients of the model at local nodes are obtained, and the original gradients are compressed to obtain compressed gradients, including:

[0012] When training the hybrid expert model iteratively at a local node, the original gradient of the model generated by the dense model in the hybrid expert model and the original gradient of the model generated by the sparse expert network in the hybrid expert model are obtained.

[0013] The original gradients of the dense model and the original gradients of the sparse expert network are compressed respectively to obtain their respective compressed gradients.

[0014] In one implementation, compressing the original gradient of the model to obtain the compressed gradient includes:

[0015] Obtain a preset compression threshold, which is used to characterize the gradient compression ratio;

[0016] The original gradient of the model is compressed based on the compression threshold to obtain the compressed gradient.

[0017] In one implementation, compressing the original gradient of the model based on the compression threshold to obtain a compressed gradient includes:

[0018] In the previous training iteration, based on the compression threshold, the elements of the original gradient of the model were sparsified to initially compress the original gradient of the model.

[0019] Based on the values ​​of the sparsed elements in the previous training iteration and the values ​​of the sparsed elements in the current training iteration, the original gradient of the model after the initial compression is compressed a second time to obtain the compressed gradient.

[0020] In one implementation, compressing the original activation value to obtain a compressed activation value includes:

[0021] Determine the maximum activation value among the original activation values;

[0022] Determine the maximum value that can be stored in FP8 data format;

[0023] The scaling factor is determined based on the maximum activation value and the maximum value.

[0024] The original activation value is compressed based on a scaling factor to compress it into a compressed activation value in the FP8 data format.

[0025] In one implementation, distributing the compressed activation value to a group of remote nodes includes:

[0026] The compressed activation value is distributed to the remote node group based on the CUDA multi-stream concurrency mechanism.

[0027] In one implementation, the trained hybrid expert model is used for text classification or text-to-language translation.

[0028] Secondly, embodiments of the present invention also provide a training apparatus for a hybrid expert model, wherein the apparatus comprises the following components:

[0029] The first compression module is used to obtain the original activation values ​​generated on the local node and compress the original activation values ​​to obtain compressed activation values ​​when iteratively training the hybrid expert model.

[0030] The second compression module is used to obtain the original gradient of the model on the local node and compress the original gradient of the model to obtain the compressed gradient when iteratively training the hybrid expert model.

[0031] The sending module is used to send the compressed activation value to the remote node group and the compressed gradient to the remote node group, the remote node group being used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0032] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a training program for a hybrid expert model stored in the memory and executable on the processor, wherein when the processor executes the training program for the hybrid expert model, it implements the steps of the hybrid expert model training method described above.

[0033] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a training program for a hybrid expert model, wherein when the training program for the hybrid expert model is executed by a processor, the steps of the training method for the hybrid expert model described above are implemented.

[0034] Beneficial Effects: During iterative training of the hybrid expert model, in each iteration, the local node generates the original activation values ​​used to activate the subnetworks on the remote node group. This invention compresses the original activation values ​​to obtain compressed activation values, which are then sent by the local node to the remote node group. After each iteration, the local node generates the original model gradient. This invention compresses the original model gradient to obtain compressed gradients, which are then sent to the remote node group. The remote node group continues iteratively training the hybrid expert model based on the compressed activation values ​​and compressed gradients. From the above analysis, it can be seen that this invention communicates compressed gradients and compressed activation values ​​between the local node and the remote node group, rather than the original model gradients and original activation values. Compressed gradients and compressed activation values ​​reduce the amount of communication compared to the original model gradients and original activation values, thereby improving the training speed of the hybrid expert model. Attached Figure Description

[0035] Figure 1 This is an overall flowchart of the present invention;

[0036] Figure 2 This is a schematic diagram of the CUDA multi-stream concurrency mechanism in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram showing the time consumption comparison in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram showing the comparison of loss values ​​in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram showing the comparison of confusion levels in an embodiment of the present invention;

[0040] Figure 6 A structural diagram of the training device for the hybrid expert model provided by the present invention;

[0041] Figure 7 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0043] Research has shown that deploying multiple sub-networks (i.e., expert models) on different nodes creates a hybrid expert model. This breaks down the task into multiple sub-tasks, with each node's expert model handling one sub-task, allowing the expert models on multiple nodes to interact and complete the task. During training, the nodes need to communicate with each other to transmit gradients and activation values. Existing technologies use All-Reduce communication (also known as ensemble communication) to synchronize gradients between nodes and All-to-All communication (a many-to-many communication mode) to transmit activation values. However, since All-Reduce and All-to-All communication expand the data to be communicated between nodes to trillions, directly using these methods encounters communication bottlenecks, thus slowing down the training speed of the hybrid expert model.

[0044] To address the aforementioned technical problems, this invention provides a training method, apparatus, device, and medium for hybrid expert models, thereby resolving the issue of reduced training speed of hybrid expert models in existing technologies.

[0045] For example, a hybrid expert architecture breaks down a hybrid expert model into multiple sub-networks, which are sparse expert networks. These sub-networks are deployed on different GPUs (GPUs are nodes). In addition to the sub-networks, each GPU also has a dense model to assist the sparse expert network in processing data. The dense model is identical on all GPUs. The sub-networks on each GPU are trained jointly to train the entire hybrid expert model.

[0046] For example, a hybrid expert model can be split into four sub-networks: sub-network A, sub-network B, sub-network C, and sub-network D. These four sub-networks are distributed across eight GPUs: GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8. Sub-network A is deployed on GPU1 and GPU2, sub-network B on GPU3 and GPU4, sub-network C on GPU5 and GPU6, and sub-network D on GPU7 and GPU8. The same dense model is deployed on all eight GPUs. When GPU1 acts as a local node, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and GPU8 form a remote node group. During iterative training of the hybrid expert model, GPU1 calculates the original gradient of sub-network A and the original gradient of the dense model in the current iteration. GPU1 also generates original activation values ​​based on the output of the dense model, using these original activation values ​​to indicate which sub-networks continue to process the output of the dense model. During the iteration, GPU1 compresses the original activation values. If subnetwork B continues to process the output of subnetwork A, GPU1 sends the compressed original activation values ​​to GPU3 and GPU4. After each iteration, GPU1 compresses the original gradients of subnetwork A and the original gradients of the dense model to obtain the compressed gradients of subnetwork A and the dense model. Then, GPU1 shares the compressed gradients of subnetwork A to the remote GPU2 and shares the compressed gradients of the dense model to all the remaining GPUs. The remote GPU2 updates the parameters of subnetwork A by combining the gradients generated by its own subnetwork A during training with the compressed gradients of subnetwork A shared by GPU1. The remote GPU2 updates the parameters of the dense model by combining the gradients generated by its own dense model during training with the compressed gradients of the dense model shared by other GPUs (i.e., GPU nodes other than GPU2).

[0047] GPU1 sends the compressed activation value to the remote GPU that needs to continue processing the output of the dense model based on the processing result of the gated network. When the GPU receives the compressed activation value, the sub-network on the GPU will continue to process based on the output result sent by GPU1, so as to train the sub-network on the GPU based on the output result. For example, if the GPU is GPU3, then GPU3 will train sub-network B based on the output result sent by GPU1. When GPU3 calculates the gradient of sub-network B, it will take into account the gradient of sub-network B on GPU4, so as to update the model parameters of sub-network B through multiple gradients.

[0048] The training method of the hybrid expert model in this embodiment can be applied to terminal devices, which can be terminal products with data processing capabilities, such as computers. In this embodiment, as... Figure 1 As shown, the training method for the hybrid expert model specifically includes the following steps:

[0049] S100, When iteratively training the hybrid expert model, obtain the original activation values ​​generated on the local node, and compress the original activation values ​​to obtain compressed activation values;

[0050] S200, when iteratively training the hybrid expert model, obtain the original gradient of the model on the local node, and compress the original gradient of the model to obtain the compressed gradient;

[0051] S300, the compressed activation value is sent to the remote node group, and the compressed gradient is sent to the remote node group, which is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0052] In this embodiment, the hybrid expert model consists of different sparse expert networks distributed across several GPUs. Dense models are also deployed on the GPUs where sparse expert networks are deployed. The same sparse expert network can be deployed on multiple GPUs. During training of the hybrid expert model, multiple GPUs with the same sparse expert network share the gradients of that sparse expert network, and all GPUs share the gradients of the dense model. The data interaction between the dense model and the sparse expert network is a prior art technique.

[0053] The original activation values ​​represent the output of the previous dense model. These original activation values ​​serve as the input to the next sparse expert network. The generation of the original activation values ​​and the generation of the model's original gradient are existing techniques. Suppose GPU1 has a sparse expert network A, and GPU3 has another sparse expert network B that continues to process the output of the dense model on GPU1. Both GPU1 and GPU3 are deployed with the same dense model. The dense model on GPU1 generates original activation values ​​based on the original data (which could be text data used for classification). GPU1 compresses the original activation values ​​and sends them to GPU3 as input to sparse expert network B. The output of sparse expert network B is returned to GPU1, and GPU3 calculates the gradient of sparse expert network B based on its output.

[0054] The hybrid expert model trained based on steps S100, S200, and S300 can be used for text classification. The specific process is as follows:

[0055] The dense model generates activation values ​​based on the input text data. The gating network (existing technology) selects a sub-network (which is a sub-network that constitutes the hybrid expert model, i.e., the sparse expert network) based on the activation values. The selected sub-networks process the text data in parallel to achieve parallel processing of text by each sub-network (i.e., data parallelism). Finally, the classification result of the text data is obtained based on the output results of each sub-network. For example, one sub-network outputs the probability that the text data belongs to the natural sciences, and another sub-network outputs the probability that the text data belongs to the humanities. Finally, the classification result of the text data is determined based on the output results of the two sub-networks.

[0056] In one embodiment, before implementing the compression algorithm in step S200, the hybrid expert model already possesses a certain degree of stability. That is, before implementing the gradient compression algorithm, the hybrid expert model has undergone pre-training, giving it a certain degree of convergence. Before implementing the compression algorithm, the following technical effects are achieved:

[0057] In the early stages of training a hybrid expert model, the model parameters are far from the optimal solution. Therefore, the loss function is very sensitive to changes in the parameters, resulting in gradient values ​​that are usually large and have a wide range. Therefore, compressing the gradient after warming up the model can allow the model training to enter a relatively stable state, thereby avoiding the initial instability caused by compression and ultimately improving the convergence of training and the performance of the model.

[0058] The specific steps of step S100 include S101, S102, S103, and S104:

[0059] S101, determine the maximum activation value among the original activation values.

[0060] The original activation value consists of several numerical values, each of which represents whether the model on the corresponding remote node is activated. The maximum activation value is the maximum value among these numerical values.

[0061] S102, determine the maximum value that can be stored in FP8 data format.

[0062] The FP8 data format includes E4M3 and E5M2. E4M3 (4-digit exponent, 3-digit mantissa) can store values ​​in the range [-448, +448], while E5M2 (5-digit exponent, 2-digit mantissa) provides a higher dynamic range, capable of storing values ​​up to [-57344, +57344]. The maximum value that the FP8 data format can store is either the maximum value in [-448, +448] or the maximum value in [-57344, +57344].

[0063] S103, determine the scaling factor based on the maximum activation value and the maximum value.

[0064] In this embodiment, the scaling factor is the maximum activation value divided by the maximum value.

[0065] S104, compress the original activation value based on the scaling factor to compress the original activation value into a compressed activation value in the FP8 data format.

[0066] The original activation values ​​contain the values ​​used to activate each sparse expert network. Each value is divided by a scaling factor and then converted to FP8 format (FP8 format represents the data format of half-precision floating point) to compress the original activation values.

[0067] This embodiment optimally scales the original activation values ​​to a range that the FP8 format can effectively represent by dividing them into blocks, thereby preserving the original information to the greatest extent. During dequantization, simply multiplying the FP8 tensor by the corresponding scaling factor of the block restores its approximate high-precision value. Dequantization occurs when a remote node receives a compressed activation value and dequantizes the compressed activation value to restore the original activation value. Blocking involves dividing the values ​​contained in the original activation value into smaller, contiguous data blocks.

[0068] The FP8 format in this embodiment has a significantly increased dynamic range, and its precision is non-uniform, with higher precision near zero and lower precision further away from zero. This characteristic aligns with the fact that parameters and activation values ​​in deep learning are typically distributed around zero. Due to its floating-point properties and larger dynamic range, FP8 can handle outliers commonly found in deep learning more naturally.

[0069] Step S200 includes the following specific steps: When the local node iteratively trains the hybrid expert model, it obtains the original gradient of the model generated by the dense model in the hybrid expert model and the original gradient of the model generated by the sparse expert network in the hybrid expert model; it compresses the original gradient of the model of the dense model and the original gradient of the model of the sparse expert network respectively to obtain their respective compressed gradients.

[0070] The same dense model is deployed on both local and remote nodes. The dense model is used to assist the sparse expert network in processing tasks assigned to the nodes where the sparse expert network (i.e., sub-network) resides. The dense model consists of an embedding layer, self-attention blocks, and a final output layer.

[0071] The parameters of a dense model are stored as complete copies on all the GPUs involved in training. After backpropagation at each training step, each GPU computes its local gradient for these shared parameters. To maintain consistency of the model parameters, the local gradients on all GPUs must be aggregated via an All-Reduce communication operation, and then the aggregated global gradient is used to update the model parameters of the dense model.

[0072] Sparse expert networks are feed-forward networks (FFNs) in the layers of traditional Transformers (Transformers are neural network models based on self-attention mechanisms).

[0073] The local nodes employ a compression method that combines local sparsification (MSTopK) and error feedback for both the original gradients of the dense model and the original gradients of the sparse expert network. The specific compression steps include:

[0074] S201, Obtain a preset compression threshold, which is used to characterize the gradient compression ratio.

[0075] Different compression thresholds are used for the gradients of dense models and sparse expert networks.

[0076] S202, In the previous iteration of training, based on the compression threshold, the elements of the original gradient of the model are sparsified to initially compress the original gradient of the model.

[0077] The gradient is a vector matrix, where each element corresponds to a parameter of the model (including dense models and sparse expert networks). During each training iteration, the gradient is compressed using sparsity. Sparsity compression means that the local node retains elements with values ​​greater than a compression threshold and sends them to the remote node, while discarding elements with values ​​less than or equal to the compression threshold. In other words, elements with values ​​less than or equal to the compression threshold are not sent from the local node to the remote node. By selectively discarding some elements in the gradient vector matrix through the compression threshold, sparsity compression of the model's original gradient is achieved.

[0078] S203, based on the values ​​of the sparsed elements in the previous training iteration and the values ​​of the sparsed elements in the current training iteration, perform a second compression on the original gradient of the model after the initial compression to obtain the compressed gradient.

[0079] The elements that were sparsified in the previous iteration (i.e., discarded elements) are used to perform secondary compression on the original gradient of the model during the current iteration of training to obtain the final compressed gradient. That is, the value of the element that was sparsified in the previous iteration is accumulated with the value of that element in the current iteration, and the accumulated result is compared with the compression threshold to determine whether to send the accumulated value of that element to the remote node.

[0080] Step S203 compresses gradients based on an error feedback mechanism, which has the following technical effects:

[0081] The error feedback mechanism introduces an error accumulation term. In each iteration, the gradient calculation of a node is no longer based solely on the current data, but incorporates the error information that was compressed in the previous iteration. Simultaneously, the node updates its local error accumulation term. Elements discarded in this compression are saved as new errors for error compensation in the next iteration. Even if some gradient elements are discarded in a particular iteration, the information they carry will gradually accumulate and be compensated for in subsequent iterations, ensuring that all gradient information ultimately impacts the model update.

[0082] The gradient compression method in this embodiment relies on a gradient compression algorithm based on statistical gradient distribution. Through progressive gradient compression, it can effectively reduce communication overhead and accelerate large-scale distributed deep learning training while ensuring model training stability and final performance. This avoids the possibility of difficulty in accurately capturing the true distribution of gradients when faced with drastically fluctuating gradients, which could lead to poor compression results or the introduction of too many errors.

[0083] The compression threshold mentioned above is a preset compression threshold, meaning it is set before training the model. The method for setting the compression threshold includes: estimating the mean and maximum values ​​of each element in the model's gradient, setting a normalized search interval, continuously generating candidate thresholds using a binary search method within a fixed number of iterations, and quickly counting the number of elements greater than or equal to the threshold using the efficient parallel reduction operation of the GPU. Based on the relationship between the counted number of elements and the target threshold, the search interval is dynamically shrunk to gradually approach the final compression threshold.

[0084] In this embodiment, the local node gradient compression in step S200, the local node activation value compression in step S100, and the decompression of the remote node are all implemented through the communication compression interface on the node. The communication compression interface includes a compression module, a communication module, and a decompression module. The compression module provides a unified interface for data compression, supports the implementation of different compression algorithms, and provides the `Compress()` method to perform data compression operations, where `Compress` represents a data compression method. The decompression module provides a unified interface for data decompression, corresponding to the compressor algorithm, and includes the `Decompress()` method to perform data decompression operations, where `Decompress` represents a data decompression method. The communication module manages distributed communication operations, integrates compression and decompression functions, provides the `AllReduce()` method to perform all-reduce communication operations for compressed gradients, and the `AllToAll()` method to perform all-to-all communication operations for compressed activation values. The MSTopK sparsity algorithm is applied to gradient all-reduce communication compression, retaining important gradient values, supporting dynamic adjustment of the sparsity rate, and balancing compression efficiency and model accuracy. The FP8 quantization algorithm is applied to all-to-all communication compression of activation values. It uses 8-bit floating-point numbers to represent the original 32-bit or 16-bit floating-point numbers and employs a dynamic scaling factor to maximize quantization accuracy.

[0085] Step S300 includes the following specific steps: The local node distributes the compressed activation value to the remote node based on the CUDA multi-stream concurrency mechanism, where CUDA stands for Unified Computing Device Architecture.

[0086] CUDA multi-stream concurrency is a powerful feature of modern GPU architectures, allowing multiple kernel tasks and memory transfer tasks to be launched and managed simultaneously on the same GPU device. Each CUDA stream has its own independent instruction queue, containing its own kernel startup and memory operation sequence. Because these streams are independent of each other, they can execute concurrently, meaning that the GPU's compute units and memory controller can process tasks from different streams simultaneously. In this way, multi-stream concurrency greatly improves the utilization of GPU compute units and memory bandwidth, thereby significantly increasing overall throughput.

[0087] In the pipeline scheduling of the MoE model (i.e., Hybrid Expert Model), the CUDA multi-stream concurrency mechanism can improve data communication efficiency. For example, when a batch of tokens (i.e., data units) is split and compressed, these compression operations can be assigned to a CUDA stream for execution. Once a portion of the data is compressed, this compressed data can initiate its all-to-all communication operation through another independent CUDA stream. Simultaneously, the first stream used for compression can immediately begin processing the compression task for the next batch of tokens. This design allows data compression (computation task) and data transmission (communication task) to be executed in parallel on the GPU. Figure 2 This is a schematic diagram of the CUDA multi-stream concurrency mechanism.

[0088] The following experiments demonstrate that the compression algorithm in this embodiment can improve communication efficiency while maintaining model performance:

[0089] The compressed training method of this embodiment is applied to the training shown in Table 1. The hybrid expert model uses standard byte-pair encoding for the input text before training. The preprocessing process generates a vocabulary by statistically analyzing high-frequency byte pairs in the text and gradually merging them, thereby achieving a balance between fully lexical segmentation and character-level segmentation.

[0090] The model training was performed on a computing cluster with 16 NVIDIA RTX A6000 GPUs. RTX stands for Ray Tracing, and Ray Tracing is a ray tracing technology. A baseline model without any optimization and a compressed model optimized using gradient compression plus activation value compression (the method of this invention) were trained separately, and the perplexity of the final language modeling was verified.

[0091] Table 1 Experimental Configuration

[0092]

[0093] The experiment was compared. The model's performance under different communication compression strategies is shown in the experimental results. Figure 3 As shown, Figure 3 This diagram illustrates the comparison of single-step time for different optimization strategies. By applying gradient compression and activation value compression algorithms, the single-step time of the model was reduced from 952.8 ms (ms means millisecond) to 644.3 ms, representing a performance improvement of 32.4%.

[0094] The classic hybrid expert language model is optimized using a hybrid expert model training method based on communication data compression. A full training iteration of 100,000 times was performed on the large text dataset OpenWebText. A comprehensive evaluation of the effectiveness of the hybrid expert model training optimization method based on communication data compression is crucial. Besides measuring its performance advantage in single-step training time, verifying its ability to guarantee model convergence and final accuracy in a complete end-to-end training task is also essential.

[0095] Experimental results are as follows Figure 4 and Figure 5 As shown, Figure 4 This is a diagram showing the comparison of loss values. Figure 5 This is a diagram illustrating the comparison of confusion levels, from... Figure 4 and Figure 5 As can be seen, the entire training process exhibits good stability. The training loss value steadily decreases with the increase of the number of iterations, while the perplexity on the validation set also continues to decrease steadily, eventually converging. This indicates that the gradient compression and activation value compression strategies adopted in this method do not introduce instability into the gradient flow and parameter updates of the model training, ensuring healthy training dynamics.

[0096] After the training, On the OpenWebText (a text dataset) validation set, the perplexity for language modeling reached 20.86, outperforming the baseline model's 21.72. These experimental results demonstrate that the hybrid expert model training optimization method based on communication data compression is not only an efficient training acceleration tool, but also a reliable and complete solution capable of producing high-performance models in practical applications.

[0097] In summary, this invention provides a communication compression optimization method for accelerating the training of large Mixture-of-Experts (MoE) models. The core contribution of this method lies in its departure from previous optimization approaches that focused solely on a single bottleneck. For the first time, it integrates specialized compression techniques for two types of heterogeneous communication data (gradients and activation values) into a unified framework, achieving collaborative optimization of the two major communication bottlenecks coexisting in MoE distributed training: All-Reduce and All-to-All. By employing the computationally efficient MSTopK approximation algorithm and contrast feedback and warm-up algorithms, this framework effectively solves the core challenge of traditional gradient sparsity. Simultaneously, by applying FP8 quantization technology and expert parallel pipeline scheduling, the bandwidth pressure for activation value exchange is significantly reduced. Experimental results demonstrate that this optimization method reduces the single-step training time of the MoE model by 32.4% on a 16-GPU cluster, achieving significant end-to-end acceleration. Crucially, this significant performance improvement does not compromise the model's convergence process or final accuracy.

[0098] In summary, the hybrid communication compression framework proposed in this invention is a practical solution that combines high efficiency and stability. It provides an effective technical path to address the communication challenges brought about by training ultra-large-scale MoE models, and has significant theoretical and engineering value for promoting the development of large language models towards larger scale and higher efficiency.

[0099] This embodiment also provides a training device for a hybrid expert model, such as... Figure 6 As shown, the device comprises the following components:

[0100] The first compression module 01 is used to obtain the original activation values ​​generated on the local node and compress the original activation values ​​to obtain compressed activation values ​​when iteratively training the hybrid expert model.

[0101] The second compression module 02 is used to obtain the original gradient of the model on the local node and compress the original gradient of the model to obtain the compressed gradient when iteratively training the hybrid expert.

[0102] The sending module 03 is used to send the compressed activation value to the remote node group and the compressed gradient to the remote node group. The remote node group is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0103] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 7 As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a hybrid expert model training method. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0104] Those skilled in the art will understand that Figure 7 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a training program for a hybrid expert model stored in the memory and executable on the processor. When the processor executes the training program for the hybrid expert model, it implements the following operation instructions:

[0106] When iteratively training a hybrid expert model, the original gradient of the model on the local node is obtained, and the original gradient of the model is compressed to obtain the compressed gradient.

[0107] When iteratively training the hybrid expert model, the original activation values ​​generated on the local node are obtained, and the original activation values ​​are compressed to obtain compressed activation values.

[0108] The compressed activation value is sent to a remote node group, and the compressed gradient is also sent to a remote node group. The remote node group is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a hybrid expert model, characterized in that, include: When iteratively training the hybrid expert model, the original activation values ​​generated on the local node are obtained, and the original activation values ​​are compressed to obtain compressed activation values. When iteratively training a hybrid expert model, the original gradient of the model on the local node is obtained, and the original gradient of the model is compressed to obtain the compressed gradient. The compressed activation value is sent to the remote node group, and the compressed gradient is sent to the remote node group. The remote node group is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient. When iteratively training a hybrid expert model, the original gradients of the model at local nodes are obtained, and the original gradients are compressed to obtain compressed gradients, including: When training the hybrid expert model iteratively at a local node, the original gradient of the model generated by the dense model in the hybrid expert model and the original gradient of the model generated by the sparse expert network in the hybrid expert model are obtained. The original gradients of the dense model and the sparse expert network are compressed separately to obtain their respective compressed gradients. Different compression thresholds are used for the original gradients of the dense model and the sparse expert network. Compressing the original activation value to obtain the compressed activation value includes: Determine the maximum activation value among the original activation values; Determine the maximum value that can be stored in FP8 data format; The scaling factor is determined based on the maximum activation value and the maximum value. The original activation value is compressed based on the scaling factor to compress the original activation value into the compressed activation value in the FP8 data format; The trained hybrid expert model is used for text classification, including: The dense model generates activation values ​​based on the input text data, selects sparse expert networks based on the activation values, and the selected sparse expert networks process the text data in parallel. Based on the output results of each sparse expert network, the classification result of the text data is obtained.

2. The training method for the hybrid expert model as described in claim 1, characterized in that, Compressing the original gradients of the model yields the compressed gradients, including: Obtain a preset compression threshold, which is used to characterize the gradient compression ratio; The original gradient of the model is compressed based on the compression threshold to obtain the compressed gradient.

3. The training method for the hybrid expert model as described in claim 2, characterized in that, The original gradient of the model is compressed based on the compression threshold to obtain the compressed gradient, including: In the previous training iteration, based on the compression threshold, the elements of the original gradient of the model were sparsified to initially compress the original gradient of the model. Based on the values ​​of the sparsed elements in the previous training iteration and the values ​​of the sparsed elements in the current training iteration, the original gradient of the model after the initial compression is compressed a second time to obtain the compressed gradient.

4. The training method for the hybrid expert model as described in any one of claims 1-3, characterized in that, Sending the compressed activation value to the remote node cluster includes: The compressed activation value is distributed to the remote node group based on the CUDA multi-stream concurrency mechanism.

5. A training device for a hybrid expert model, characterized in that, The device comprises the following components: The first compression module is used to obtain the original activation values ​​generated on the local node and compress the original activation values ​​to obtain compressed activation values ​​when iteratively training the hybrid expert model. The second compression module is used to obtain the original gradient of the model on the local node and compress the original gradient of the model to obtain the compressed gradient when iteratively training the hybrid expert model. The sending module is used to send the compressed activation value to the remote node group and the compressed gradient to the remote node group, the remote node group being used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient; When iteratively training a hybrid expert model, the original gradients of the model at local nodes are obtained, and the original gradients are compressed to obtain compressed gradients, including: When training the hybrid expert model iteratively at a local node, the original gradient of the model generated by the dense model in the hybrid expert model and the original gradient of the model generated by the sparse expert network in the hybrid expert model are obtained. The original gradients of the dense model and the sparse expert network are compressed separately to obtain their respective compressed gradients. Different compression thresholds are used for the original gradients of the dense model and the sparse expert network. Compressing the original activation value to obtain the compressed activation value includes: Determine the maximum activation value among the original activation values; Determine the maximum value that can be stored in FP8 data format; The scaling factor is determined based on the maximum activation value and the maximum value. The original activation value is compressed based on the scaling factor to compress the original activation value into the compressed activation value in the FP8 data format; The trained hybrid expert model is used for text classification, including: The dense model generates activation values ​​based on the input text data, selects sparse expert networks based on the activation values, and the selected sparse expert networks process the text data in parallel. Based on the output results of each sparse expert network, the classification result of the text data is obtained.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a training program for a hybrid expert model stored in the memory and executable on the processor. When the processor executes the training program for the hybrid expert model, it implements the steps of the training method for the hybrid expert model as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a training program for a hybrid expert model, which, when executed by a processor, implements the steps of the training method for a hybrid expert model as described in any one of claims 1-4.

Images