Training method and device of hybrid expert model, equipment and medium

Abstract

The invention relates to the technical field of large language models, in particular to a training method and device of a hybrid expert model, equipment and a medium. In each iteration process, a local node generates an original activation value for activating a sub-network on a remote node group, the original activation value is compressed to obtain a compressed activation value, and then the local node sends the compressed activation value to the remote node group. And after each iteration, the local node generates an original gradient of the model and compresses the gradient to obtain a compression gradient, the compression gradient is sent to the remote node group, and the remote node group continues to iteratively train the hybrid expert model based on the compression activation value and the compression gradient. According to the method, the compression gradient and the compression activation value are communicated between the local node and the remote node group instead of the original gradient and the original activation value of the model, and the communication quantity of the compression gradient and the compression activation value is reduced relative to the original gradient and the original activation value of the model, so that the training speed of the hybrid expert model is increased.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a training method and device of hybrid expert model, equipment and medium. BACKGROUND

[0002] A plurality of sub-networks (sub-networks are expert models) are respectively deployed on different nodes, and the expert models on different nodes constitute a hybrid expert model. The task is divided into a plurality of sub-tasks, and the expert model on each node processes a sub-task, so that the expert models on the plurality of nodes interact to complete the processing of the task. When training the hybrid expert model, the plurality of nodes need to communicate with each other to transmit gradients and activation values. The prior art uses All-Reduce communication to synchronize gradients between nodes and uses All-to-All communication to transmit activation values between nodes. Because All-Reduce communication and All-to-All communication expand the data to be communicated between nodes to the order of trillions, directly using All-Reduce communication and All-to-All communication will face a communication bottleneck, thereby reducing the training speed of the hybrid expert model.

[0003] In summary, the prior art reduces the training speed of the hybrid expert model.

[0004] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0005] To solve the above technical problems, the present application provides a training method, device, equipment and medium of hybrid expert model, which solves the problem that the prior art reduces the training speed of the hybrid expert model.

[0006] To achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a training method of hybrid expert model, comprising: When iteratively training the hybrid expert model, the model original gradient on the local node is obtained, and the model original gradient is compressed to obtain a compressed gradient; When iteratively training the hybrid expert model, the original activation value generated on the local node is obtained, and the original activation value is compressed to obtain a compressed activation value; The compressed activation value is sent to a remote node group, and the compressed gradient is sent to the remote node group, and the remote node group is used to continue iteratively training the hybrid expert model based on the compressed activation value and the compressed gradient.

[0007] In an implementation mode, when iteratively training the hybrid expert model, the model original gradient on the local node is obtained, and the model original gradient is compressed to obtain a compressed gradient, comprising: In the iteration training of the mixed expert model, the model original gradient generated by the dense model in the mixed expert model and the model original gradient generated by the sparse expert network in the mixed expert model are obtained. The model original gradient of the dense model and the model original gradient of the sparse expert network are compressed respectively to obtain respective compressed gradients.

[0008] In an implementation manner, the compression of the model original gradient to obtain the compressed gradient comprises: A preset compression threshold is obtained, and the compression threshold is used to represent a gradient compression ratio. The model original gradient is compressed based on the compression threshold to obtain a compressed gradient.

[0009] In an implementation manner, the compression of the model original gradient based on the compression threshold to obtain the compressed gradient comprises: In the last iteration training, the elements of the model original gradient are sparsified based on the compression threshold to preliminarily compress the model original gradient. According to the value of the sparsified element in the last iteration training and the value of the sparsified element in the current iteration training, the model original gradient after the preliminary compression is secondarily compressed to obtain a compressed gradient.

[0010] In an implementation manner, the compression of the original activation value to obtain the compressed activation value comprises: The maximum activation value in the original activation value is determined. The maximum value that can be stored in the FP8 data format is determined. The scaling factor is determined based on the maximum activation value and the maximum value. The original activation value is compressed based on the scaling factor to realize the compression of the original activation value into the compressed activation value in the FP8 data format.

[0011] In an implementation manner, the compressed activation value is distributed to the remote node group, comprising: The compressed activation value is distributed to the remote node group based on the CUDA multi-stream concurrent mechanism.

[0012] In an implementation manner, the mixed expert model after the training is used for text classification or text language translation.

[0013] In a second aspect, the embodiments of the present application further provide a training device of a mixed expert model, wherein the device comprises the following components: A first compression module is configured to obtain original activation values generated on a local node in the iteration training of the mixed expert model, and compress the original activation values to obtain compressed activation values. The second compression module is configured to obtain model original gradients on the local node during iterative training of the mixed expert model, and compress the model original gradients to obtain compressed gradients. The sending module is configured to send the compressed activation values to the remote node group and send the compressed gradients to the remote node group, and the remote node group is configured to continue iterative training of the mixed expert model based on the compressed activation values and the compressed gradients.

[0014] In a third aspect, an embodiment of the present application further provides a terminal device, wherein the terminal device comprises a memory, a processor, and a training program of a mixed expert model stored in the memory and executable on the processor, and the processor implements the steps of the training method of the mixed expert model when executing the training program of the mixed expert model.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a training program of a mixed expert model, and the steps of the training method of the mixed expert model are implemented when the training program of the mixed expert model is executed by a processor.

[0016] Beneficial effects: During iterative training of the mixed expert model, the local node generates original activation values for activating sub-networks on the remote node group in each iteration process. The original activation values are compressed to obtain compressed activation values, and then the local node sends the compressed activation values to the remote node group. After each iteration, the local node generates model original gradients, which are compressed to obtain compressed gradients, and then the compressed gradients are sent to the remote node group. The remote node group continues to iteratively train the mixed expert model based on the compressed activation values and the compressed gradients. From the above analysis, it can be seen that the compressed gradients and the compressed activation values are communicated between the local node and the remote node group, rather than the model original gradients and the original activation values. The compressed gradients and the compressed activation values reduce the number of communications compared with the model original gradients and the original activation values, thereby improving the training speed of the mixed expert model. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The overall flowchart of the present application is shown in the figure; Figure 2 The CUDA multi-stream concurrency mechanism in the embodiment of the present application is shown in the figure; Figure 3 The time-consuming comparison diagram in the embodiment of the present application is shown in the figure; Figure 4 The loss value comparison diagram in the embodiment of the present application is shown in the figure; Figure 5 The perplexity comparison diagram in the embodiment of the present application is shown in the figure; Figure 6A training device structure diagram of the mixed expert model is provided for the present application. Figure 7 An internal structure principle block diagram of the terminal device is provided for the embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the present application are described clearly and completely in combination with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] It is found through research that multiple sub-networks (the sub-networks are expert models) are respectively deployed on different nodes, the expert models on different nodes constitute a mixed expert model, a task is divided into multiple sub-tasks, and an expert model on each node processes a sub-task, so that the expert models on multiple nodes interact to complete the processing of the task. When training the mixed expert model, the multiple nodes need to communicate with each other to transmit gradients and activation values. The prior art uses All-Reduce communication (All-Reduce communication is set communication) to synchronize the gradients between the nodes and uses All-to-All communication (All-to-All communication is a many-to-many communication mode) to transmit the activation values between the nodes. Because the All-Reduce communication and the All-to-All communication expand the data to be communicated between the nodes to the order of trillions, directly using the All-Reduce communication and the All-to-All communication will face a communication bottleneck, thereby reducing the training speed of the mixed expert model.

[0020] To solve the above technical problems, the present application provides a mixed expert model training method, device, equipment and medium, which solves the problem of reducing the training speed of the mixed expert model in the prior art.

[0021] For example, the mixed expert architecture divides the mixed expert model into multiple different sub-networks, the sub-networks are sparse expert networks, multiple sub-networks are respectively deployed on different GPUs (GPU is a node), and in addition to the sub-networks, there are dense models on the GPUs for assisting the sparse expert networks in processing data. The dense models on all GPUs are the same. The sub-networks on each GPU are jointly trained to realize the training of the entire mixed expert model.

[0022] For example, the hybrid expert model can be split into four sub-networks: sub-network A, sub-network B, sub-network C and sub-network D. The four sub-networks are distributed on eight GPUs: GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7 and GPU8. Sub-network A is deployed on GPU1 and GPU2, sub-network B is deployed on GPU3 and GPU4, sub-network C is deployed on GPU5 and GPU6, and sub-network D is deployed on GPU7 and GPU8. The same dense model is deployed on the above eight GPUs. When GPU1 is the local node, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7 and GPU8 constitute the remote node group. When iteratively training the hybrid expert model, GPU1 calculates the model original gradient of sub-network A and the model original gradient of the dense model at the current iteration, and also generates original activation values based on the output result of the dense model, which represent which sub-networks continue to process the output result of the dense model. During the iteration process, GPU1 compresses the original activation values. If sub-network B continues to process the output result of sub-network A, GPU1 sends the compressed original activation values to GPU3 and GPU4. After each iteration, GPU1 compresses the model original gradient of sub-network A and the model original gradient of the dense model to obtain the compressed gradient of sub-network A and the compressed gradient of the dense model, and then GPU1 shares the compressed gradient of sub-network A to the remote GPU2 and shares the compressed gradient of the dense model to all the other GPUs. Remote GPU2 updates the parameters of sub-network A by combining the gradient of sub-network A generated during training and the compressed gradient of sub-network A shared by GPU1, and updates the parameters of the dense model by combining the gradient of the dense model generated during training and the compressed gradient of the dense model shared by the other GPUs (i.e., the GPUs other than GPU2).

[0023] GPU1 sends the compressed activation values to the remote GPUs that need to continue processing the output result of the dense model according to the processing result of the gating network. When the GPU receives the compressed activation values, the sub-network on the GPU continues to process based on the output result sent by GPU1 to train the sub-network on the GPU based on the output result. For example, if the GPU is GPU3, GPU3 trains sub-network B based on the output result sent by GPU1, and when GPU3 calculates the gradient of sub-network B, it considers the gradient of sub-network B on GPU4 to update the model parameters of sub-network B through multiple gradients.

[0024] The training method of the hybrid expert model of the embodiment can be applied to a terminal device, which can be a terminal product with data processing function, such as a computer, etc. In the embodiment, as shown in Figure 1As shown in the method for training the mixed expert model, the method specifically comprises the following steps: In step S100, when the mixed expert model is iteratively trained, original activation values generated on the local node are obtained, and the original activation values are compressed to obtain compressed activation values. In step S200, when the mixed expert model is iteratively trained, original model gradients on the local node are obtained, and the original model gradients are compressed to obtain compressed gradients. In step S300, the compressed activation values are sent to a remote node group, and the compressed gradients are sent to the remote node group, and the remote node group is used to continue iteratively training the mixed expert model based on the compressed activation values and the compressed gradients.

[0025] In this embodiment, the mixed expert model is composed of different sparse expert networks distributed on several GPUs, and the GPUs on which the sparse expert networks are deployed also have dense models deployed thereon. The same sparse expert network can be deployed on multiple GPUs. When the mixed expert model is trained, the multiple GPUs on which the same sparse expert network is deployed share the gradients of the sparse expert network with each other, and the gradients of the dense model are shared among all the GPUs. The data interaction between the dense model and the sparse expert network is prior art.

[0026] The original activation values represent the output of the previous dense model, and the original activation values are used as the input of the next sparse expert network. The generation of the original activation values and the generation of the original model gradients are prior art. Assuming that there is a sparse expert network A on GPU1 and another sparse expert network B on GPU3 for processing the output of the dense model on GPU1. GPU1 and GPU3 both have the same dense model. The dense model on GPU1 generates original activation values based on original data (the original data can be text data for processing classification), and the original activation values are compressed and then sent to GPU3 and used as the input of the sparse expert network B. The output of the sparse expert network B is returned to GPU1, and the gradient of the sparse expert network B is calculated based on the output of the sparse expert network B.

[0027] The mixed expert model trained based on steps S100, S200 and S300 can be used for text classification, and the specific process is as follows: The dense model generates activation values based on input text data, and a gating network (prior art) selects a sub-network (which is a sub-network constituting a hybrid expert model, i.e., a sparse expert network) according to the activation values. The selected sub-network processes the text data in parallel to realize parallel processing of the text by each sub-network (i.e., data parallelism). Finally, the classification result of the text data is obtained according to the output results of each sub-network. For example, one sub-network outputs a result that the text data belongs to natural science with a probability, and another sub-network outputs a result that the text data belongs to humanities with a probability. The classification result of the text data is finally determined according to the output results of the two sub-networks.

[0028] In an embodiment, before implementing the compression algorithm in step S200, the hybrid expert model already has a certain stability, that is, before implementing the gradient compression algorithm, the hybrid expert model has been preheated and trained to have a certain convergence. Before implementing the compression algorithm, the following technical effects are achieved: In the initial stage of training the hybrid expert model, the model parameters are far from the optimal solution, so the loss function is very sensitive to the change of the parameters, resulting in generally large and wide range of gradient values. Therefore, after preheating and training the model, the gradient is compressed, which can make the model training enter a relatively stable state, thereby avoiding the initial instability caused by compression and ultimately improving the convergence of training and the performance of the model.

[0029] The specific steps of step S100 include S101, S102, S103, and S104: S101, determining the maximum activation value in the original activation value.

[0030] The original activation value includes a plurality of numerical values, each of which is used to represent whether the model on the corresponding remote node is activated. The maximum activation value is the maximum value in the plurality of numerical values.

[0031] S102, determining the maximum value that can be stored in the FP8 data format.

[0032] The FP8 data format includes E4M3 and E5M2. E4M3 (4-bit exponent, 3-bit mantissa) can store values in the range [-448, +448], and E5M2 (5-bit exponent, 2-bit mantissa) provides a higher dynamic range and can store values up to [-57344, +57344]. The maximum value that can be stored in the FP8 data format is the maximum value in [-448, +448] or the maximum value in [-57344, +57344].

[0033] S103, determining a scaling factor based on the maximum activation value and the maximum value.

[0034] The maximum activation value of the embodiment divided by the maximum value is the scaling factor.

[0035] In S104, the original activation value is compressed based on the scaling factor to achieve compressed activation values of the original activation value compressed into the FP8 data format.

[0036] The original activation value contains numerical values for activating individual sparse expert networks, and after each numerical value is divided by the scaling factor, it is converted into the FP8 format (FP8 format represents the data format of half-precision floating-point numbers) to compress the original activation value.

[0037] The embodiment optimally scales the numerical values of the original activation value into the range that can be effectively represented by the FP8 format in a block manner, thereby maximizing the preservation of original information. When dequantizing, only the FP8 tensor multiplied by the scaling factor corresponding to the block is required to recover its approximate high-precision value. Dequantization is when the remote node receives the compressed activation value, and the compressed activation value is dequantized to restore the original activation value. Block means that each numerical value contained in the original activation value is divided into smaller, continuous data blocks.

[0038] The dynamic range of the FP8 format of the embodiment is significantly increased, and the precision of FP8 is non-uniform, with higher precision near zero and lower precision away from zero. This feature is consistent with the characteristics that parameters and activation values in deep learning are usually distributed near zero. Due to its floating-point characteristics and larger dynamic range, FP8 can more naturally handle outliers common in deep learning.

[0039] The step S200 includes the following specific steps: when the local node iteratively trains the hybrid expert model, obtaining the model original gradient generated by the dense model in the hybrid expert model and the model original gradient generated by the sparse expert network in the hybrid expert model; respectively compressing the model original gradient of the dense model and the model original gradient of the sparse expert network to obtain respective compressed gradients.

[0040] The same dense model is deployed on the local node and the remote node, and the dense model is used to assist the sparse expert network in processing tasks allocated to the node where the sparse expert network (i.e., the subnetwork) is located. The dense model includes an embedding layer (Embedding Layer), a self-attention module (Self-Attention Blocks), and a final output layer.

[0041] The parameters of the dense model have complete copies on all GPUs participating in training. After backpropagation at each training step, each GPU calculates the local gradient for the shared parameters. To maintain consistency of the model parameters, the local gradients on all GPUs must be aggregated through an All-Reduce communication operation, and then the global gradient after aggregation is used to update the model parameters of the dense model.

[0042] The sparse expert network is a feed-forward network (FFN) in a traditional Transformer layer.

[0043] The local node adopts a compression mode in which local sparsification (MSTopK) and error feedback are coordinated for both the model original gradient of the dense model and the model original gradient of the sparse expert network. The specific compression steps include: S201, a preset compression threshold is obtained, the compression threshold being used to represent a gradient compression ratio.

[0044] Different compression thresholds are used for the gradient of the dense model and the gradient of the sparse expert network.

[0045] S202, in the last iteration training, based on the compression threshold, the elements of the model original gradient are sparsified to preliminarily compress the model original gradient.

[0046] The gradient is a vector matrix, each element in the vector matrix corresponding to a parameter of the model (including the dense model and the sparse expert network). In each iteration training, the gradient is compressed in a sparsified manner, that is, the local node retains the elements with values greater than the compression threshold to send to the remote node, and discards the elements with values less than or equal to the compression threshold, that is, the elements with values less than or equal to the compression threshold will not be sent to the remote node by the local node. By selectively discarding part of the elements in the gradient vector matrix through the compression threshold, the model original gradient is sparsified and compressed.

[0047] S203, according to the value of the sparsified element in the last iteration training and the value of the sparsified element in the current iteration training, the model original gradient after preliminary compression is compressed again to obtain a compressed gradient.

[0048] The sparsified element (i.e. discarded element) in the last iteration is used to compress the model original gradient again in the current iteration training to obtain a final compressed gradient. That is, the value of the sparsified element in the last iteration and the value of the element in the current iteration are added, and the added result is compared with the compression threshold to determine whether the added value of the element is sent to the remote node.

[0049] Step S203 is to compress the gradient based on the error feedback mechanism, which has the following technical effects: The error feedback mechanism introduces an error accumulation term. In each iteration, the gradient calculation process of the node is no longer based on the current data alone, but combines the error information discarded in the last iteration. At the same time, the node updates its local error accumulation term. Those elements discarded in this compression will be saved as new error for error compensation in the next iteration. Even if some gradient elements are discarded in an iteration, the information they carry will be gradually accumulated and compensated in subsequent iterations, ensuring that all gradient information eventually has an impact on model updates.

[0050] The gradient compression method of the embodiment relies on the gradient compression algorithm that statistically distributes the gradient. Through gradual gradient compression, it can effectively reduce communication overhead and accelerate large-scale distributed deep learning training while ensuring model training stability and final performance, to avoid the difficulty in accurately capturing the true distribution of the gradient when facing dramatic fluctuations, which may lead to poor compression effect or introduce too much error.

[0051] The compression threshold is a preset compression threshold, that is, the compression threshold has been set before training the model. The setting method of the compression threshold includes: estimating the mean and maximum of each element in the gradient of the model, and setting a normalized search interval. Within a fixed number of iterations, candidate threshold values are generated continuously by bisection method, and the number of elements greater than or equal to the threshold value is quickly counted by using the efficient parallel reduction operation of GPU. According to the relationship between the counted number of elements and the target threshold value, the search interval is dynamically shrunk to gradually approach the final compression threshold.

[0052] The local node gradient compression of step S200 and the local node activation value compression of step S100 and the decompression of the remote node of the present embodiment are all realized through a communication compression interface on the node. The communication compression interface contains a compression module, a communication module and a decompression module. The compression module provides a unified interface of data compression function, supports the implementation of different compression algorithms, and provides a Compress() method to perform data compression operation, where Compress represents a data compression method. The decompression module provides a unified interface of data decompression function, corresponds to the compressor algorithm, contains a Decompress() method to perform data decompression operation, where Decompress represents a data decompression method. The communication module is responsible for managing distributed communication operations, integrating compression and decompression functions, providing an AllReduce() method to perform compressed gradient All-Reduce pass operation, and an AllToAll() method to perform compressed activation value All-to-All communication operation. The MSTopK sparsification algorithm is applied to the gradient All-Reduce communication compression, retains important gradient values, supports dynamic adjustment of sparsity, and balances compression efficiency and model accuracy. The FP8 quantization algorithm is applied to the activation value All-to-All communication compression, uses 8-bit floating point numbers to represent original 32-bit or 16-bit floating point numbers, and uses a dynamic scaling factor to maximize quantization precision.

[0053] Step S300 includes the following specific steps: the local node distributes the compressed activation values to the remote nodes based on the CUDA multi-stream concurrent mechanism, where CUDA represents Compute Unified Device Architecture.

[0054] The CUDA multi-stream concurrent mechanism is a powerful feature in modern GPU architecture, which allows multiple kernel tasks and memory transfer tasks to be started and managed simultaneously on the same GPU device. Each CUDA stream has its own instruction queue, which contains its own kernel launch and memory operation sequence. Since these streams are independent of each other, they can be executed in a concurrent manner, which means that the computing units and memory controllers of the GPU can handle tasks from different streams simultaneously. In this way, multi-stream concurrency greatly improves the utilization of GPU computing units and memory bandwidth, thereby significantly improving overall throughput.

[0055] In the pipeline scheduling of MoE model (i.e., mixture-of-experts model), the data communication efficiency can be improved by using the CUDA multi-stream concurrency mechanism. For example, when a batch of Tokens (Token is a data unit) is split and compressed, these compression operations can be assigned to a CUDA stream for execution. Once the partial data is compressed, the compressed data can start its All-to-All communication operation through another independent CUDA stream. At the same time, the first stream for compression can immediately start processing the compression task of the next batch of Tokens. This design enables the data compression (computing task) and data transfer (communication task) to be executed in parallel on the GPU. Figure 2 An illustration of the CUDA multi-stream concurrency mechanism.

[0056] The following experiments demonstrate that the compression algorithm of the embodiment can improve the communication efficiency while maintaining the model performance: The compression training method of the embodiment is applied to train the mixture-of-experts model as shown in Table 1. Before training, the input text has undergone standard byte pair encoding preprocessing procedure. By counting the high-frequency byte pairs in the text, the word table is gradually merged to balance between full-vocabulary segmentation and character-level segmentation.

[0057] The model training part is performed on a computing cluster with 16 Nvidia RTX A6000 GPUs. RTX stands for Ray Tracing, a ray tracing technology. The baseline model without any optimization and the compressed model optimized by gradient compression and activation value compression method (the method of the present application) are trained respectively, and the perplexity of the final language modeling of the model is verified.

[0058] Table 1 Experimental configuration

[0059] The performance of the model under different communication compression strategies is compared through experiments, and the experimental results are shown in Figure 3 Figure 3 By applying the gradient compression and activation value compression algorithm, the single-step time of the model is reduced from 952.8 ms to 644.3 ms, with a performance improvement of 32.4%.

[0060] The classical mixture-of-experts language model ​​A full training iteration of 100,000 times was performed on the large text dataset OpenWebText. A comprehensive evaluation of the effectiveness of the hybrid expert model training optimization method based on communication data compression is crucial. Besides measuring its performance advantage in single-step training time, verifying its ability to guarantee model convergence and final accuracy in a complete end-to-end training task is also essential.

[0061] Experimental results are as follows Figure 4 and Figure 5 As shown, Figure 4 This is a diagram showing the comparison of loss values. Figure 5 This is a diagram illustrating the comparison of confusion levels, from... Figure 4 and Figure 5 As can be seen, the entire training process exhibits good stability. The training loss value steadily decreases with the increase of the number of iterations, while the perplexity on the validation set also continues to decrease steadily, eventually converging. This indicates that the gradient compression and activation value compression strategies adopted in this method do not introduce instability into the gradient flow and parameter updates of the model training, ensuring healthy training dynamics.

[0062] After the training, the On the OpenWebText (a text dataset) validation set, the perplexity for language modeling reached 20.86, outperforming the baseline model's 21.72. These experimental results demonstrate that the hybrid expert model training optimization method based on communication data compression is not only an efficient training acceleration tool, but also a reliable and complete solution capable of producing high-performance models in practical applications.

[0063] In summary, this invention provides a communication compression optimization method for accelerating the training of large Mixture-of-Experts (MoE) models. The core contribution of this method lies in its departure from previous optimization approaches that focused solely on a single bottleneck. For the first time, it integrates specialized compression techniques for two types of heterogeneous communication data (gradients and activation values) into a unified framework, achieving collaborative optimization of the two major communication bottlenecks coexisting in MoE distributed training: All-Reduce and All-to-All. By employing the computationally efficient MSTopK approximation algorithm and contrast feedback and warm-up algorithms, this framework effectively solves the core challenge of traditional gradient sparsity. Simultaneously, by applying FP8 quantization technology and expert parallel pipeline scheduling, the bandwidth pressure for activation value exchange is significantly reduced. Experimental results demonstrate that this optimization method reduces the single-step training time of the MoE model by 32.4% on a 16-GPU cluster, achieving significant end-to-end acceleration. Crucially, this significant performance improvement does not compromise the model's convergence process or final accuracy.

[0064] In summary, the mixed communication compression framework proposed in the application is a practical solution with high efficiency and stability, which provides an effective technical path for coping with the communication challenges brought by the training of super large MoE models, and has important theoretical significance and engineering value for promoting large language models to develop in a larger scale and higher efficiency.

[0065] The embodiment also provides a training device of a mixed expert model, as shown in the figure, the device comprises the following components: Figure 6 The first compression module 01 is configured to obtain original activation values generated on a local node during iterative training of the mixed expert model, and compress the original activation values to obtain compressed activation values. The second compression module 02 is configured to obtain model original gradients on the local node during iterative training of the mixed expert, and compress the model original gradients to obtain compressed gradients. The sending module 03 is configured to send the compressed activation values to a remote node group and send the compressed gradients to the remote node group, and the remote node group is configured to continue iterative training of the mixed expert model based on the compressed activation values and the compressed gradients.

[0066] Based on the above embodiment, the application further provides a terminal device, and a principle block diagram of the terminal device can be as shown in the figure. Figure 7 The terminal device comprises a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the terminal device is configured to provide computing and control capabilities. The memory of the terminal device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a training method of a mixed expert model. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0067] Those skilled in the art can understand that, Figure 7 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal device to which the application scheme is applied. The specific terminal device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0068] In one embodiment, a terminal device is provided, which comprises a memory, a processor and a training program of a mixed expert model stored in the memory and executable on the processor. When the processor executes the training program of the mixed expert model, the following operation instructions are implemented: ​In the iterative training of the mixed expert model, a model original gradient on a local node is obtained, and a compressed gradient is obtained by compressing the model original gradient; In the iterative training of the mixed expert model, an original activation value generated on a local node is obtained, and a compressed activation value is obtained by compressing the original activation value; The compressed activation value is sent to a remote node group, and the compressed gradient is sent to the remote node group, and the remote node group is used to continue iterative training of the mixed expert model based on the compressed activation value and the compressed gradient.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a hybrid expert model, characterized in that, The method comprises the following steps: During iterative training of the mixed expert model, obtaining original activation values generated on the local node, and compressing the original activation values to obtain compressed activation values; During iterative training of the mixed expert model, obtaining model original gradients on the local node, and compressing the model original gradients to obtain compressed gradients; The compressed activation values are sent to a remote node group, and the compressed gradients are sent to the remote node group, and the remote node group is used to continue iterative training of the mixed expert model based on the compressed activation values and the compressed gradients.

2. The training method of a hybrid expert model according to claim 1, wherein, During iterative training of the mixed expert model, obtaining model original gradients on the local node, and compressing the model original gradients to obtain compressed gradients, comprising: During iterative training of the mixed expert model on the local node, obtaining model original gradients generated by a dense model in the mixed expert model and model original gradients generated by a sparse expert network in the mixed expert model; Respectively compressing the model original gradients of the dense model and the model original gradients of the sparse expert network to obtain respective compressed gradients.

3. The training method of a hybrid expert model of claim 1, wherein, Compressing the model original gradients to obtain compressed gradients, comprising: Obtaining a preset compression threshold, the compression threshold being used to represent a gradient compression ratio; Based on the compression threshold, the model original gradients are compressed to obtain compressed gradients.

4. The training method of a hybrid expert model according to claim 3, wherein, Based on the compression threshold, the model original gradients are compressed to obtain compressed gradients, comprising: In the last iteration training, based on the compression threshold, the elements of the model original gradients are sparsified to preliminarily compress the model original gradients; According to the values of the sparsified elements in the last iteration training and the values of the sparsified elements in the current iteration training, the model original gradients after preliminary compression are secondarily compressed to obtain compressed gradients.

5. The training method of a hybrid expert model of claim 1, wherein, Compressing the original activation values to obtain compressed activation values, comprising: Determining a maximum activation value in the original activation values; Determining a maximum value that can be stored in an FP8 data format; Based on the maximum activation value and the maximum value, a scaling factor is determined; Based on the scaling factor, the original activation values are compressed to realize compression of the original activation values into compressed activation values in the FP8 data format.

6. The method of training a hybrid expert model according to any one of claims 1-5, wherein, The compressed activation values are sent to a remote node group, comprising: Based on a CUDA multi-stream concurrent mechanism, the compressed activation values are distributed to the remote node group.

7. The training method of a hybrid expert model according to claim 6, wherein The mixed expert model after training is used for text classification or text language translation.

8. A training apparatus of a hybrid expert model, characterized by, The device comprises the following components: A first compression module is configured to, during iterative training of a mixed expert model, obtain original activation values generated on a local node, and compress the original activation values to obtain compressed activation values; A second compression module is configured to, during iterative training of the mixed expert model, obtain model original gradients on the local node, and compress the model original gradients to obtain compressed gradients; A sending module is configured to send the compressed activation values to a remote node group, and send the compressed gradients to the remote node group, and the remote node group is used to continue iterative training of the mixed expert model based on the compressed activation values and the compressed gradients.

9. A terminal device, comprising: The terminal device comprises a memory, a processor, and a training program of the hybrid expert model stored in the memory and executable on the processor, and the processor implements the steps of the training method of the hybrid expert model according to any one of claims 1-7 when executing the training program of the hybrid expert model.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the training program of the hybrid expert model, and the training program of the hybrid expert model is executable on the processor, and the processor implements the steps of the training method of the hybrid expert model according to any one of claims 1-7 when executing the training program of the hybrid expert model.

Images