MoE fine tuning training method and device, electronic equipment and storage medium

Abstract

The invention provides a fine tuning training method and device of MoE, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of distributed training and optimization of a large-scale machine learning model, compression and acceleration of a deep learning model and the like. According to the implementation scheme, model structure analysis is conducted on a MoE model to be finely adjusted, and an expert module and a non-MoE module in the MoE model are recognized; quantizing the weight parameter of the expert module by adopting first low-bit precision, and quantizing the weight parameter of the non-MoE module by adopting second low-bit precision; performing model forward calculation and back propagation calculation based on the quantized weight by using a mixed precision calculation operator; and based on an LoRA fine tuning technology, performing fine tuning training on the quantized MoE model by using a distributed training framework to obtain a target MoE model. The scheme can significantly reduce video memory occupation, effectively improve training efficiency, break through model scale limitation, and maintain excellent model performance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of distributed training and optimization of large-scale machine learning models, deep learning model compression and acceleration, and more particularly to a low-bit quantization and efficient distributed fine-tuning training method for a mixture of experts (MoE). BACKGROUND

[0002] In related technologies, there are obvious deficiencies in fine-tuning large-scale MoEs: first, a unified quantization strategy is adopted, which cannot balance the accuracy and efficiency requirements of different modules in the MoE structure; second, there is a lack of optimization support for distributed training mechanisms such as expert parallelism; and third, the quantization calculation process itself has an efficiency bottleneck, which affects the overall training speed. SUMMARY

[0003] The present disclosure provides a fine-tuning training method, device, electronic equipment and storage medium for a MoE.

[0004] According to a first aspect of the present disclosure, a fine-tuning training method for a MoE is provided, which includes: performing model structure analysis on a MoE model to be fine-tuned to identify expert modules and non-MoE modules inside the MoE model; quantizing weight parameters of the expert modules using a first low-bit precision and quantizing weight parameters of the non-MoE modules using a second low-bit precision, wherein the first low-bit precision is lower than the second low-bit precision; performing forward calculation and backpropagation calculation of the model based on the quantized weights using a mixed-precision calculation operator; and fine-tuning the quantized MoE model using a distributed training framework based on a low-rank adaptation (LoRA) fine-tuning technology to obtain a target MoE model.

[0005] According to a second aspect of the present disclosure, a fine-tuning training device for a MoE is provided, which includes: a structure analysis module configured to perform model structure analysis on a MoE model to be fine-tuned to identify expert modules and non-MoE modules inside the MoE model; a quantization processing module configured to quantize weight parameters of the expert modules using a first low-bit precision and quantize weight parameters of the non-MoE modules using a second low-bit precision, wherein the first low-bit precision is lower than the second low-bit precision; a calculation control module configured to perform forward calculation and backpropagation calculation of the model based on the quantized weights using a mixed-precision calculation operator; and a fine-tuning training module configured to fine-tune the quantized MoE model using a distributed training framework based on a low-rank adaptation (LoRA) fine-tuning technology to obtain a target MoE model.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] By employing the scheme disclosed herein, differentiated low-bit quantization strategies are used for expert modules and non-MoE modules within the MoE model. This deeply couples the extreme compression of expert modules with the accuracy guarantee of non-MoE modules, significantly reducing the memory threshold and computational overhead for large-scale model training. Through the use of mixed-precision computation and customized high-performance operators, numerical accuracy and computational efficiency are adaptively balanced, ensuring the stability and throughput of the quantized model training process. Furthermore, the introduction of LoRA fine-tuning technology breaks through the resource bottleneck of traditional full-parameter fine-tuning, and in collaboration with distributed training frameworks, further optimizes the allocation of training resources. Simultaneously, the fusion of multiple parallel strategies effectively leverages the sparsity characteristics and hardware parallelism potential of the MoE model, demonstrating excellent scalability in ultra-large-scale model training.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a flowchart illustrating the MoE fine-tuning training method according to an embodiment of the present disclosure.

[0013] Figure 2 This is a diagram of a MoE hierarchical low-bit quantization computing architecture with integrated LoRA fine-tuning according to an embodiment of this disclosure;

[0014] Figure 3 This is a schematic diagram of the structure of a fine-tuning training device for MoE according to an embodiment of the present disclosure;

[0015] Figure 4 is a scenario schematic diagram of the fine-tuning training method of the MoE according to the embodiments of the present disclosure;

[0016] Figure 5 is a structural diagram of an electronic device for implementing the fine-tuning training method of the MoE according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of the embodiments of the present disclosure, and should be considered in conjunction with the preceding Description and Drawings. Those of ordinary skill in the art will realize that the embodiments of the present disclosure can be easily adapted to different applications and equivalent structures, and that various modifications and changes can be made within the scope of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0018] The terms "first", "second", etc. herein refer to and distinguish a plurality of similar technical terms, and do not mean to limit the order or mean to limit to only two, for example, the first feature and the second feature refer to two types / two features, the first feature can be one or more, and the second feature can also be one or more.

[0019] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art will understand that the present disclosure can be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0020] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that can be used in the present disclosure are further described:

[0021] MoE: a kind of neural network architecture, its core idea is to split a large network into multiple small sub-networks (called "experts (Expert)"), and introduce a gating network (Gating Network) to dynamically select the most relevant few experts for each input sample. Its characteristic is "sparse activation", that is, for any input, only a small part of all experts is used, thereby significantly increasing the total amount of model parameters while controlling the cost of calculation. For example, a MoE model with 1000 experts, the gating network may select only the top 2 experts for each input token for forward calculation, while the other 998 experts are in an inactive state.

[0022] LoRA: A parameter-efficient fine-tuning method for large language models. The basic principle of LoRA is to freeze the pre-trained model weight parameters, add additional network layers to the model, and only train these newly added network layer parameters while keeping the original model parameters unchanged.

[0023] Quantized LoRA (QLoRA): An optimized extension of LORA, designed to further reduce the computational and memory requirements of large language model fine-tuning.

[0024] General Matrix Multiply (GEMM): GEMM plays a very important role in neural network computation, representing the global matrix-to-matrix multiplication, essentially multiplying two input matrices together to get an output matrix. It is the underlying core of most computations in neural networks, such as fully connected layers and convolutional layers.

[0025] Operator: In computer science and high-performance computing, an "operator" refers to a function or program segment that performs a specific computational operation.

[0026] GEMM Operator: A highly optimized function that performs matrix multiplication calculations.

[0027] In related technologies, there are three main deficiencies in fine-tuning MoE: a) Single quantization granularity: MoE models are not distinguished between expert modules and non-MoE parts, and a unified quantization strategy (such as all INT4 or all INT8) is used, resulting in either resource waste or precision loss. b) Insufficient support for distributed training: unable to well combine the unique expert parallel strategy of MoE models, and the parallel potential is not fully realized. c) Low quantization computation efficiency: the quantization operator itself may have poor performance, becoming a training bottleneck.

[0028] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure proposes a fine-tuning training method for MoE (MoE Model). By employing differentiated low-bit quantization strategies for expert modules and non-MoE modules within the MoE model, it deeply couples the extreme compression of expert modules with the accuracy guarantee of non-MoE modules, significantly reducing the memory threshold and computational overhead for large-scale model training. Through the use of mixed-precision computing and a customized high-performance operator, it adaptively balances numerical accuracy and computational efficiency, ensuring the stability and throughput of the quantized model training process. Furthermore, the introduction of LoRA fine-tuning technology breaks through the resource bottleneck of traditional full-parameter fine-tuning, and in collaboration with distributed training frameworks, further optimizes the allocation of training resources. Simultaneously, the fusion of multiple parallel strategies effectively leverages the sparsity characteristics and hardware parallelism potential of the MoE model, demonstrating excellent scalability in ultra-large-scale model training.

[0029] This disclosure provides a method for fine-tuning training of MoE. Figure 1 This is a flowchart illustrating a MoE fine-tuning training method according to an embodiment of the present disclosure. This MoE fine-tuning training method can be applied to a MoE fine-tuning training device. The MoE fine-tuning training device is located in an electronic device. This electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, fine-tuning training devices, which can be mobile phones, tablets, etc. In some possible implementations, the MoE fine-tuning training method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the fine-tuning training method for this MoE includes:

[0030] S101. Perform model structure analysis on the MoE model to be fine-tuned to identify the expert modules and non-MoE modules within it.

[0031] S102. The weight parameters of the expert module are quantized using the first low bit precision, and the weight parameters of the non-MoE module are quantized using the second low bit precision, wherein the first low bit precision is lower than the second low bit precision.

[0032] S103. Perform forward and backward propagation calculations of the model based on quantized weights using mixed-precision computation operators;

[0033] S104. Based on LoRA fine-tuning technology, a distributed training framework is used to fine-tune the quantized MoE model to obtain the target MoE model.

[0034] Here, the expert module (Expert Module) is a plurality of sub-networks with the same structure and independent parameters in the MoE model. The expert module is usually a feed-forward neural network (FFN), which is the main carrier of model parameters and has sparsity. The expert module can include a plurality of parallel FFN layers.

[0035] Here, the non-MoE module (Non-MoE Module) refers to a shared component in the MoE model that all inputs must pass through and does not contain an expert routing mechanism. They are the basic framework of the model and are crucial to the overall performance of the model. For example, non-MoE modules include self-attention layers (including queries (Query), keys (Key), and values (Value)), gated networks, input / output embedding layers (Embedding Layer), layer normalization layers (LayerNorm), etc., all of which belong to non-MoE modules.

[0036] Here, low-bit quantization (Low-Bit Quantization) is a model compression technique that reduces model storage and computational overhead by reducing the number of bits used to represent weights or activation values. For example, converting weights originally represented by 32-bit floating-point numbers (FP32) to 8-bit integers (INT8) or 4-bit integers (INT4). For example, INT8 quantization maps weights from FP32 (about -3.4x10 38 to 3.4x10 38 ) to 256 discrete integer intervals (-128 to 127). INT4 maps to 16 intervals (-8 to 7), with higher compression rates but greater risk of precision loss.

[0037] Here, LoRA is a parameter-efficient fine-tuning (PEFT) method that assumes that weight updates during model fine-tuning are low-rank. By freezing the original weights of the pre-trained model and injecting an adapter that is a low-rank decomposition to simulate weight updates, only these adapter parameters are trained during fine-tuning.

[0038] Here, the distributed training framework is a software framework for training super-large models that cannot be accommodated on a single device by distributing the model and / or data to multiple computing devices (such as GPUs) for parallel collaboration to complete the training. Parallel strategies include but are not limited to: 1. Data Parallelism: Splitting batch data to multiple devices, each device has a complete model copy. 2. Tensor Parallelism: Splitting a single weight tensor to multiple devices. 3. Pipeline Parallelism: Splitting the model by layer depth to multiple devices. 4. Expert Parallelism: MoE specific, distributing different expert modules to different devices.

[0039] In the embodiments of the present disclosure, the MoE model to be fine-tuned can be first analyzed for structure to identify the internal expert modules and non-MoE modules. Specifically, all multi-layer perceptron subnetworks in the model identified as "expert" or "ffn_expert" can be classified as expert modules by parsing the model configuration file or the forward computation graph, while the self-attention layer, the gating network layer, the input-output embedding layer, and the layer normalization layer can be classified as non-MoE modules. In particular, this identification process is automatically completed in a programmed manner, ensuring the generality and reusability of the solution for different structure MoE models. The above is only an exemplary illustration and is not intended to limit all possible cases of identifying model modules, and no exhaustive enumeration is made here.

[0040] In the embodiments of the present disclosure, after completing the module identification, the weight parameters of the expert modules can be quantized using a first low-bit precision, and the weight parameters of the non-MoE modules can be quantized using a second low-bit precision. Specifically, the first low-bit precision can use INT4 precision, and scale and zero-point parameters can be calculated independently for each weight matrix by channel dimension to achieve fine quantization; the second low-bit precision can use INT8 precision, and can use tensor-level or channel-level quantization methods. In particular, this differential quantization strategy takes full advantage of the characteristics of concentrated parameter distribution and high fault tolerance of expert modules, significantly reducing the memory usage while ensuring the accuracy of key components of the model. The above is only an exemplary illustration and is not intended to limit all possible cases of implementing the quantization strategy, and no exhaustive enumeration is made here.

[0041] In this embodiment, a mixed-precision computation operator can be used to perform model computation based on quantized weights. Specifically, during forward and backward propagation, an activation tensor in FP16 or BF16 format and a quantized weight tensor in INT8 or INT4 format are used as inputs, and a customized mixed-precision GEMM kernel is invoked for computation. In particular, this operator can be implemented in two ways: one is to dequantize the quantized weights into a high-precision format in real time before computation, and the other is to directly utilize low-bit matrix multiplication primitives supported by hardware instructions for computation, thereby greatly improving computational throughput while ensuring numerical accuracy. The above is merely an illustrative example and does not constitute a limitation on all possible implementations of mixed-precision computation; it is simply not exhaustive.

[0042] In this embodiment, the quantized MoE model can be fine-tuned using a distributed training framework based on LoRA fine-tuning technology. Specifically, trainable low-rank adapter matrices can be injected into all linear layers of the model, and the original quantization weights can be frozen. Simultaneously, a distributed strategy combining expert parallelism, data parallelism, tensor parallelism, and pipeline parallelism is employed to coordinate the training process. In particular, the expert parallelism strategy distributes different experts to different computing devices and uses a gating network to achieve dynamic routing of inputs, thereby fully leveraging the sparsity characteristics and hardware parallelism potential of the MoE model. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for implementing distributed fine-tuning; it is simply not exhaustive.

[0043] The technical solution of this disclosure employs a differentiated low-bit quantization strategy for expert modules and non-MoE modules within the MoE model, deeply coupling the extreme compression of expert modules with the accuracy guarantee of non-MoE modules, significantly reducing the memory threshold and computational overhead for large-scale model training. By using mixed-precision computation and customized high-performance operators, numerical accuracy and computational efficiency are adaptively balanced, ensuring the stability and throughput of the quantized model training process. Furthermore, the introduction of LoRA fine-tuning technology breaks through the resource bottleneck of traditional full-parameter fine-tuning, and in collaboration with the distributed training framework, further optimizes the allocation of training resources. Simultaneously, the fusion of multiple parallel strategies effectively leverages the sparsity characteristics and hardware parallel potential of the MoE model, demonstrating excellent scalability in ultra-large-scale model training. For gradient synchronization and quantization parameter optimization during training, efficient and stable convergence can be achieved through targeted design of the distributed framework, improving the system's fault tolerance for large-scale distributed training. Moreover, the compressed model obtained from training can be efficiently deployed in resource-constrained real-world environments, significantly reducing hardware costs while ensuring model performance, providing a solid technical foundation for the widespread deployment of hybrid expert models.

[0044] In some embodiments, the first low-bit precision is INT4 (4-bit integer) precision, and the second low-bit precision is INT8 (8-bit integer) precision.

[0045] In one specific implementation, the weight parameters of the expert module are quantized using the first low-bit precision, specifically INT4 precision; the weight parameters of the non-MoE modules are quantized using the second low-bit precision, specifically INT8 precision. The expert module has a huge number of parameters but exhibits sparsity and fault tolerance. Using INT4 quantization reduces its memory usage to a mere 1 / 4 of the original FP16 model, which is the primary contributor to the significant reduction in overall memory requirements. Simultaneously, using INT8 quantization for the non-MoE modules, which are more critical to model accuracy, significantly reduces quantization errors while still achieving a 2x memory compression, providing crucial assurance for the fine-tuning performance and final accuracy of the entire model.

[0046] Specifically, for weight matrices in the MoE model identified as expert modules (such as multiple independent FFN layers), each parameter is converted from its original FP16 or FP32 precision to a 4-bit integer representation in the range [-8, 7] through quantization operations (such as scaling and zero-pointing). For weight matrices in the model identified as non-MoE modules (such as self-attention layers and gated networks), their parameters are converted to an 8-bit integer representation in the range [-128, 127]. By clearly defining the target quantization precision (INT4 / INT8) for each module, the design and invocation of the underlying mixed-precision computing kernel (such as GEMM) can be targeted, allowing for in-depth optimization for specific modes such as INT4-INT8 mixed precision or INT4-FP16 mixed precision, thereby maximizing the hardware computing potential and improving overall training efficiency.

[0047] Thus, by applying the two specific precision quantization schemes, INT4 and INT8, to different functional modules of the MoE model, an optimal balance between extreme memory compression and reasonable precision assurance is achieved. The relatively concentrated parameter distribution and insensitivity to precision changes in expert modules are precisely utilized, resulting in a more aggressive compression; while the computationally intensive nature of non-MoE modules and their global impact on the overall model performance are respected, leading to a more conservative compression. Therefore, the inherent characteristics of different modules are leveraged, and intelligent resource allocation is achieved.

[0048] In some embodiments, the weight parameters of the expert module are quantized using a first low bit precision, including: obtaining a first weight matrix of the expert module; independently calculating a first quantization parameter for each channel of the weight matrix according to the channel dimension; and performing INT4 quantization on the first weight matrix based on the first quantization parameter calculated independently for each channel.

[0049] Here, the first weight matrix refers to the set of learnable parameters in the neural network layer that constitutes an expert module, existing in the form of a matrix. In one expert module, there are usually at least two weight matrices: the first projects the input to a high-dimensional space, and the second projects it back to the original dimension. "First" here does not specifically refer to the order, but is used to distinguish the weights of different modules, which is a general concept, referring to any weight matrix that needs to be quantized from an expert module. For example, suppose the expert module is an FFN, whose first layer is a linear layer that projects the input dimension from d model = 768 to d ff = 3072, d model and d ff represent the model dimension and the feed-forward dimension, respectively. Then the weight of this linear layer is a matrix with dimensions [768, 3072], which is a "first weight matrix".

[0050] Here, the channel dimension (Channel Dimension), in the context of neural network weight matrices, "channel" usually refers to the output channel (Output Channel), i.e., the row (or column of the weight matrix. In channel-level quantization, "channel" refers to this output dimension. Quantizing independently for each output channel means that each row / column of the matrix has its own quantization parameters. Continuing the example above, the weight matrix with dimensions [768, 3072] has 3072 output channels. Channel-level quantization means calculating a set of independent quantization parameters for each of the 3072 channels (i.e., 3072 columns of the matrix).

[0051] Here, the first quantization parameter (First Quantization Parameters) refers to the scaling factor and offset calculated to convert high-precision (such as FP16) weight values to low-precision (such as INT4) representation. The most common quantization parameters are scale and zero point. "First" here is used to distinguish the quantization parameters of expert modules from those of non-MoE modules. For example, for all weight values of the i-th output channel (i.e., the i-th column) of the above matrix, calculate its maximum value max i and minimum value min i . Then according to the formula, the scale (S i ) and zero point (Z i ) of this channel are calculated.

[0052] Here, the zero point For INT4, the value range is 15.

[0053] Here, S i is an integer to ensure that 0 can be accurately represented after quantization.

[0054] This set of (S i , Z i ) is the "first quantization parameter" of the i-th channel.

[0055] In one specific implementation, the process of INT4 quantization of the expert module adopts a Per-Channel quantization granularity. Specifically, first, a weight matrix (denoted as the first weight matrix W expert ) in the target expert module is obtained, which usually has a dimension of [output_channels x input_channels]. Then, the exclusive quantization parameter (denoted as the first quantization parameter) of each output channel (i.e., each row of the matrix) is independently calculated along the output channel (output_channel) dimension of the matrix. The quantization parameter usually includes a scale and a zero point. Finally, all weight values in the channel are normalized and rounded to convert them to 4-bit integer (INT4) representation using the scale and zero point independently calculated for each channel.

[0056] The technical solution of the embodiments of the present disclosure couples the extreme parameter compression with the fine error control depth by introducing a Per-Channel granularity for the INT4 quantization of the expert module, significantly improving the numerical representation capability of low-bit quantization. By adopting the strategy of independently calculating the quantization parameter according to the output channel, the numerical distribution characteristics of different channels in the weight matrix are adaptively matched, ensuring the retention degree of key information of the model under the extremely low precision of INT4. On this basis, the Per-Channel quantization can realize the accuracy improvement without introducing additional overhead in the inference calculation stage, which, in cooperation with the high-performance mixed precision operator, further guarantees the efficiency and stability of the training process; at the same time, this fine quantization strategy effectively avoids the problem of damaging the overall quantization effect due to a small number of abnormal value channels, so that the expert module can maintain its functionality while obtaining a 4-fold compression benefit. In view of the structural characteristics that different channels in the expert module may carry different feature information, individualized protection can be performed on them through personalized quantization parameters, thereby improving the fault tolerance of the model to quantization noise. Moreover, this technology provides key support for reliably fine-tuning and deploying MoE models with hundreds of billions of parameters under ultra-low precision, and lays a solid technical foundation for breaking through the hardware memory bottleneck.

[0057] In some embodiments, the weight parameters of the non-MoE module are quantized at a second low-bit precision, including: obtaining a second weight matrix of the non-MoE module; calculating a second quantization parameter for the second weight matrix according to the channel dimension or the tensor level; and performing INT8 quantization on the second weight matrix based on the second quantization parameter.

[0058] Here, the second weight matrix is a generic concept, referring to any weight matrix from non-MoE modules that needs to be quantized. The "second" is used here to distinguish from the "first weight matrix" of the expert module. For example, in a self-attention layer, the weight matrix W Q that projects the input vector into the Query space, which can have a dimension of 768x768, is a typical "second weight matrix".

[0059] Here, Tensor-Level / Per-Tensor: a unique set of quantization parameters (a scale parameter S, a zero-point parameter Z

[0060] Here, Channel-Level / Per-Channel Quantization: a set of quantization parameters is calculated independently for each output channel of the weight matrix.

[0061] For example, for a matrix W Q with a dimension of 768x768, if tensor-level quantization is used, a global scale S and zero-point Z are calculated by traversing all 768x768 weight values in the matrix. If channel-level quantization is used, a set of scale S i and zero-point Z i is calculated independently for each column (i.e., each of the 768 output channels) in the matrix (where i = 1, 2,..., 768).

[0062] Here, the second quantization parameter refers to the parameter calculated to convert the high-precision weight of the non-MoE module to INT8 representation, which also includes scale and zero-point. The "second" is used to distinguish from the "first quantization parameter" of the non-MoE module. Here, the "second quantization parameter" and the "first quantization parameter" have the same calculation method, but the weight matrix and the precision target (INT8) are different.

[0063] Here, performing INT8 quantization refers to the process of converting high-precision weight values to 8-bit integers (usually ranging from -128 to 127) using the second quantization parameter calculated above. For example, for an FP16 weight value ω, the formula for quantizing it to an INT8 value q is:

[0064]

[0065] wherein S is a scale parameter, Z is a zero-point parameter, round(·) is a rounding function, clamp(x, a, b) is a clamping function, satisfying x < a, clamp(x, a, b) = a; a ≤ x ≤ b, clamp(x, a, b) = x; x > b, clamp(x, a, b) = b.

[0066] In some embodiments, the weight parameters of the non-MoE module are quantized with INT8 precision, and the specific implementation includes: first, obtaining the weight matrix (denoted as a second weight matrix) of the target non-MoE module (such as the Key projection matrix in the self-attention layer). Subsequently, according to a pre-configured strategy or the numerical distribution characteristics of the weight matrix, a tensor-level or channel dimension is selected to calculate the quantization parameter (denoted as a second quantization parameter). For a matrix with a relatively uniform numerical distribution, a tensor-level quantization with higher calculation efficiency can be used; for a matrix with a large difference in numerical distribution, a channel-level quantization with higher precision can be used. Finally, based on the calculated second quantization parameter, each FP32 or FP16 weight value in the second weight matrix is converted to INT8 format, completing the quantization process.

[0067] The technical scheme of the embodiments of the present disclosure couples the significant memory compression with the accuracy guarantee of the key module by using INT8 quantization for the non-MoE module and providing flexible quantization granularity selection, thereby achieving an optimized balance of the overall model performance. By allowing adaptive selection between tensor-level and channel-level granularity, the accuracy of the non-MoE core module is guaranteed while providing configuration flexibility for the quantization process that takes into account both efficiency and accuracy. On this basis, the INT8 precision provides higher numerical representation for the non-MoE module compared to the INT4 precision of the expert module, effectively avoiding risks such as attention mechanism failure or distorted gating routing decision that may be caused by excessive quantization. At the same time, the INT8 format is highly optimized to support the current mainstream hardware acceleration unit, ensuring that this part of the calculation still has excellent inference speed. In view of the characteristics of the diverse and functionally different non-MoE modules, they can be personalized processed through differentiated quantization granularity strategies, thereby improving the relevance and effectiveness of the quantization strategy. Moreover, this technology lays a solid foundation for reliably maintaining the performance of core functional modules in large-scale model fine-tuning, and is a key technical link to achieve the dual goals of reducing resource threshold and ensuring model effectiveness.

[0068] In the embodiments of the present disclosure, the forward calculation of the model based on the quantized weight is performed using a mixed-precision calculation operator, including: in the forward propagation process, taking the activation tensor in FP16 or BF16 format and the quantized weight tensor in INT8 or INT4 format as input; calling a mixed-precision GEMM operator as a mixed-precision calculation operator to perform calculation to obtain an output tensor in FP16 or BF16 format.

[0069] Here, the activation tensor refers to the input and output data of each layer of the neural network in the forward propagation process. It is an intermediate result obtained after the input sample is calculated by the previous layer, and represents the layer-by-layer abstraction and feature extraction of the network on the input data. During training, the activation tensor needs to be saved for gradient calculation in the backward propagation process, and therefore occupies a large amount of video memory. For example, when calculating self-attention, the tensor input to the linear projection layer; or the tensor input to the expert module FFN layer, are all activation tensors. The dimension is usually B x L x D, B is the batch size, L is the sequence length, and D is the hidden layer dimension.

[0070] Here, FP16 and BF16 formats are two mainstream half-precision floating-point formats, which are used to reduce video memory occupation and accelerate calculation. FP16 (Float16): 16-bit floating-point number, with a small representation range, which is prone to gradient underflow problems. BF16 (Brain Floating Point): 16-bit floating-point number, which sacrifices part of the precision to exchange for a representation range similar to FP32, is more stable and less prone to underflow, and is widely used in modern artificial intelligence (AI) training.

[0071] Here, the quantized INT8 or INT4 weight tensor refers to the low-bit integer format model weight parameter converted from high-precision (FP32 / FP16). For example, a weight matrix originally stored in FP16 format and occupying 2MB video memory, after INT4 quantization, only occupies 0.5MB video memory.

[0072] Here, the mixed-precision GEMM operator is a high-performance calculation kernel, which is the core of matrix multiplication. "Mixed precision" means that the precisions of the inputs (operands) are different, but the calculation process is specially optimized to achieve the highest efficiency. For example, the call of a mixed-precision GEMM operator can be represented in the following function form:

[0073] cublasGemmEx(…,A dtype =BF16,B dtype =INT4,C dtype =BF16,…); where the data type of the input matrix A (activation tensor) is BF16, the data type of the input matrix B (weight tensor) is INT4, and the data type of the output matrix C is BF16.

[0074] Here, the output tensor refers to the tensor obtained after computation by the mixed-precision GEMM operator. It will serve as the input to the next layer (activation tensor) or as the final output of the entire model. For example, in a self-attention layer, after QK... T The calculated attention fraction tensor A∈R B×L×L ; or the output feature tensor H obtained after computation by the FFN layer. out ∈R B×L×D Q and K represent the query and key matrices, respectively; B represents the batch size; L represents the sequence length; and D represents the hidden layer dimension.

[0075] In some implementations, mixed-precision computation operators are used to perform model forward computation based on quantized weights, as follows: during forward propagation, activation tensors of the previous layer in FP16 or BF16 format are obtained from the current layer; simultaneously, weight tensors of the previous layer in INT8 format (for non-MoE modules) or INT4 format (for expert modules) are loaded from memory.

[0076] In some implementations, the two tensors of different precisions mentioned above are used as inputs to invoke the underlying optimized high-performance mixed-precision GEMM operator. This operator internally achieves efficient computation through two optional paths:

[0077] First, the INT8 / INT4 weight tensor is dequantized in real time to FP16 / BF16 format, and then standard half-precision matrix multiplication is performed with the FP16 / BF16 format activation tensor.

[0078] Secondly, by directly utilizing mixed-precision computation instructions supported by hardware acceleration units (such as the Tensor Core of a GPU), the multiply-accumulate operations with low bit weights are integrated with the calculation of activation values ​​at the hardware level, avoiding the overhead of explicit dequantization.

[0079] Finally, the operator outputs the computation result and preserves it as a tensor in FP16 or BF16 format for use in subsequent layer computations.

[0080] The technical solution of the embodiments of the present disclosure deeply couples the storage advantage of low-bit weights and the calculation stability of high-precision activations by calling the mixed-precision GEMM operator to perform forward calculation, and realizes the collaborative optimization of training efficiency and numerical accuracy. By adopting a high-performance calculation kernel supporting heterogeneous input, the calculation gap between the quantized weights and the floating-point format activations is adaptively bridged, ensuring the numerical accuracy of the forward propagation process. On this basis, the two implementation paths (dequantization calculation and direct hardware calculation) provided by the operator provide flexible choices that balance generality and extreme performance, and cooperate with different hardware environments to further release the acceleration potential of low-bit quantization; at the same time, the output result maintains a high-precision format, effectively avoiding the accumulation and amplification of errors between layers, providing a key guarantee for the stable training and reliable convergence of the model. In view of the characteristics of the calculation-intensive operation in large-scale model training, the throughput of the core calculation unit GEMM can be greatly improved through a highly optimized special operator, thereby overcoming the performance bottleneck of quantization training and improving the training speed.

[0081] In the embodiments of the present disclosure, when performing calculation, the mixed-precision GEMM operator first dequantizes the quantized weight tensor in INT8 or INT4 format to FP16 or BF16 format, and then performs matrix multiplication calculation with the activation tensor in FP16 or BF16 format.

[0082] Here, dequantization is the inverse operation of the quantization process, which refers to the process of converting the numerical value represented in low-bit integer format back to its original high-precision floating-point format representation. This operation uses the quantization parameters (scale and zero point) stored during quantization to reconstruct a value similar to the original floating-point value. For example, for an INT4 value q = 5 and the scale S = 0.1 and zero point Z = 1 of the channel it belongs to, the dequantized floating-point value ω is calculated as: ω = (q - Z) · S = (5 - 1) × 1 = 0.4. i i i i

[0083] Here, matrix multiplication calculation refers to the basic matrix multiplication operation, i.e., the GEMM (General Matrix Multiplication) operation. Its definition is as follows: if matrix A ∈ R m×k A, matrix B ∈ R k×n , then their product matrix C = AB ∈ R m×n , where each element c ij in C is the sum of the products of the corresponding elements of the i-th row of A and the j-th column of B.

[0084] ​​​​In the embodiments of the present disclosure, when performing calculation, the mixed-precision GEMM operator is implemented in the following manner: first, the operator receives an input quantized weight tensor in INT8 or INT4 format and an activation tensor in FP16 / BF16 format. Then, the operator performs a dequantization step: based on the pre-stored quantization parameters (scale and zero point) corresponding to the quantized weight tensor, the entire INT8 / INT4 weight tensor is converted from its low-bit integer format to a floating-point tensor in FP16 or BF16 format in the calculation core on-the-fly. This process reconstructs discrete integer values into continuous floating-point values. Finally, the operator calls a highly optimized calculation unit that performs standard floating-point matrix multiplication to perform matrix multiplication calculation on the dequantized weight tensor in FP16 / BF16 format and the input activation tensor in FP16 / BF16 format, to obtain an output tensor in FP16 or BF16 format.

[0085] The technical solution of the embodiments of the present disclosure adopts the operator implementation path of dequantization first and then calculation, deeply combines the storage advantages of low-bit compression with the mature and stable high-precision calculation process, and ensures the high reliability and universality of the quantized training process. By introducing real-time dequantization operation in the core link of forward calculation, the problem is adaptively converted into a standard floating-point matrix multiplication problem, thereby completely eliminating the non-standard numerical processing and precision risks that may be caused by direct participation of low-bit format in calculation. On this basis, the path can seamlessly reuse the existing highly optimized standard floating-point GEMM calculation library, without relying on specific hardware instruction support, thereby achieving excellent hardware compatibility and software portability; at the same time, the calculation process is highly consistent with the unquantized training process, greatly reducing the complexity of system implementation and guaranteeing the numerical stability of training, and avoiding convergence problems that may be caused by the introduction of a new type of calculation mode.

[0086] In the embodiments of the present disclosure, the mixed-precision GEMM operator directly uses a quantized weight tensor in INT8 or INT4 format and an activation tensor in FP16 or BF16 format to perform mixed-precision matrix multiplication calculation on a hardware acceleration unit.

[0087] Here, the hardware acceleration unit (Hardware Acceleration Unit) refers to a processor or a specific component within a processor specially designed to efficiently perform specific calculation tasks (such as matrix operations). Through a customized instruction set and hardware architecture, it provides a much higher calculation throughput and energy efficiency ratio than general-purpose processors for target operations. The hardware acceleration unit can natively support mixed-precision matrix multiplication operations.

[0088] Here, Directly Using refers to the mixed-precision GEMM operator performing calculation without the need to dequantize the low-bit integer weight tensor into floating-point format, but keeping it in the original INT8 / INT4 format and directly sending it into the calculation unit as an operand to participate in the operation. For example, the operator directly sends the INT4 format weight data stream and the BF16 format activation data stream into the instruction interface of the Tensor Core, rather than first converting the INT4 data into BF16 data through a series of instructions.

[0089] Here, Mixed-Precision Matrix Multiplication Calculation specifically refers to performing matrix multiplication on a hardware acceleration unit using operands of different precisions (such as INT4 weights and BF16 activations), and the hardware internally completes the process of format conversion, scaling, and accumulation, and the calculation rules are defined by hardware instructions, which are usually represented as: low-precision operands are promoted to intermediate precision before multiplication, and the results are accumulated into higher-precision accumulators.

[0090] In the embodiments of the present disclosure, the mixed-precision GEMM operator performs calculation in the following way: the operator is configured to call the mixed-precision calculation instruction specific to the hardware acceleration unit (such as the Tensor Core of the GPU). During the forward calculation process, the operator takes the quantized weight tensor in INT8 or INT4 format and the activation tensor in FP16 or BF16 format as input operands, and directly passes them to the instruction interface of the hardware acceleration unit. After the hardware acceleration unit receives the two tensors of different precisions, it directly performs mixed-precision matrix multiplication calculation internally: without the need for explicit dequantization by software intervention, the hardware logic automatically completes the decoding of low-bit weights, multiplication with high-precision activation values, and accumulation of partial results. Finally, the hardware acceleration unit outputs the calculation result to obtain an output tensor in FP16 or BF16 format.

[0091] The technical solutions of the embodiments of the present disclosure realize a leap in training efficiency by using an operator implementation path that directly uses low-bit weights and high-precision activation to perform calculation on a hardware acceleration unit, deeply coupling the algorithm-level compression advantage and the hardware-level extreme performance to realize a leap in training efficiency. Through the inverse quantization step and directly using the original mixed-precision calculation instruction of the hardware acceleration unit, the additional calculation overhead and data movement overhead caused by the inverse quantization operation are completely eliminated, so that the calculation process is completely focused on the core multiplication and addition operation. On this basis, the path fully plays the design advantages of the special hardware, and the highly parallel data flow and customized calculation unit inside the hardware can realize a throughput and energy efficiency ratio far exceeding that of the traditional process; at the same time, the numerical conversion and calculation completed inside the hardware maintain a high degree of determinism and optimization, further guaranteeing the low delay and high precision of the calculation process. For scenes that pursue extreme training and inference performance, the path provides a nearly optimal calculation implementation, which converts the theoretical performance gain brought by low-bit quantization into a real, measurable significant acceleration.

[0092] In the embodiments of the present disclosure, based on the LoRA fine-tuning technology, a low-rank decomposition matrix is added to the linear layer weight matrix in the MoE model; during the fine-tuning training process, only the low-rank decomposition matrix is updated, and the original quantized weight parameter is frozen.

[0093] Here, the LoRA fine-tuning technology is a parameter-efficient fine-tuning method. Its core assumption is that the weight update of the model when adapting to a new task has a low-rank characteristic. By freezing the pre-trained model weight and injecting a trainable low-rank decomposition matrix to approximate the weight update, the number of parameters that need to be trained is greatly reduced.

[0094] Here, the linear layer weight matrix refers to the learnable parameter matrix of the fully connected layer (linear transformation layer) in the neural network. Its function is to perform linear transformation on the input features through matrix multiplication operation. For example: in the Transformer structure, the query (Q), key (K), and value (V) projection matrices in the self-attention module, and the two linear transformation matrices in the FFN are all linear layer weight matrices.

[0095] Here, the low-rank decomposition matrix refers to a method of approximating the original matrix by decomposing it into the product of two or more smaller matrices. In LoRA, it specifically refers to two small matrices A and B used to simulate the weight update amount. For example, for a weight matrix W0∈R 768×768 , LoRA can introduce two low-rank matrices: A∈R 16×768 and B∈R 768×16 , whose product BA∈R 768×768A 768x768 weight update matrix is constructed, and the method only needs to train 768x16+16x768=24576 parameters, which is much less than the 768x768=589824 parameters required for the original matrix W0.

[0096] Here, Freeze is an operation that makes certain parameters un-updatable by the optimizer during training, keeping their values unchanged. The gradients of these parameters are not computed or, if computed, are not used for updating.

[0097] Here, Quantized Weight Parameters are model weights converted from high-precision floating-point format to low-bit integer format (such as INT4 / INT8). They are stored in compressed form and may need to be dequantized or use specific hardware instructions when calculating. For example, a weight matrix originally stored in FP16 format is quantized to INT4, and its values are stored in 4-bit integer form with quantization parameters such as scale and zero point.

[0098] In some embodiments, based on the LoRA fine-tuning technology, the implementation is as follows: First, identify all linear layers in the MoE model that need to be fine-tuned, including linear layers within the expert module and linear layers in non-MoE modules (such as self-attention layers, gated networks). Then, for the weight matrix of each target linear layer, add a pair of trainable low-rank decomposition matrices A and B (i.e., LoRA adapters). During fine-tuning training, set all original weight parameters (including weights that have been quantized to INT4 / INT8) to a frozen state, i.e., prohibit the optimizer from updating them. At the same time, set all low-rank decomposition matrices A and B to a trainable state. During training, only update the parameters of these low-rank decomposition matrices using the gradient descent algorithm. During forward propagation calculation, the output calculation of the linear layer is modified as: Y = W0X + B(AX), where X ∈ R k×n Y ∈ R d×n for input, and Y ∈ R

[0099] The technical solutions of the embodiments of the present disclosure combine the LoRA fine-tuning technology and the quantization technology, create a new paradigm of synergistic optimization among parameter efficiency, hardware efficiency and performance, freeze all original quantized weight parameters in the fine-tuning process, and only update the newly added low-rank decomposition matrix, thereby achieving three significant benefits: first, the number of trainable parameters is reduced by several orders of magnitude, greatly reducing the memory occupied by the gradient and the optimizer state, making it possible to fine-tune a super large-scale model under limited resources; second, since the original weights are frozen, the precision drift problem that may occur in the training process due to the update of quantized weights is avoided, ensuring the numerical stability of quantized training and the accuracy of the final model; third, the technology has strong universality, can realize efficient fine-tuning without modifying the original structure of the model, and can flexibly balance parameter efficiency and model capacity by controlling the size of the rank.

[0100] In the embodiments of the present disclosure, the low-rank decomposition matrix is added, including adding LoRA adapters for linear layers within expert modules and linear layers within non-MoE modules.

[0101] Here, the linear layers within expert modules refer to the fully connected layers (linear transformation layers) that constitute the internal full connection layers of each expert subnetwork in the MoE model. In the Transformer-based MoE model, this usually refers to the two linear layers contained in each expert feedforward network: the first one projects the input to a high-dimensional space, and the second one projects the high-dimensional representation back to the original dimension. For example, assuming that an expert is a feedforward network with a 3072-dimensional hidden layer, it contains two linear layers: the weight matrix of the first linear layer has a dimension of d model × 3072, and the weight matrix of the second linear layer has a dimension of 3072 × d model , where d model represents the hidden dimension of the model.

[0102] Here, the linear layers within non-MoE modules refer to the fully connected layers contained in all non-expert shared components in the MoE model. All input data of these layers will pass through, which is crucial to the overall function of the model. For example, it includes the query (Q), key (K), value (V) projection matrices and output projection matrices in the self-attention mechanism, as well as the linear transformation layers in the gating network.

[0103] Here, LoRA Adapter refers to the pair of trainable low-rank decomposition matrices (matrices A and B) injected into the original linear layer to implement LoRA fine-tuning. They collectively form an "adapter" to simulate the incremental update of the original weight without modifying the original weight. For example, for a 768x768 linear layer weight matrix, adding a LoRA adapter with rank r = 16, i.e., introducing A = R 16×768 and B = R 768×16 two trainable matrices.

[0104] In some embodiments, adding low-rank decomposition matrices to linear layer weight matrices in MoE models is implemented as follows: First, parse the MoE model structure to identify all linear layers that need to be fine-tuned. This includes two categories: linear layers within expert modules: traverse all expert modules in the model, and for each FFN contained in each expert, add a pair of independent LoRA adapters to the two linear layer weight matrices inside each expert. Linear layers within non-MoE modules: for all shared linear layers in the model, including Q, K, V, O projection matrices in self-attention layers, and linear layer weight matrices in gating networks, also add LoRA adapters.

[0105] In some embodiments, the addition operation is implemented by modifying the forward computation logic of the model: for the forward computation Y = W0X of each target linear layer, it is extended to Y = W0X + BAX, where W0∈R d×k is the original pre-trained weight matrix, X∈R k×n is the input matrix, Y∈R d×n is the output matrix, A∈R r×k and B∈R d×r are the injected LoRA adapter matrices, and r is the rank of the low-rank adaptation r << min(d, k). All adapters can be configured with the same rank r, or different ranks r can be configured according to the importance of the layer layer to achieve more fine-grained parameter allocation.

[0106] The technical solutions of the embodiments of the present disclosure add LoRA adapters to all linear layers in the expert module and the non-MoE module, which realizes the efficient fine-tuning of the parameters of the MoE model in a comprehensive, consistent and refined manner. By covering all functional components of the model (sparse experts and dense shared layers), it ensures that the entire model can adapt to downstream tasks, avoids the loss of capabilities caused by partial parameter freezing, and guarantees the overall performance and optimal task adaptability of the fine-tuned model. The unified LoRA fine-tuning strategy is adopted for experts and non-MoE modules, which maintains the mathematical consistency of the original model architecture. The update method of all linear layers is the same, which ensures the stability of the fine-tuning process and the predictability of the convergence behavior. Although adapters are added to all linear layers, the amount of new trainable parameters is still negligible compared to the original model parameters due to the low-rank nature of LoRA. This scheme achieves the maximum parameter efficiency with minimal parameter overhead, which maximizes the parameter efficiency. The unified implementation method is simple and easy to automate, and can be generalized to MoE models of various structures and sizes, significantly improving the universality and applicability of the technical solutions and reducing the complexity of application in different scenarios.

[0107] In some embodiments, the distributed training framework supports an expert parallel strategy.

[0108] Here, the distributed training framework refers to a software system that can automatically distribute the training task of a large-scale machine learning model to multiple computing devices (such as GPUs, Tensor Processing Units (TPUs)) for parallel execution. It is responsible for coordinating communication, data synchronization and computation scheduling between devices, aiming to break through the resource bottleneck of a single device and accelerate the training process.

[0109] In some embodiments, the distributed training framework is configured to have the ability to identify expert modules in the MoE model and provide corresponding interfaces and scheduling mechanisms, allowing users to declare the parallel mode of distributing experts to different computing devices.

[0110] In this way, it lays the foundation for supporting expert parallelism by declaration, enabling the framework to provide necessary system-level support for subsequent specific implementation of the expert parallel strategy, and laying the foundation for efficient training of MoE models.

[0111] In some embodiments, the expert parallel strategy includes: distributing different expert modules within the MoE model to different computing devices; in the forward propagation process, according to the routing result output by the gating network, dynamically routing the input data to the computing device where the corresponding expert is stored for calculation.

[0112] Here, the expert parallel strategy is a distributed parallel strategy specially designed for the hybrid expert model. The core idea is to distribute the numerous expert networks in the model to different computing devices, thereby distributing the memory overhead of storing expert parameters and the computing overhead of computing experts to the entire device cluster. For example, a MoE layer with 64 experts can be placed on each GPU by the expert parallel strategy, with about 8 experts on each GPU, on a device cluster with 8 GPUs.

[0113] In some embodiments: when implementing expert parallel, the initialization process of the framework will evenly or according to a strategy distribute the sub-networks identified as experts to different computing devices in the cluster. During forward propagation, the gating network first calculates the expert number to which each input token should be routed on the logical control device, and then the communication library of the framework executes an all-to-all communication operation according to this routing table to send the token to the physical device where the corresponding expert is stored for calculation. After the calculation is completed, the results are aggregated through communication.

[0114] In this way, the core memory bottleneck of the MoE model is directly solved and its sparsity is utilized. Distributing experts to different devices allows the total parameter quantity of the model to be much larger than the memory capacity of a single device, thereby supporting the training of super-large-scale models. Dynamic routing and communication allow the computing load to be dynamically allocated according to the input content, achieving flexible utilization and high efficiency of computing resources.

[0115] In some embodiments, the distributed training framework also supports a data parallel strategy.

[0116] Here, the data parallel strategy is the most basic parallel strategy. It divides the training data batch into multiple smaller sub-batches, each computing device holds a complete model copy, and independently processes a sub-batch, and finally updates the model by aggregating the gradients calculated by each device. For example, there is a 32-sample batch and 4 GPUs, and data parallelism will divide the batch into 4 parts, each with 8 samples, and distribute them to 4 GPUs for parallel computation.

[0117] In some embodiments, on top of expert parallel or other model parallel strategies, the framework supports further dividing the training data batch and distributing it to multiple parallel model copy groups. Each copy group processes a portion of the data and synchronously updates through the gradient all-reduce operation.

[0118] Thus, on the basis of the expanded model scale, the data throughput is further expanded. By processing larger global batch sizes, the training speed is accelerated, and the resource utilization is improved. Especially when combined with expert parallelism, data parallelism can be performed within each expert parallel group, effectively increasing the concurrency of the training task.

[0119] In some embodiments, the distributed training framework also supports a tensor parallelism strategy.

[0120] Here, the tensor parallelism strategy (Tensor Parallelism Strategy) is a form of model parallelism. It splits a single large weight tensor in the row or column dimension and distributes it to multiple computing devices. Each device only holds a part of the tensor, and communication with other devices is needed to complete the complete matrix operation. For example, a linear layer weight matrix of size 4096x4096 is split by column into 4 GPUs, each holding a 4096x1024 slice.

[0121] In some implementations, for very large single layers in the model (such as super-large-scale expert FFN layers or attention layers), the framework supports automatic splitting of their weight tensors and distributing them to a group of devices. During forward and backward propagation, the framework manages the communication between these devices (such as All-Reduce, global data aggregation distribution (All-Gather)) to complete the correct calculation.

[0122] In this way, the distributed training framework also supports the tensor parallelism strategy, breaking through the limitation of the maximum model component size by the single device memory. It makes it possible to train models with extremely wide FFN layers or extremely large attention heads. It is a supplement to expert parallelism, specifically designed to solve the internal memory problem of "large experts" or "large attention layers".

[0123] In some embodiments, the distributed training framework also supports a pipeline parallelism strategy.

[0124] Here, the pipeline parallelism strategy (Pipeline Parallelism Strategy) is another form of model parallelism. It splits the model into multiple stages (Stage) according to the layer depth, and each stage is placed on a different computing device. Input data is calculated by each stage device in turn like a pipeline. For example, a 24-layer Transformer model uses pipeline parallelism on 4 GPUs, and each GPU may be responsible for the calculation of 6 consecutive layers.

[0125] In some embodiments, the framework supports splitting the non-MoE part of the model in the depth direction into multiple stages, each of which is placed on a different device group. The data transmission between stages is managed by a scheduling algorithm, such as micro-batch scheduling or asynchronous scheduling, to overlap computation and communication.

[0126] In this way, the distributed training framework also supports pipeline parallelism strategy, mainly solving the memory and computing limitations in the depth direction of the model. This allows the model to have very deep layers without being limited by the memory of a single device. It combines with expert parallelism (processing width expansion) and tensor parallelism (processing width / height expansion) to achieve comprehensive expansion of the model in three dimensions (depth, width, height).

[0127] In some embodiments, the expert parallelism strategy is used in combination with at least one of data parallelism, tensor parallelism, and pipeline parallelism strategy.

[0128] Here, Combined Use refers to simultaneously using two or more parallel strategies in the same training task to address the complex constraints of memory, computation, and communication faced in training super-large-scale models. For example, first use tensor parallelism to distribute the parameters of a huge expert across a group of GPUs, then use expert parallelism to distribute different expert groups to different device groups, and finally use data parallelism to replicate the entire structure to handle larger batches.

[0129] In some embodiments, the distributed training framework allows users to flexibly combine multiple parallel dimensions through configuration files or programming interfaces. For example, a parallel scheme can be configured: first use pipeline parallelism to split the model in the depth direction, then use expert parallelism to distribute the MoE layer experts in each pipeline stage, and finally use data parallelism to increase data throughput within each expert parallel group. The framework is responsible for coordinating the complex communication between all device groups.

[0130] In this way, by combining multiple strategies, the challenges of huge model parameter size, huge single component size, extremely deep model depth, and high data batch requirement can be addressed simultaneously. This combination fully leverages the complementary advantages of various parallel strategies, making it possible to train trillion-parameter-level models, and is a key technology to support the continued growth of future AI model sizes.

[0131] In some embodiments, the distributed training framework is used to fine-tune the quantized MoE model, including: when aggregating the gradients generated during the distributed training process, the gradients of the quantized parameters and the LoRA adapter parameters are optimized to adapt to low-bit precision.

[0132] Here, Gradient Aggregation is a process in which, in distributed data parallel training, the gradients computed by each computing device based on local data need to be collected and combined into a global gradient through some operation (such as averaging) for updating the model. This process is called gradient aggregation, which is usually implemented through an All-Reduce operation. For example, in 4-GPU data parallel training, each GPU computes a local gradient tensor, denoted as g1, g2, g3, and g4. Through the All-Reduce operation, the global average gradient g = (g1+g2+g3+g4) / 4 is obtained, and then g is used to update the model parameters on all GPUs.

[0133] Here, the quantization parameter refers to a parameter generated in the quantization process, which is used to describe how to dequantize a low-bit integer representation into a high-precision floating-point number. The most important ones are scale and zero-point. These parameters themselves may be differentiable in training and need to participate in gradient update. For example, for a quantized weight tensor, its gradient may not only contain the gradient of the weight, but also contain the gradient of the scale and zero-point, especially when using quantization-aware training.

[0134] In some embodiments, when aggregating the gradients generated in the distributed training process, the gradients of the quantization parameters and the LoRA adapter parameters are optimized. In the gradient All-Reduce communication link of distributed training, the framework identifies the gradients from the quantization parameters (such as Scale / Zero-Point) and the LoRA parameters. These gradients are optimized using dynamic gradient clipping or special precision processing (such as maintaining FP32 precision for aggregation). For example, since the LoRA gradient is usually small, and the gradient of the quantization parameter may be sensitive, a clipping threshold independent of the main model weight can be set for it to prevent gradient explosion or underflow and ensure numerical stability.

[0135] In this way, the stability and convergence of low-bit mixed fine-tuning training are significantly improved. By optimizing the gradients of specific parameters, the problems of gradient mismatch, numerical overflow or underflow that may be introduced by the coexistence of quantization-aware training and low-rank adapter are effectively avoided, providing a key guarantee for the smooth training of the entire complex system.

[0136] In some embodiments, obtaining the target MoE model includes: after the fine-tuning training is completed, fusing the quantized weight parameters and the LoRA adapter parameters to generate final fine-tuned model parameters.

[0137] Here, LoRA Adapter Parameters refer to the elements in trainable low-rank matrices A and B introduced in LoRA technique. These parameters are updated through gradient descent during training process.

[0138] Parameter Fusion refers to the operation of merging multiple independent parameters into a single parameter after training is completed. Here, it refers to the operation of merging the frozen quantized weight matrix W quant and the trained low-rank delta BA into a new weight matrix W final , i.e.:

[0139] W final = D(W quant ) + BA.

[0140] Where D(·) represents the dequantization operation. The fused W final can be re-quantized to INT8 or other formats, or stored in FP16 / BF16 or other floating-point formats.

[0141] Here, the final fine-tuned model parameters refer to the model weights obtained after the complete low-bit fine-tuning process, which can be used for inference and deployment. It no longer contains independent LoRA adapters, but integrates all adjustments into a unified model file.

[0142] In some specific embodiments, after the fine-tuning training is completed, an offline model conversion step is performed: 1. Reading: loading the frozen, quantized main weight W quant and its quantization parameters, as well as the trained LoRA weights A and B; 2. Dequantization and merging: dequantizing W quant to high-precision format (such as FP16), and then adding it to the low-rank delta BA; 3. Output: outputting W final as the final weight of the linear layer. W final can be re-quantized to the target precision (such as INT8) for maximum deployment performance, or saved directly in FP16 format.

[0143] In this way, in addition to the calculation overhead and performance loss during inference, zero-delay deployment is achieved. The fused model is a standard neural network that does not require any additional conditional branches (such as judgment in forward calculation), and can be directly deployed using highly optimized inference engines, avoiding the delay caused by dynamically loading adapters or performing addition operations during inference, while maintaining all performance improvements brought by fine-tuning.

[0144] Figure 2 A MoE hierarchical low-bit quantization computing architecture diagram integrating LoRA fine-tuning is shown. Specifically, Figure 2 The left side gives a differentiated low-bit weight quantization strategy for MoE. According to the functional characteristics and precision sensitivity of different components of the model, adaptive quantization configuration is implemented. Specifically, the non-MoE module (linear transformation part) includes linear weights such as query, key, value projection (qkv_proj) in the self-attention mechanism, self-attention calculation itself (Self Attention), and output projection (o_proj), which adopts channel-level INT8 quantization. This strategy significantly reduces the memory occupation while ensuring the accuracy of the core attention mechanism calculation of the model, achieving a good balance between accuracy and compression efficiency. MoE-related modules, including MoE gating and distribution, internal calculation of each expert module (Experts), and MoE output integration, adopt channel-level INT4 quantization. Since the expert module has large parameter quantity and relatively independent function, it has high tolerance to quantization noise, so using lower-bit INT4 quantization can greatly compress the parameter size and significantly improve the storage and calculation efficiency, while the synchronous quantization of the gating and integration modules ensures the overall performance and stability of the MoE computation graph. This hierarchical quantization method relies on channel-level granularity to fine-tune the quantization error, maintaining the model's expressive ability while maximizing the reduction of deployment resource consumption, making it suitable for efficient fine-tuning and inference deployment of large-scale pre-trained MoE models.

[0145] Specifically, Figure 2 The lower right shows a mixed-precision computing data flow integrating LoRA fine-tuning technology. The input vector x is fed into two parallel computing paths:

[0146] 1. Main quantized weight path: x is multiplied by the original weight matrix W quant that has undergone channel-level INT4 / INT8 quantization, to obtain W quant x.

[0147] 2. LoRA adaptation path: x is first multiplied by the low-rank matrix A (LoRA-A), and the output is then multiplied by the low-rank matrix B (LoRA-B) to obtain the low-rank update term B(Ax).

[0148] Finally, the output y is the sum of the results of the two paths:

[0149] y = W quant x + B(Ax)

[0150] This structure shows that the model uses both the quantized frozen weights and the lightweight LoRA adapter during inference, retaining the efficiency of the original model parameters while flexibly injecting downstream task knowledge through the additional low-rank adaptation module, achieving both deployment efficiency and fine-tuning flexibility.

[0151] It should be understood that Figure 2 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 2 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0152] This disclosure provides a fine-tuning training device for MoE, such as... Figure 3 As shown, the device may include: a structure analysis module 301, used to perform model structure analysis on the MoE model to be fine-tuned, and identify the expert modules and non-MoE modules within it; a quantization processing module 302, used to quantize the weight parameters of the expert modules using a first low bit precision, and to quantize the weight parameters of the non-MoE modules using a second low bit precision, wherein the first low bit precision is lower than the second low bit precision; a computation control module 303, used to perform forward and backward propagation computations of the model based on the quantized weights using a mixed precision computation operator; and a fine-tuning training module 304, used to fine-tune and train the quantized MoE model using a distributed training framework based on LoRA fine-tuning technology to obtain the target MoE model.

[0153] In some embodiments, the first low bit precision is INT4 precision, and the second low bit precision is INT8 precision.

[0154] In some embodiments, the quantization processing module 302 includes: a first quantization processing submodule, configured to: obtain a first weight matrix of the expert module; independently calculate a first quantization parameter for each channel of the weight matrix according to the channel dimension; and perform INT4 quantization on the first weight matrix based on the first quantization parameter calculated independently for each channel.

[0155] In some embodiments, the quantization processing module 302 includes: a second quantization processing submodule, configured to: obtain a second weight matrix of a non-MOE module; calculate a second quantization parameter for the second weight matrix by channel dimension or tensor level; and perform INT8 quantization on the second weight matrix based on the second quantization parameter.

[0156] In some embodiments, the calculation control module 303 is specifically used to: during the forward propagation process, take the activation tensor in FP16 or BF16 format and the weight tensor in quantized INT8 or INT4 format as input; call the mixed precision GEMM operator to perform calculations to obtain the output tensor in FP16 or BF16 format.

[0157] In some embodiments, when performing calculations, the mixed-precision GEMM operator first dequantizes the quantized weight tensor in INT8 or INT4 format to FP16 or BF16 format, and then performs matrix multiplication with the activation tensor in FP16 or BF16 format.

[0158] In some embodiments, the mixed-precision GEMM operator directly uses quantized weight tensors in INT8 or INT4 format and activation tensors in FP16 or BF16 format to perform mixed-precision matrix multiplication calculation on a hardware acceleration unit.

[0159] In some embodiments, the fine-tuning training module 304 includes a matrix addition submodule for adding a low-rank decomposition matrix to a linear layer weight matrix in the MoE model; and a first training submodule for updating only the low-rank decomposition matrix during the fine-tuning training process, while freezing the original quantized weight parameters.

[0160] In some embodiments, the matrix addition submodule is configured to add a LoRA adapter to both the linear layer within the expert module and the linear layer within the non-MoE module.

[0161] In some embodiments, the distributed training framework supports an expert parallel strategy.

[0162] In some embodiments, the expert parallel strategy includes distributing different expert modules within the MoE model to different computing devices; and dynamically routing input data to the computing devices storing corresponding experts for calculation according to the routing results output by the gating network during the forward propagation process.

[0163] In some embodiments, the distributed training framework also supports a data parallel strategy.

[0164] In some embodiments, the distributed training framework also supports a tensor parallel strategy.

[0165] In some embodiments, the distributed training framework also supports a pipeline parallel strategy.

[0166] In some embodiments, the expert parallel strategy is used in combination with at least one of the data parallel, tensor parallel, and pipeline parallel strategies.

[0167] In some embodiments, the fine-tuning training module 304 includes a second training submodule for optimizing the gradients of the quantized parameters and the LoRA adapter parameters to adapt to low-bit precision when aggregating the gradients generated during the distributed training process.

[0168] In some embodiments, the fine-tuning training module 304 further includes a model determination submodule for fusing the quantized weight parameters and the LoRA adapter parameters to generate final fine-tuned model parameters after the fine-tuning training is completed.

[0169] The specific functions and examples of the modules and submodules of the apparatuses of the embodiments of the present disclosure are described above in the related description of the corresponding steps in the method embodiments, which will not be described here again.

[0170] The MoE fine-tuning training device in this embodiment achieves multiple technical improvements in distributed MoE training by integrating differentiated low-bit quantization, high-performance mixed-precision computing, and efficient parameter fine-tuning techniques: 1. Significantly reduced memory usage: By using differentiated quantization for expert modules and non-MoE modules (e.g., expert modules use INT4, non-MoE modules use INT8), extreme compression of model parameters is achieved, significantly reducing memory consumption during training and making it possible to train ultra-large-scale models with limited hardware resources. 2. Effectively improved training efficiency: Through customized mixed-precision GEMM operators and deep optimization of the distributed training framework, the hardware computing potential is fully utilized, accelerating the forward and backward propagation processes. Simultaneously, strategies such as expert parallelism maximize the use of computing resources, significantly shortening model training time. 3. Overcoming model size limitations: Combined with LoRA fine-tuning technology, only a small number of parameters need to be trained to adapt to downstream tasks, further reducing resource requirements; coupled with various distributed parallel strategies (expert parallelism, data parallelism, etc.), it supports efficient training and deployment of MoE models with hundreds of billions of parameters. 4. Maintain excellent model performance: While improving efficiency, numerical accuracy is ensured through techniques such as channel-level quantization and mixed-precision computation, enabling the quantized model to maintain performance comparable to the full-precision model in downstream tasks. 5. Enhance system usability and scalability: This solution allows for efficient fine-tuning without modifying the core model architecture, exhibiting good hardware compatibility and algorithmic scalability, providing a reliable technical path for the widespread deployment of large-scale AI models.

[0171] This disclosure provides a scenario illustration of a MoE fine-tuning training method, as shown in the following embodiments. Figure 4 As shown.

[0172] As previously described, the MoE fine-tuning training method provided in this disclosure is applied to electronic devices. These electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0173] Specifically, the electronic device may perform the following operations:

[0174] The model structure of the fine-tuned MoE model is analyzed, and the expert modules and non-MoE modules inside the model are identified; the weight parameters of the expert modules are quantized using a first low-bit precision, and the weight parameters of the non-MoE modules are quantized using a second low-bit precision, wherein the first low-bit precision is lower than the second low-bit precision; the forward calculation and back propagation calculation of the model based on the quantized weights are performed using a mixed precision calculation operator; based on the LoRA fine-tuning technology, the fine-tuned training of the quantized MoE model is performed using a distributed training framework, and a target MoE model is obtained.

[0175] It should be understood that Figure 4 The scenario shown is merely illustrative and not limiting, and those skilled in the art can make various obvious changes and / or replacements based on the examples Figure 4 The resulting technical solutions still belong to the disclosure range of the embodiments of the present disclosure.

[0176] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

[0177] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0178] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0179] As Figure 5 shown, the device 500 includes a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in a Read-Only Memory (ROM) 502 or a computer program loaded into a Random Access Memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0180] A plurality of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0181] The computing unit 501 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various special-purpose Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the fine-tuning training method of MoE. For example, in some embodiments, the fine-tuning training method of MoE can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded to the RAM 503 and executed by the computing unit 501, one or more steps of the fine-tuning training method of MoE described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the fine-tuning training method of MoE by any other appropriate means, such as by means of firmware.

[0182] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system on chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0183] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general or special purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0184] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a Cathode Ray Tube (CRT) or Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0186] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0188] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps described above. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0189] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A fine-tuning training method of MoE, wherein, The method comprises the following steps: Performing model structure analysis on the fine-tuned MoE model to identify the expert modules and non-MoE modules inside the model; Quantizing the weight parameters of the expert modules using a first low-bit precision and quantizing the weight parameters of the non-MoE modules using a second low-bit precision, wherein the first low-bit precision is lower than the second low-bit precision; Performing forward calculation and back propagation calculation based on the quantized weights using a mixed-precision calculation operator; Fine-tuning the quantized MoE model using a distributed training framework based on the low-rank adaptive LoRA fine-tuning technology to obtain a target MoE model.

2. The method of claim 1, wherein, The first low-bit precision is INT4 precision, and the second low-bit precision is INT8 precision.

3. The method of claim 1 or 2, wherein, The quantization of the weight parameters of the expert modules using the first low-bit precision comprises the following steps: Obtaining a first weight matrix of the expert modules; Independently calculating a first quantization parameter for each channel of the weight matrix in the channel dimension; Performing INT4 quantization on the first weight matrix based on the first quantization parameter independently calculated for each channel.

4. The method of claim 1 or 2, wherein, The quantization of the weight parameters of the non-MoE modules using the second low-bit precision comprises the following steps: Obtaining a second weight matrix of the non-MoE modules; Calculating a second quantization parameter for the second weight matrix in the channel dimension or tensor level; Performing INT8 quantization on the second weight matrix based on the second quantization parameter.

5. The method of claim 1, wherein, The use of the mixed-precision calculation operator to perform forward calculation based on the quantized weights comprises the following steps: In the forward propagation process, the activation tensor in FP16 or BF16 format and the quantized weight tensor in INT8 or INT4 format are used as inputs; A mixed-precision GEMM operator is called as the mixed-precision calculation operator to perform calculation to obtain an output tensor in FP16 or BF16 format.

6. The method of claim 5, wherein, The mixed-precision GEMM operator first dequantizes the quantized weight tensor in INT8 or INT4 format to FP16 or BF16 format before performing matrix multiplication calculation with the activation tensor in FP16 or BF16 format.

7. The method of claim 5, wherein, The mixed-precision GEMM operator directly uses the quantized weight tensor in INT8 or INT4 format and the activation tensor in FP16 or BF16 format to perform mixed-precision matrix multiplication calculation on the hardware acceleration unit.

8. The method of claim 1, wherein, Based on the LoRA fine-tuning technology, the method comprises the following steps: Adding a low-rank decomposition matrix to the linear layer weight matrix in the MoE model; In the fine-tuning training process, only the low-rank decomposition matrix is updated, and the original quantized weight parameters are frozen.

9. The method of claim 8, wherein, The addition of the low-rank decomposition matrix comprises the following steps: Adding a LoRA adapter to the linear layer in the expert module and the linear layer in the non-MoE module.

10. The method of claim 1, wherein, The distributed training framework supports an expert parallel strategy.

11. The method of claim 10, wherein, The expert parallel strategy comprises the following steps: Distributing different expert modules in the MoE model to different computing devices; In the forward propagation process, the input data is dynamically routed to the computing device storing the corresponding expert according to the routing result output by the gating network for calculation.

12. The method of claim 1, wherein, The distributed training framework also supports a data parallel strategy.

13. The method of claim 1, wherein, The distributed training framework also supports a tensor parallel strategy.

14. The method of claim 1, wherein, The distributed training framework also supports a pipeline parallel strategy.

15. The method of claim 10, wherein, The expert parallel strategy is used in combination with at least one of a data parallel, a tensor parallel, and a pipeline parallel strategy.

16. The method of claim 1, wherein, The fine-tuning training of the quantized MoE model by using the distributed training framework comprises: When aggregating the gradients generated in the distributed training process, the gradients of the quantized parameters and the LoRA adapter parameters are optimized to adapt to low-bit precision.

17. The method of claim 1, wherein, The target MoE model is obtained by: After the fine-tuning training is completed, the quantized weight parameters and the LoRA adapter parameters are fused to generate final fine-tuned model parameters.

18. A fine-tuning training device for MoE, comprising: a structure analysis module configured to perform model structure analysis on a MoE model to be fine-tuned, and identify expert modules and non-MoE modules in the MoE model; a quantization processing module configured to quantize weight parameters of the expert modules using a first low-bit precision, and quantize weight parameters of the non-MoE modules using a second low-bit precision, wherein the first low-bit precision is lower than the second low-bit precision; a calculation control module configured to perform forward calculation and back propagation calculation of a model based on quantized weights using a mixed precision calculation operator; a fine-tuning training module configured to fine-tune train the quantized MoE model by using a distributed training framework based on a LoRA fine-tuning technology, and obtain a target MoE model.

19. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-17.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-17.

21. A computer program product comprising computer programs stored on a storage medium, the computer programs being executed by a processor to implement the method of any one of claims 1-17.

Images