Model training method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of neural network models and model training. The specific implementation scheme is as follows: a calculation unit executes quantization matrix multiplication based on Hadamard pre-transformation on an activation tensor and a weight tensor of a target model stored in a memory so as to generate an output tensor of a linear layer based on a low-precision tensor with smaller data bit width; using the output tensor and a subsequent network layer of the target model to complete forward propagation so as to obtain a loss value; and according to the loss value, updating model parameters of the target model stored in a memory through a back propagation algorithm. By means of the technical scheme, on the premise that the model training precision is guaranteed, memory resource occupation and the calculation amount in the calculation process can be remarkably reduced, and the training cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of neural network models and model training technology. Background Art

[0002] In recent years, artificial intelligence technologies, exemplified by large-scale pre-trained models, have achieved breakthroughs in key areas such as natural language processing, computer vision, and multimodality. In pursuit of higher model performance, the number of model parameters and the required computing power continue to climb, posing significant challenges to computing hardware, particularly the memory capacity, computing bandwidth, and training time of graphics processing units (GPUs). Currently, mainstream large-scale model training typically requires the use of massive computing clusters comprised of hundreds or even thousands of GPU cards. The high cost of training has become a major bottleneck restricting the development and widespread application of large-scale model technologies. Summary of the Invention

[0003] The present disclosure provides a model training method, apparatus, device, and storage medium.

[0004] According to one aspect of the present disclosure, a model training method is provided, which is applied to a computing system including a computing unit and a memory, the method comprising:

[0005] The computing unit performs a quantization matrix multiplication based on a Hadamard pre-transform on the activation tensor and the weight tensor of the target model stored in the memory, so as to generate an output tensor of the linear layer based on a low-precision tensor with a smaller data bit width;

[0006] Complete forward propagation using the output tensor and subsequent network layers of the target model to obtain the loss value;

[0007] According to the loss value, the model parameters of the target model stored in the memory are updated through the back propagation algorithm.

[0008] According to another aspect of the present disclosure, there is provided a model training device, comprising:

[0009] The quantization module is used to perform quantization matrix multiplication based on Hadamard pre-transformation on the activation tensor and weight tensor of the target model to obtain the output tensor of the linear layer;

[0010] The forward propagation module is used to complete the forward propagation using the output tensor and the subsequent network layers of the target model to obtain the loss value;

[0011] The back propagation module is used to adjust the model parameters of the target model through the back propagation algorithm according to the loss value.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0018] By utilizing the technical solution disclosed in the present invention, it is possible to significantly reduce the memory resource usage and computational complexity during the calculation process while ensuring the accuracy of model training, thereby reducing training costs.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is an exemplary system architecture for applying the model training method disclosed herein;

[0022] Figure 2 is a flowchart of a model training method provided according to an embodiment of the present disclosure;

[0023] Figure 3 is a structural diagram of a model training device provided according to an embodiment of the present disclosure;

[0024] Figure 4 A block diagram of an electronic device used to implement the model training method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] To reduce training resource consumption, the use of low-bit numerical formats (such as 8-bit floating-point FP8) for model training has been widely researched. Due to its extremely small bit width, the FP8 format can theoretically significantly reduce graphics memory usage and data transmission bandwidth, making it a promising low-resource training solution for large models.

[0027] Currently, existing FP8 training technologies mainly revolve around different quantization strategies, but these existing technologies have obvious limitations in practical applications.

[0028] A mainstream approach is tensor-wise static quantization. This method quantizes the entire tensor using a single scaling factor. While simple to implement and hardware-friendly, its fundamental drawback is its limited fitting capabilities. When the data within a tensor is unevenly distributed or has a large dynamic range (for example, after an activation function), a single scaling factor can lead to significant quantization errors and severe precision loss. Therefore, this method struggles to meet the demands of complex tasks and is particularly unsuitable for tasks such as fine-tuning large models or inputting long sequences with demanding precision.

[0029] To address the above-mentioned accuracy issues, researchers proposed a finer-grained block-wise dynamic quantization scheme. This scheme does improve quantization accuracy and adaptability to data changes by dividing the tensor into blocks and dynamically calculating independent scaling factors for each block. However, the shortcomings of this method are also very prominent. First, the dynamic calculation of quantization parameters for a large number of small blocks brings significant additional computing overhead, which slows down the training speed. Secondly, in the inference deployment stage, since the quantization factors of the activation values ​​need to be generated in real time based on the input, this dynamic and fine-grained feature makes it difficult to use existing hardware for efficient inference acceleration, losing the performance advantages that the low-bit format should have on the inference side.

[0030] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the embodiments of the present disclosure provide a model training method. By utilizing the technical solutions of the embodiments of the present disclosure, it is possible to maintain high quantization accuracy while also having training simplicity and inference friendliness.

[0031] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the model training method, apparatus, electronic device, and computer-readable storage medium of the present disclosure may be applied.

[0032] like Figure 1 As shown, system architecture 100 may include one or more servers (e.g., servers 101 to 105), and a network for communication between servers. These servers are typically high-performance computing nodes equipped with accelerators such as high-performance processors (CPUs) and graphics processing units (GPUs). The network is used to provide high-speed, low-latency communication links between nodes in the server cluster and can include various connection types, such as high-speed Ethernet or fiber optic cables, which is crucial for large-scale distributed training.

[0033] One or more servers can work together and interact over a network to perform large-scale model training tasks. For example, an application for implementing the disclosed model training method can be installed on the server, embedding core logic such as the Hadamard pre-transform-based quantization matrix multiplication, the phased scaling factor strategy, and the optimizer state offload described herein. The target model and the massive training data required for training can be stored in a distributed file system or storage device connected to the server.

[0034] In one embodiment of the present disclosure, when the number of target model parameters to be trained is huge and cannot be completed on a single server node, the following can be used: Figure 1 The distributed cluster system shown in the figure is used for training. The distributed cluster system includes multiple computing nodes (such as server 101, server 102, server 103, server 104, and server 105 in the figure). Multiple nodes can jointly execute a large-scale model training task based on specific distributed training strategies (such as data parallelism, tensor parallelism, pipeline parallelism, etc.).

[0035] Specifically, in this distributed cluster system, each node can be responsible for a portion of the training process. For example, in data parallel mode, each node holds a complete copy of the model and processes a portion of the training data. After each training iteration, efficient data exchange between multiple nodes is required to synchronize the model state.

[0036] Crucially, this data exchange process is closely related to the technical solution of the present disclosure. For example, after performing a quantization operation based on a Hadamard pre-transform, according to the technical solution of the present disclosure, the weight scaling factors and activation scaling factors calculated independently on each node need to be synchronized through the network (for example, performing an All-Reduce aggregation operation) to ensure that all nodes use completely consistent quantization standards in subsequent calculations. In addition, after backpropagation is completed, the gradient information calculated by each node also needs to be synchronized through the network before a unified parameter update is performed on each node.

[0037] In addition, the server nodes in this system architecture also support asynchronous unloading of optimizer states (such as full-precision model parameters) from GPU memory to the node's CPU main memory or locked-page memory, and data interaction with CPU memory through the network, thereby effectively managing and optimizing the use of GPU video memory and supporting the training of larger-scale models.

[0038] Figure 2 1 is a flow chart of a model training method according to an embodiment of the present disclosure. The model training method can be applied to a computing system including a computing unit and a memory, and the computing system can adopt the system architecture 100 provided in the above embodiment. Figure 2 As shown, the method comprises at least the following steps:

[0039] S210: The computing unit performs Hadamard pre-transformation-based quantization matrix multiplication on the activation tensor and weight tensor of the target model stored in the memory to generate an output tensor of the linear layer based on a low-precision tensor with a smaller data bit width.

[0040] In this step, the computing unit (such as a graphics processing unit GPU or a neural network processing unit NPU) obtains the activation tensors and weight tensors to be processed from the memory (such as high-bandwidth memory HBM or system memory DRAM). These original tensors are usually stored in a format with higher precision (such as bfloat16). When executing this step, the computing unit first performs a Hadamard pre-transform and then performs a quantization operation to convert the tensor from a higher-precision format to a lower-precision tensor with a smaller data bit width (such as FP8 format).

[0041] The converted low-precision tensor has a smaller bit width and occupies significantly less storage space. This allows more computational data to be stored in the high-speed but limited on-chip memory (SRAM) within the computing unit, reducing the frequency of access to main memory.

[0042] When performing matrix multiplication, the compute unit needs to feed data from its internal storage to the arithmetic logic unit (ALU) or dedicated compute cores (such as Tensor Cores). As the data bit width is reduced, the total amount of data transfer required to complete the same scale of calculations is also proportionally reduced, greatly alleviating the memory bandwidth bottleneck and improving data flow efficiency.

[0043] Compute units typically contain hardware specifically designed for low-precision operations. Performing matrix multiplication of low-precision tensors allows these hardware units to achieve much higher throughput than high-precision calculations, thereby accelerating the computation process itself.

[0044] Finally, the calculation result is restored to a higher precision through the dequantization operation to form the output tensor of the linear layer.

[0045] The target model can be a deep learning network to be trained, such as a large-scale language model. Activation tensors are intermediate results generated by neural network layers during the model's forward propagation. Weight tensors are parameters in the model that need to be learned through training.

[0046] S220: Complete forward propagation using the output tensor and subsequent network layers of the target model to obtain the loss value.

[0047] Forward propagation is the process of inputting data into the model and calculating the final prediction layer by layer. The output tensor obtained in S210 is passed as input to the next layer of the model. After all network layers have completed calculations, the model's predictions are compared with the ground truth labels. A loss function is used to calculate a loss value, which measures the accuracy of the current model's predictions.

[0048] S230: Based on the loss value, update the model parameters of the target model stored in the memory through the back propagation algorithm.

[0049] The backpropagation algorithm is a core algorithm that calculates gradient information based on the loss value. The gradient indicates how the model parameters should be adjusted to reduce the loss value. This algorithm can efficiently calculate the partial derivatives of the loss with respect to all weight tensors in the model. This gradient information is then used to update the model parameters (i.e., the main weights) stored in memory, typically in full-precision form (such as FP32), thereby gradually improving model performance.

[0050] According to the solution of the embodiment of the present disclosure, by introducing quantization matrix multiplication based on Hadamard pre-transformation, high-precision data is converted into low-precision data for processing. At the hardware level, not only the demand for high-speed storage capacity inside the computing unit is effectively reduced, but also the pressure on data transmission bandwidth is reduced, and the high-throughput computing capability of the computing unit for low-precision data can be fully utilized. Ultimately, without sacrificing model accuracy, the training speed is improved and the overall hardware resource consumption is reduced, thereby reducing the training cost.

[0051] In one possible implementation, S210 is performed by the computing unit on the activation tensor and weight tensor of the target model stored in the memory, and the quantization matrix multiplication based on the Hadamard pre-transform is performed to generate the output tensor of the linear layer based on the low-precision tensor with a smaller data bit width, further including:

[0052] S211: Use the Hadamard matrix to pre-transform the initial activation tensor and the initial weight tensor stored in the memory to obtain a first input tensor and a first weight tensor.

[0053] In this step, the Hadamard matrix is ​​a special square matrix whose elements consist only of +1 and -1 and have good orthogonality. The initial activation tensor and initial weight tensor refer to the original working tensors with high numerical precision before the pre-transformation. The pre-transformation obtains the transformed first input tensor and first weight tensor by performing mathematical operations on these initial tensors and the Hadamard matrix. This process aims to optimize the data distribution of the tensors, making them more suitable for subsequent quantization operations.

[0054] S212: Perform a quantization operation on the first input tensor and the first weight tensor according to the scaling factor to obtain a second input tensor and a second weight tensor with a smaller data bit width.

[0055] Quantization is the process of mapping high-precision values ​​(such as 32-bit floating-point numbers) to low-precision values ​​(such as 8-bit floating-point numbers). The data width is reduced, resulting in reduced accuracy. The scaling factor is a key parameter used to scale the range of values ​​in this mapping process. Through the quantization operation, the first input tensor and the first weight tensor are converted into a second input tensor and a second weight tensor with a smaller data width.

[0056] S213: The computing unit performs matrix multiplication on the second input tensor and the second weight tensor to obtain a calculation result.

[0057] This step is the core computational step. Because both the second input tensor and the second weight tensor are in low-precision format, the computational resources and memory bandwidth required to perform matrix multiplication are far less than using high-precision tensors, thus achieving computational acceleration.

[0058] S214: Perform an inverse quantization operation on the calculation result to obtain the output tensor of the linear layer.

[0059] Dequantization is the inverse of quantization, mapping low-precision calculation results back to a high-precision numerical format so that subsequent network layers of the model can continue to perform high-precision calculations. The output tensor obtained in this step is the final output of the linear layer.

[0060] In one example, an orthogonal Hadamard matrix is ​​introduced before matrix multiplication to perform a pre-transformation operation:

[0061] y=dequant(quant(x bf16 ×HT)×quant(H×w bf16 ))

[0062] in represents the activation input feature (initial activation tensor), Represents the weight matrix, H∈{-1,+1} mxm .

[0063] According to the solution of the embodiment of the present disclosure, it is explained in detail how to decompose a complex quantization matrix multiplication into four steps: pre-transformation, quantization, low-precision calculation and inverse quantization, and demonstrates the specific path to achieve computational optimization while maintaining the equivalence of mathematical operations.

[0064] In one possible implementation, S211 uses a Hadamard matrix to pre-transform the initial activation tensor and the initial weight tensor stored in the memory to obtain a first input tensor and a first weight tensor, including the steps of:

[0065] S310: Multiply the initial activation tensor by the transposed matrix of the Hadamard matrix to obtain a first input tensor.

[0066] Specifically, if the initial activation tensor is x, the Hadamard matrix is ​​H, and its transposed matrix is ​​H:T, then the first input tensor x' is calculated by the formula x'=x*H:T.

[0067] S320: Multiply the Hadamard matrix by the initial weight tensor to obtain a first weight tensor.

[0068] If the initial weight tensor is w, the first weight tensor w' is calculated by the formula w'=H*w.

[0069] Among them, the dimension of the initial activation tensor is k×m, the dimension of the initial weight tensor is m×n, and the Hadamard matrix is ​​an m×m square matrix whose elements are selected from the set {+1,-1}.

[0070] This step defines the dimensionality of each matrix. k typically represents the batch size or sequence length, m represents the feature dimension, and n represents the output dimension. The Hadamard matrix's dimensions (mxm) match the number of columns in the activation tensor and the number of rows in the weight tensor, ensuring that the matrix multiplication described above is dimensionally efficient. Its elements are exclusively +1 and -1, allowing for highly optimized computations.

[0071] According to the solution of the embodiment of the present disclosure, the embodiment clarifies the specific mathematical form of the pre-transformation, ensures the operability and uniqueness of the transformation process, and ensures the correct execution of the entire calculation process through precise matching of dimensions.

[0072] In one possible implementation, the initial activation tensor, the initial weight tensor, and the output tensor use a first precision numerical format, and the second input tensor and the second weight tensor use a second precision numerical format. The first precision numerical format is a numerical format with higher representation precision, such as bfloat16 (BF16) or FP16. The second precision numerical format is a numerical format with a smaller bit width and lower representation precision, such as FP8.

[0073] Both the first-precision numerical format and the second-precision numerical format are floating-point formats, and the precision of the first-precision numerical format is greater than that of the second-precision numerical format. For example, BF16 (16 bits) has higher precision than FP8 (8 bits). During training, most calculations occur in the low-precision second-precision numerical format, while data is transferred between layers in the higher-precision first-precision numerical format, thereby maximizing computational efficiency while ensuring the accuracy of information flow.

[0074] According to the solution of the embodiment of this disclosure, by defining numerical formats with different precisions at different stages, a framework for mixed-precision computing is constructed. This can maintain the overall stability and convergence accuracy of the model by retaining higher-precision intermediate states while taking advantage of the performance advantages brought by low-precision computing.

[0075] In one possible implementation, the Hadamard matrix is ​​constructed by concatenating one or more standard Hadamard sub-matrices in a block-like diagonal fashion.

[0076] In the disclosed embodiment, the Hadamard matrix is ​​constructed by splicing one or more standard Hadamard submatrices in a block diagonal form. A standard Hadamard matrix refers to a Hadamard matrix whose order (dimension) is a power of 2 (such as 2, 4, 8, 16...). When the characteristic dimension m of the required transformation is not a power of 2, a large block diagonal matrix can be formed by splicing multiple smaller standard Hadamard submatrices along the main diagonal. For example, a Hadamard matrix with a dimension of 12 can be constructed by splicing an 8x8 and a 4x4 submatrix. For another example, k identical mxm Hadamard submatrices H can be used. m To construct H.

[0077]

[0078] According to the solution of the embodiment of the present disclosure, this embodiment provides a flexible and scalable Hadamard matrix construction method. This method enables the pre-transformation technique to be applied to tensor calculations of any dimension, not just power-of-two dimensions, greatly enhancing the versatility and practicality of the method.

[0079] In a possible implementation, the model training method of the embodiment of the present disclosure further includes the steps of:

[0080] S410: Configuring the optimizer so that the momentum terms of the first-order moment and the second-order moment are stored in a first-precision numerical format.

[0081] The optimizer is the component responsible for updating model parameters during training, such as the AdamW optimizer. First-order and second-order moments are momentum states used internally by adaptive optimizers like AdamW to track gradient history. Traditionally, these states are stored in FP32 format. This step configures their storage format to a first-precision numeric format (such as bfloat16), which has half the bit width of FP32.

[0082] According to the solution of the disclosed embodiment, by reducing the storage precision of the optimizer state, video memory usage can be directly reduced by nearly half, further alleviating the video memory bottleneck. Because the numerical dynamic range of bfloat16 is similar to that of FP32, this configuration significantly saves video memory while barely affecting the convergence stability of the model.

[0083] In one possible implementation, based on the previous embodiment, step S230 further includes adjusting the model parameters of the target model through the back propagation algorithm according to the loss value:

[0084] S510: Based on the loss value, use a back propagation algorithm to determine the gradient information of the loss with respect to multiple parameters in the target model.

[0085] S520: Using the optimizer, update the learnable parameters of the target model according to the gradient information.

[0086] Learnable parameters are weight tensors. The optimizer combines gradient information and its internal state (such as first-order moments and second-order moments) to calculate new parameter values ​​according to its specific update rules (such as the AdamW update formula).

[0087] The gradient information is stored in a first-precision numerical format.

[0088] This step limits the storage format of the gradient information to, for example, bfloat16. This matches the format of the momentum term in S410, allowing the update calculation to be performed with uniform precision while also reducing video memory usage compared to traditional FP32 gradients.

[0089] According to the solution of the embodiment of the present disclosure, by unifying the storage precision of the momentum term and gradient information, the calculation process is simplified, and the peak value of the video memory during the training process is further reduced, thereby improving the overall resource utilization efficiency.

[0090] In a possible implementation, based on the previous embodiment, the following steps are further included:

[0091] S530: After the parameter update is completed, the latest state of the optimizer is asynchronously unloaded to the CPU or page-locked memory.

[0092] Asynchronous offloading is a technique that transfers data from GPU memory to CPU main memory. This operation is initiated immediately after the parameter update calculation is completed and is performed asynchronously, meaning that the data transfer can proceed in parallel with the next round of GPU calculation, thus hiding the transmission latency.

[0093] The latest state of the optimizer includes the full-precision model parameters.

[0094] Full-precision model parameters are a highly accurate copy of the parameters in FP32 format, often referred to as the master weight. During mixed-precision training, the optimizer always updates this FP32 master weight to avoid precision loss. Offloading this master weight, which occupies the largest space, along with other optimizer states, maximizes GPU memory usage.

[0095] The solution in the disclosed embodiments introduces an asynchronous offloading mechanism for optimizer state, significantly reducing GPU memory requirements without compromising training speed. This makes it possible to train larger models on a single or small number of GPUs, and is a key technology for enabling low-resource training of large models.

[0096] In one possible implementation, S220 performs a quantization operation on the first input tensor and the first weight tensor according to the scaling factor to obtain a second input tensor and a second weight tensor with a smaller data bit width, including:

[0097] S610: Determine a weight scaling factor based on the model parameters managed by the optimizer.

[0098] The weight scaling factor is a parameter used to quantize the weight tensor. Since the weight tensor changes relatively steadily during training, its scaling factor can be calculated based on the FP32 master weight (i.e., model parameter) managed by the optimizer. For example, the maximum absolute value of all elements in the master weight tensor is taken as the scaling factor.

[0099] S620: quantize the first weight tensor into a second precision numerical format with lower precision using the weight scaling factor to obtain a second weight tensor.

[0100] This step is a specific quantization process, for example, converting the first weight tensor in bfloat16 format into the second weight tensor in FP8 format.

[0101] S630: Determine an activation scaling factor using a phased strategy according to the stage of model training.

[0102] Unlike weights, the numerical distribution of activation values ​​changes more dramatically during training. Therefore, a dynamic, staged strategy is designed to determine its scaling factor.

[0103] S640: quantize the first input tensor into a second precision numerical format using the activation scaling factor to obtain a second input tensor.

[0104] This step is similar to S620 and is to quantize the activation tensor.

[0105] According to the solution of the embodiment of this disclosure, this embodiment distinguishes the different characteristics of weights and activations and designs different scaling factor determination strategies for them. This differentiated processing method can better adapt to the dynamic changes of different tensors during training while ensuring quantization accuracy, thereby improving the stability of overall training and the final model performance.

[0106] In one possible implementation, S620 determines the activation scaling factor using a phased strategy according to the stage of model training, including:

[0107] S621: When the model training is in the first stage, real-time dynamic statistics are performed on the first input tensor of the current training batch to obtain a target statistical value.

[0108] The first phase typically refers to the early stages of training. During this phase, model parameters are not yet stable, and the distribution of activation values ​​fluctuates significantly. Real-time dynamic statistics directly analyze the numerical characteristics of the current input data batch (mini-batch). For example, calculating the maximum absolute value of all elements in the first input tensor is used. This maximum value serves as the target statistic. This early phase can be the first 10% or 20% of the entire training process, for example.

[0109] S622: Determine an activation scaling factor based on the target statistical value.

[0110] Use the target statistic (such as abs_max) directly as the activation scaling factor for the current batch.

[0111] According to the solution of the embodiment of the present disclosure, the method of this embodiment can quickly respond to changes in the distribution of activation values ​​in the early stages of training, providing a dynamically adjusted quantization range for the model. This helps the model explore and converge in the initial stage.

[0112] In one possible implementation, S620 determines the activation scaling factor using a phased strategy according to the stage of model training, including:

[0113] S623: When the model training is in the second stage, determine the historical statistical value of the activation scaling factor and the dynamic statistical value of the current batch.

[0114] The second stage generally refers to the middle and late stages of training, when the model tends to stabilize. The middle and late stages can be the stage after the early stages, such as the 10% to 100% stage of the entire training process. The historical statistical value refers to a smoothed value that represents the range of activation values ​​over a period of time. The current batch dynamic statistics are calculated in the same manner as in Example 10.

[0115] S624: Using a preset moving average rate, perform exponential sliding average on the current batch dynamic statistical value and the historical statistical value to obtain an activation scaling factor.

[0116] Exponential Moving Average (EMA) is a technique for smoothing time series data. New dynamic statistics are weighted averaged with older historical statistics using a preset moving average ratio (e.g., 0:9) to produce a new, smoother activation scaling factor. This process effectively filters out short-term fluctuations in the activation range.

[0117] In one example, the activation scaling factor is calculated as follows:

[0118] s t =(1-α)·s stat +α·s t-1

[0119] Among them, s t is the currently active scaling factor, s stat is the dynamic statistical value of the current batch, and α is the moving average rate (e.g. 0:99).

[0120] According to the solution of the embodiment of the present disclosure, a smoother and more stable scaling factor can be provided in the later stage of training, thereby avoiding the impact of accidental jitter between data batches on the final convergence accuracy of the model and enhancing the robustness of the training process.

[0121] In one possible implementation, the model training method of the embodiment of the present disclosure further includes:

[0122] When the model is trained in distributed mode, the weight scaling factors and activation scaling factors are synchronized between different computing nodes.

[0123] Distributed training mode involves using multiple machines or GPUs to train models in parallel to accelerate the training process. In this mode, each compute node (e.g., a GPU) independently calculates the scaling factor for its own data batch. Synchronization involves aggregating the scaling factors calculated by all nodes (e.g., taking the maximum or average value) through communication (e.g., All-Reduce), ensuring that all nodes use the same scaling factor in subsequent quantization calculations.

[0124] According to the solution of the disclosed embodiment, by introducing a scaling factor synchronization mechanism, it ensures that all parallel computing tasks in a distributed training environment adhere to a unified quantization standard. This avoids gradient errors and model convergence failures caused by inconsistent quantization scales, and is key to ensuring that this method can be correctly and efficiently expanded to large-scale parallel training environments.

[0125] Figure 3 Schematic diagram of the structure of the model training device provided according to an embodiment of the present disclosure. Figure 3 As shown, the device 300 includes:

[0126] A quantization module 301 is configured to perform a quantization matrix multiplication based on a Hadamard pre-transform on the activation tensor and the weight tensor of the target model to generate an output tensor of a linear layer based on a low-precision tensor with a smaller data bit width;

[0127] A forward propagation module 302 is used to complete forward propagation using the output tensor and subsequent network layers of the target model to obtain a loss value;

[0128] The back propagation module 303 is used to adjust the model parameters of the target model through the back propagation algorithm according to the loss value.

[0129] In a possible implementation, the quantization module 301 is configured to:

[0130] Pre-transforming the initial activation tensor and the initial weight tensor stored in the memory using a Hadamard matrix to obtain a first input tensor and a first weight tensor;

[0131] Performing a quantization operation on the first input tensor and the first weight tensor according to the scaling factor to obtain a second input tensor and a second weight tensor having a smaller data bit width;

[0132] The computing unit performs matrix multiplication on the second input tensor and the second weight tensor to obtain a calculation result;

[0133] The calculation results are dequantized to obtain the output tensor of the linear layer.

[0134] In a possible implementation, the quantization module 301 is configured to:

[0135] Multiply the initial activation tensor by the transposed matrix of the Hadamard matrix to get the first input tensor;

[0136] Multiply the Hadamard matrix by the initial weight tensor to obtain the first weight tensor;

[0137] Among them, the dimension of the initial activation tensor is k×m, the dimension of the initial weight tensor is m×n, and the Hadamard matrix is ​​an m×m square matrix whose elements are selected from the set {+1,-1}.

[0138] In one possible implementation, the initial activation tensor, the initial weight tensor, and the output tensor adopt a first precision numerical format, and the second input tensor and the second weight tensor adopt a second precision numerical format; the first precision numerical format and the second precision numerical format are both floating point formats, and the precision of the first precision numerical format is greater than the precision of the second precision numerical format.

[0139] In one possible implementation, the Hadamard matrix is ​​constructed by concatenating one or more standard Hadamard sub-matrices in a block-like diagonal fashion.

[0140] In a possible implementation, the device further includes:

[0141] The configuration module is used to configure the optimizer so that the momentum terms of the first-order moment and the second-order moment are stored in the first-precision numerical format.

[0142] In one possible implementation, the back propagation module 303 is used to:

[0143] Based on the loss value, the back propagation algorithm is used to determine the gradient information of the loss to multiple parameters in the target model;

[0144] Using the optimizer, the learnable parameters of the target model are updated based on the gradient information.

[0145] The gradient information is stored in a first-precision numerical format.

[0146] In a possible implementation, an asynchronous uninstallation module is further included, configured to:

[0147] After the parameter update is completed, the latest state of the optimizer is asynchronously offloaded to the CPU or page-locked memory.

[0148] The latest state of the optimizer includes the full-precision model parameters.

[0149] In a possible implementation, the quantization module 301 is configured to:

[0150] Determine weight scaling factors based on model parameters managed by the optimizer;

[0151] quantizing the first weight tensor into a second precision numerical format with lower precision using a weight scaling factor to obtain a second weight tensor;

[0152] According to the stage of model training, a stage-by-stage strategy is used to determine the activation scaling factor;

[0153] Quantizes the first input tensor to the second precision numeric format using the activation scaling factor, resulting in the second input tensor.

[0154] In a possible implementation, the quantization module 301 is configured to:

[0155] When the model training is in the first stage, the first input tensor of the current training batch is dynamically counted in real time to obtain the target statistical value;

[0156] Based on the target statistic, an activation scaling factor is determined.

[0157] In a possible implementation, the quantization module 301 is configured to:

[0158] When the model training is in the second stage, determine the historical statistics of the activation scaling factor and the dynamic statistics of the current batch;

[0159] Using the preset moving average rate, the exponential sliding average of the current batch dynamic statistics and historical statistics is performed to obtain the activation scaling factor.

[0160] In a possible implementation, the method further includes:

[0161] The synchronization module is used to synchronize weight scaling factors and activation scaling factors between different computing nodes when the model training uses distributed training mode.

[0162] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0163] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0164] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0165] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0166] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0167] Various components in device 400 are connected to I / O interface 405, including an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0168] The computing unit 401 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the model training method. For example, in some embodiments, the model training method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the model training method in any other appropriate manner (e.g., by means of firmware).

[0169] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0174] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0175] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0176] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model training method, applied to a computing system including a computing unit and a memory, comprising: The computing unit performs a quantization matrix multiplication based on a Hadamard pre-transform on the activation tensor and the weight tensor of the target model stored in the memory, so as to generate an output tensor of a linear layer based on a low-precision tensor with a smaller data bit width; Performing forward propagation using the output tensor and subsequent network layers of the target model to obtain a loss value; According to the loss value, the model parameters of the target model stored in the memory are updated through a back propagation algorithm.

2. The method according to claim 1, wherein The computing unit performs a quantization matrix multiplication based on a Hadamard pre-transform on the activation tensor and the weight tensor of the target model stored in the memory to generate an output tensor of the linear layer based on a low-precision tensor with a smaller data bit width, including: Pre-transforming the initial activation tensor and the initial weight tensor stored in the memory using a Hadamard matrix to obtain a first input tensor and a first weight tensor; Performing a quantization operation on the first input tensor and the first weight tensor according to a scaling factor to obtain a second input tensor and a second weight tensor having a smaller data bit width; Performing matrix multiplication on the second input tensor and the second weight tensor by a computing unit to obtain a calculation result; The calculation result is subjected to an inverse quantization operation to obtain an output tensor of the linear layer.

3. The method according to claim 2, wherein: The method of pre-transforming the initial activation tensor and the initial weight tensor stored in the memory using the Hadamard matrix to obtain the first input tensor and the first weight tensor includes: Multiplying the initial activation tensor by the transposed matrix of the Hadamard matrix to obtain a first input tensor; Multiplying the Hadamard matrix by the initial weight tensor to obtain a first weight tensor; The dimension of the initial activation tensor is k×m, the dimension of the initial weight tensor is m×n, and the Hadamard matrix is ​​an m×m square matrix whose elements are selected from the set {+1,-1}.

4. The method according to claim 2 or 3, wherein: The initial activation tensor, the initial weight tensor, and the output tensor adopt a first precision numerical format, and the second input tensor and the second weight tensor adopt a second precision numerical format; The first precision numerical format and the second precision numerical format are both floating point formats, and the precision of the first precision numerical format is greater than the precision of the second precision numerical format.

5. The method according to claim 4, wherein The Hadamard matrix is ​​constructed by splicing one or more standard Hadamard sub-matrices in a block-like diagonal form.

6. The method according to claim 1, further comprising: Configure the optimizer so that the momentum terms of the first-order and second-order moments are stored in first-precision numerical format.

7. The method according to claim 6, wherein: The step of adjusting the model parameters of the target model by a back propagation algorithm according to the loss value includes: Determining, based on the loss value, gradient information of the loss with respect to multiple parameters in the target model using a back-propagation algorithm; Using an optimizer, updating the learnable parameters of the target model according to the gradient information; The gradient information is stored in a first precision numerical format.

8. The method according to claim 7, further comprising: After the parameter update is completed, the latest state of the optimizer is asynchronously unloaded to the CPU or locked page memory; The latest state of the optimizer includes full-precision model parameters.

9. The method according to claim 2, wherein: The performing a quantization operation on the first input tensor and the first weight tensor according to the scaling factor to obtain a second input tensor and a second weight tensor with a smaller data bit width includes: Determine weight scaling factors based on model parameters managed by the optimizer; quantizing the first weight tensor into a second precision numerical format with lower precision using the weight scaling factor to obtain a second weight tensor; According to the stage of model training, a stage-by-stage strategy is used to determine the activation scaling factor; The first input tensor is quantized into the second precision numerical format using the activation scaling factor to obtain a second input tensor.

10. The method according to claim 9, wherein: The activation scaling factor is determined using a phased strategy according to the stage of model training, including: When the model training is in the first stage, the first input tensor of the current training batch is dynamically counted in real time to obtain the target statistical value; Based on the target statistic, an activation scaling factor is determined.

11. The method according to claim 9, wherein The activation scaling factor is determined using a phased strategy according to the stage of model training, including: When the model training is in the second stage, determine the historical statistics of the activation scaling factor and the dynamic statistics of the current batch; The activation scaling factor is obtained by performing an exponential sliding average on the current batch dynamic statistical value and the historical statistical value using a preset moving average rate.

12. The method according to any one of claims 9 to 11, further comprising: When the model training uses a distributed training mode, the weight scaling factor and the activation scaling factor are synchronized between different computing nodes.

13. A model training device comprising: The quantization module is used to perform quantization matrix multiplication based on Hadamard pre-transformation on the activation tensor and weight tensor of the target model to generate the output tensor of the linear layer based on low-precision tensors with smaller data bit width; A forward propagation module, configured to perform forward propagation using the output tensor and subsequent network layers of the target model to obtain a loss value; A back propagation module is used to adjust the model parameters of the target model through a back propagation algorithm according to the loss value.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images