FP8 Gradient Adaptive Optimization Method and Apparatus for Training Large Language Models

Abstract

This invention discloses an FP8 gradient adaptive optimization method and apparatus for training large language models, relating to the field of computer machine learning technology. The method includes: during the iterative learning process of the large language model on user input data, the processor acquires the original high-precision gradient tensor of the current iteration step and historical information containing historical gradients stored in memory; quantizes the scaling factor and gradient values ​​according to the target FP8 format to obtain quantized input data; after filtering and denoising the quantized input data stored in memory, the optimizer's state is updated and stored in memory; and the processor obtains the system parameter update amount based on the base learning rate and system parameters using the learning mechanism corresponding to the optimizer. The large language model is optimized based on the system parameter update amount and a preset evaluation strategy to obtain a trained large language model. This method achieves an optimal balance between computer storage precision, efficiency, and system resource consumption under low-precision training of large language models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer machine learning, in particular to an FP8 gradient adaptive optimization method and device for large language model training. BACKGROUND

[0002] With the breakthrough of large language model parameter scale from hundreds of billions to trillions, the demand for computing resources (such as GPU computing power and memory bandwidth) for its training and inference increases exponentially. Traditional high-precision (such as FP32 and FP16) training mode gradually becomes a bottleneck restricting the development of model scale due to large storage occupation and low computing efficiency. To alleviate this contradiction, low-precision numerical format (FP16, BF16, especially FP8) has become a core technology direction for efficient training of large language models due to its lower memory occupation (only 1 / 4 of FP32) and higher computing throughput. However, existing low-precision training techniques face insurmountable technical barriers in practical applications, which are rooted in the inherent "low representation precision" and "narrow dynamic range" characteristics of low-precision format.

[0003] Specifically, the effective number of bits of FP8 and other low-precision formats is only 8 bits, which has a significant precision loss compared to FP32 (23 effective bits), and the dynamic range (the ratio of the maximum value to the minimum value of the representable numerical value) is only about 1 / 1000 of FP32, resulting in information loss of gradient, weight and other key parameters during storage and calculation. This characteristic directly causes the parameter update deviation of traditional optimizers (such as SGD Adam): low representation precision makes the subtle changes of gradient truncated or covered by quantization noise, and the optimizer cannot accurately capture the direction and amplitude of parameter adjustment; narrow dynamic range leads to extreme gradient values (such as large gradient truncated due to overflow and small gradient zeroed due to insufficient precision), causing gradient vanishing or explosion, and further causing the loss function to oscillate sharply or even diverge during training.

[0004] Existing technologies lack systematic solutions to this problem: most low-precision training methods simply rely on quantization algorithms to compress data format without designing adaptive mechanisms for the temporal correlation of gradients, resulting in historical gradient information being unable to effectively assist current update decisions; the learning rate scheduling and momentum update mechanisms of traditional optimizers are based on high-precision assumptions, which are prone to mechanism mismatch in low-precision scenarios - for example, the second moment estimation of Adam will produce cumulative errors under the interference of quantization noise, exacerbating the instability of parameter update. These defects collectively result in a model convergence speed that is more than 30% slower than high-precision training. Ultimately, in ImageNet classification, GLUE language understanding and other tasks, the accuracy decreases by 5%-15%, the perplexity increases by more than 20%, and the practical value of low-precision technology in large language models is severely restricted. Therefore, it is urgent to build a gradient adaptive mechanism that adapts to the characteristics of low precision, while maintaining the stability and performance of model training while ensuring resource efficiency. SUMMARY

[0005] Therefore, it is necessary to provide an FP8 gradient adaptive optimization method and device for large language model training, which can improve the learning efficiency of the large language model and balance the computer storage precision, efficiency and resource consumption.

[0006] An FP8 gradient adaptive optimization method for large language model training, the method comprising:

[0007] In the learning iteration process of the large language model on the user input data, the processor acquires the original high-precision gradient tensor of the current iteration step and the historical information containing the historical gradient stored in the memory, and the processor quantizes the scaling factor and the gradient value according to the target FP8 format to obtain quantized input data; the quantized input data is stored in the memory.

[0008] After the processor filters and denoises the quantized input data stored in the memory, the state of the optimizer is updated and stored in the memory, and the processor acquires the system parameter update amount using the learning mechanism corresponding to the optimizer according to the base learning rate and the system parameter.

[0009] The processor optimizes the large language model according to the system parameter update amount and the preset evaluation strategy to obtain a trained large language model, and uses the trained large language model to complete the gradient adaptive optimization of the input data of the next iteration step, and outputs high-precision data, wherein the intermediate data in the iteration process are stored through the memory and processed through the processor.

[0010] An FP8 gradient adaptive optimization device for large language model training, the device comprising:

[0011] A quantization module is configured to, in the learning iteration process of the large language model on the user input data, acquire, by a processor, an original high-precision gradient tensor of a current iteration step and historical information containing a historical gradient stored in a memory, and quantize, by the processor, a scaling factor and a gradient value according to a target FP8 format to obtain quantized input data; the quantized input data is stored in the memory.

[0012] A gradient optimization module is configured to, after the processor filters and denoises the quantized input data stored in the memory, update the state of the optimizer and store it in the memory, and acquire, by the processor, a system parameter update amount using a learning mechanism corresponding to the optimizer according to a base learning rate and the system parameter.

[0013] The gradient adaptive module is used for the processor to update the large language model according to the system parameter update quantity and the preset evaluation strategy optimization, to obtain a trained large language model, and to complete gradient adaptive optimization of input data of the next iteration step by using the trained large language model, and to output high-precision data, wherein the intermediate data in the iteration process are stored by the memory and processed by the processor.

[0014] The above FP8 gradient adaptive optimization method and device for large language model training firstly break through the limitation of traditional single-step gradient dependence by synchronously obtaining the current iteration step original high-precision gradient tensor and historical gradient information, and provide a data basis for processing narrow dynamic range, that is, the historical gradient, which can dynamically capture the global characteristics of gradient distribution, avoid representation deviation caused by single-step gradient fluctuation, and alleviate the problem of insufficient dynamic range under low precision. On this basis, the target FP8 format quantization scaling factor and gradient value are scaled, the key gradient information is retained by using the precisely designed scaling strategy while reducing the representation precision (FP8), the system storage and calculation resource occupation are reduced (in line with low resource demand), the information loss caused by simple low-precision quantization is avoided, and the performance decline hidden danger caused by "low representation precision" is solved. In view of the noise that may be introduced in the quantization process, the scheme purifies the input data by filtering and denoising, directly weakens the interference of noise on the parameter update of the optimizer, and alleviates the inaccurate update problem of the traditional optimizer caused by noise amplification under low precision. When updating the optimizer state, the adaptive mechanism of the basic learning rate and the system parameter is combined, so that the optimizer learning mechanism and the FP8 gradient characteristics are deeply coordinated, the mechanism mismatch of the traditional optimizer under low precision is avoided, and the numerical instability phenomena such as gradient vanishing / explosion and loss shock are reduced. Finally, the model is continuously optimized by the preset evaluation strategy, and the training results are fed back to the gradient adaptive optimization of the next iteration step, forming a "quantization-optimization-feedback" closed loop, accelerating the model convergence speed, and ensuring that the final model approaches the high-precision training level in terms of accuracy, perplexity and other performance indicators. The overall scheme realizes the balance between precision, efficiency and resource consumption under low-precision training by adaptively managing the FP8 gradient while reducing the system resource occupation rate. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart of the FP8 gradient adaptive optimization method for large language model training in one embodiment is shown in the figure.

[0016] Figure 2 A flowchart of the implementation method of the multi-scale gradient noise filtering and signal enhancement module in one embodiment is shown in the figure.

[0017] Figure 3 A structural block diagram of the FP8 gradient adaptive optimization device for large language model training in one embodiment is shown in the figure. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0019] In one embodiment, as shown in Figure 1 A FP8 gradient adaptive optimization method for large language model training is provided, comprising the following steps:

[0020] Step 102, in the learning iteration process of the large language model to the user input data, the processor acquires the original high-precision gradient tensor of the current iteration step and the historical information containing the historical gradient stored in the memory, and the processor quantizes the scaling factor and the gradient value according to the target FP8 format to obtain quantized input data; the quantized input data is stored in the memory.

[0021] Step 104, the processor filters and denoises the quantized input data stored in the memory, updates the state of the optimizer and stores it in the memory, and the processor acquires the system parameter update amount using the learning mechanism corresponding to the optimizer according to the base learning rate and the system parameter.

[0022] Step 106, the processor optimizes the large language model according to the system parameter update amount and the preset evaluation strategy, obtains the trained large language model, and uses the trained large language model to complete the gradient adaptive optimization of the next iteration step input data, and outputs high-precision data, wherein the intermediate data in the iteration process are stored through the memory and processed through the processor.

[0023] In the above FP8 gradient adaptive optimization method for large language model training, a complete link of "full gradient information capture-quantization precision adaptive-noise suppression-optimizer collaboration-closed loop iterative optimization" is constructed, which specifically solves the core pain points in low precision training and realizes efficient model training of FP8 gradient adaptive. First, by synchronously obtaining the current iteration step original high-precision gradient tensor and the historical gradient information stored in the memory, the limitation of traditional single-step gradient dependence is broken through, providing data basis for processing narrow dynamic range. The inclusion of historical gradient can dynamically capture the global characteristics of gradient distribution, avoid the representation deviation caused by single-step gradient fluctuation, and alleviate the problem of insufficient dynamic range under low precision. On this basis, the scaling factor is quantized according to the target FP8 format and the gradient value, and through the precise design of the scaling strategy, the key gradient information is preserved while reducing the representation precision (FP8), which not only reduces the system storage and computing resource occupation (consistent with the low resource requirement), but also avoids the information loss caused by simple low-precision quantization, solving the performance decline problem caused by "low representation precision". In view of the noise that may be introduced in the quantization process, the scheme purifies the input data through filtering and denoising, directly weakens the interference of noise on the optimizer parameter update, and alleviates the inaccurate update problem of traditional optimizer caused by noise amplification under low precision. When updating the optimizer state, combined with the adaptation mechanism of the basic learning rate and the system parameters, the optimizer learning mechanism and the FP8 gradient characteristics are deeply coordinated to avoid the mechanism mismatch of traditional optimizer in low precision scene, reduce the numerical instability phenomena such as gradient vanishing / explosion and loss shock. Finally, through the preset evaluation strategy, the model is continuously optimized, and the training results are fed back to the gradient adaptive optimization of the next iteration step, forming a "quantization-optimization-feedback" closed loop to accelerate the model convergence speed and ensure that the final model approaches the high-precision training level in terms of accuracy, perplexity and other performance indicators. The overall scheme realizes the balance between system storage precision, efficiency and resource consumption under low precision training through the adaptive management of FP8 gradient, significantly improves the model learning efficiency while reducing the system resource occupation rate.

[0024] In one embodiment, during the learning iteration process of the large language model on the user input data, the processor obtains the original high-precision gradient tensor of the data and the historical information containing the historical gradient stored in the memory, calculates the acceleration value of the original high-precision gradient tensor and pushes it into the circular queue of the historical information in the memory, takes the maximum value of the acceleration value in the new circular queue in the memory by the processor, and quantizes the scaling factor after format conversion according to the target FP8 format;

[0025] The processor scales the original high-precision gradient tensor according to the quantized scaling factor:

[0026] ;

[0027] ;

[0028] wherein, is the original high-precision gradient tensor, is a quantization scaling factor, is a scaled high-precision gradient value, is a dynamic range margin, is an acceleration value of the original high-precision gradient tensor, is a target FP8 format; the scaled original high-precision gradient tensor is quantized into the target FP8 format to obtain quantized input data and stored in the memory.

[0029] In one of the embodiments, the processor filters the dequantized gradient value of the working precision of the quantized input data stored in the memory by using a time index exponential smoothing filtering algorithm:

[0030] ;

[0031] wherein, is the filtered gradient value, is the dequantized gradient value of the working precision, is a smoothing factor. The processor updates the state of the optimizer according to the dequantized gradient value of the working precision stored in the memory and the filtered gradient value of the last iteration step, and stores it in the memory, and outputs the filtered quantized gradient corresponding to the current iteration step.

[0032] In one of the embodiments, if the optimizer is Adam, the processor corrects the deviation of the updated first-order momentum and second-order momentum after updating the first-order momentum and second-order momentum of the optimizer according to the filtered quantized gradient stored in the memory and the system parameters:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] wherein, is the updated first-order momentum, is the updated second-order momentum, , are different hyperparameters of the large language model, is the filtered quantized gradient, For the gradient squared term, This represents the quantized gradient value after filtering in the current iteration step. The first-order momentum after correcting for the deviation. This refers to the second-order momentum after correcting for the bias. The processor calculates the system parameter updates based on the first-order momentum after correcting for the bias, the second-order momentum after correcting for the bias, the principal weights of the large language model, and the base learning rate stored in memory.

[0039] ;

[0040] ;

[0041] ;

[0042] in, This represents the system parameter update amount corresponding to the current iteration step. The learning rate for the Adam optimizer. Based on the learning rate, This is the weight decay rate. Main weight, This is a FP8-specific stability term.

[0043] In one embodiment, the processor uses an anomaly detection strategy to check the loss value of the main weights and system parameter update amounts stored in memory. When an anomaly is detected, the processor rolls back to the stable main weights of the previous iteration stored in memory after completing the parameter update of the large language model before the next iteration step.

[0044] In one embodiment, if the optimizer is SGD, the processor updates the momentum accumulation term of the optimizer based on the filtered quantization gradient and system parameters stored in memory and stores it in memory.

[0045] ;

[0046] in, This is the momentum accumulation term updated in the current iteration step. The momentum coefficient, This is the momentum accumulation term after the previous iteration was not updated. This is the quantized gradient value after filtering in the current iteration step. The processor obtains the system parameter updates and stores them in memory based on the updated momentum accumulation term, the main weights of the large language model, and the base learning rate stored in memory.

[0047] ;

[0048] in, This refers to the amount of system parameter updates. Based on the learning rate, This is the momentum accumulation term updated in the current iteration step.

[0049] In one embodiment, an anomaly detection strategy is used to check the loss values ​​of the sovereign weights and system parameter update amounts stored in memory. When an anomaly is detected, after the parameter update of the large language model is completed before the next iteration step, the process rolls back to the stable sovereign weights of the previous iteration step stored in memory, so as to warm-start the FP8 format SGD optimizer with the updated learning rate. The learning rate and optimizer state are both stored in memory.

[0050] In one embodiment, an FP8 gradient adaptive optimization step for training large language models is provided, the details of which are as follows:

[0051] 1. Initialization:

[0052] Model parameters (Typically stored as BF16 or FP32-Master Weights).

[0053] Adam state: First-order momentum (BF16 / FP32), second momentum (BF16 / FP32).

[0054] Hyperparameters: (e.g., 0.9, 0.999), original base learning rate Weight decay .

[0055] FP8 related configurations: initial parameters of the GP-ADR module (e.g., amax history window size), initial parameters of the GNF-SE module (e.g., smoothing factor). ), OCM-F8 module Initial value or adjustment strategy: initial threshold or adaptive strategy parameters for gradient clipping in the NSC-E module.

[0056] Iteration counter .

[0057] 2. Iteration loop begins :

[0058] a. Obtain the current batch of data, calculate the model's forward propagation and loss function.

[0059] b. Perform backpropagation to obtain the current parameters. High-precision gradient (e.g., BF16).

[0060] c. Module 1: GP-ADR Dynamic Range Adaptive Scaling (DRAS) and Gradient Feature Aware Format Selection (GAFS) are applied to obtain the pre-quantized and preprocessed FP8 gradients. and its inverse scaling factor .

[0061] d. Module Two: GNF-SE (First through) To obtain the filtered gradient (after dequantization to working precision), a Temporal Exponential Smoothing Filter (TESF) and a possible Gradient Confidence Weighted Filter (GCW) are applied. (Working precision, such as BF16).

[0062] e. Module Four: NSC-E-ACT Applying the Adaptive Clipping Threshold (ACT), the clipped gradient is obtained. .

[0063] f. Module 3: OCM-F8-FP8-AdamGuard Core Update:

[0064] i. Update first-order momentum: .

[0065] ii. Update second-order momentum: .

[0066] iii. Calculate the bias-corrected momentum: .

[0067] iv. Obtain the current scheduling learning rate .in, This is the learning rate scheduling function.

[0068] v. Calculate parameter update amount: . Dynamically configured based on FP8 characteristics.

[0069] g. Module four parts: NSC-E-PLCR (optional) If It is quantized before being applied to the FP8 parameters, and the quantization residual is recorded and used for subsequent compensation.

[0070] h. Parameter update:

[0071] • Apply weight decay: (If AdamW method is used).

[0072] Last Update: (here) It is a high-precision sovereignty measurement.

[0073] i. Module Four: NSC-E-NDR Inspection Check if the current loss value contains NaN / Inf. If so, execute a recovery strategy (such as rollback or reducing the learning rate).

[0074] j. If it is necessary to reclaim sovereignty If the model parameters are quantized to FP8 for use in the next round of computation (e.g., the model parameters themselves are also stored in FP8), then quantization is performed at this point.

[0075] 3. Iteration End: Determine if the preset convergence condition is met (e.g., validation set performance no longer improves) or if the maximum number of iterations has been reached. .

[0076] In one embodiment, an overall processing flow for an adaptive optimization algorithm that adapts to FP8 gradients is provided, and the specific steps are as follows:

[0077] Example 1: An adaptive optimization algorithm based on AdamW that adapts to FP8 gradients (FP8-AdamWAdapt)

[0078] This embodiment deeply integrates four core modules designed to adapt to the characteristics of FP8 gradients, based on the widely used AdamW optimization algorithm. AdamW was chosen as the foundation due to its excellent performance in training large models and its correct handling of weight decay.

[0079] 1. Specific implementation of the FP8 gradient adaptive preprocessing and dynamic representation module (GP-ADR):

[0080] Input: The original high-precision gradient tensor calculated in the current training iteration step (Typically in BF16 or FP32 format), and a dictionary state_gp_adr storing the module state, which contains, for example, past... The history of the maximum absolute value of gradients (amax) for each iteration step is state_gp_adr[amax_history] and a counter / flag for dynamic format selection.

[0081] Step 1.1: Gradient Statistics and Analysis: Calculate the current gradient tensor The amax value is denoted as current_amax. Additionally, some characteristics of its statistical distribution, such as mean, standard deviation, and sparsity (proportion of zero values), can be calculated to assist in subsequent dynamic format selection.

[0082] Step 1.2: Dynamic scaling factor Determining (scale_for_quantization) (DRAS implementation):

[0083] a. Push current_amax into the fixed-length circular queue state_gp_adr[amax_history].

[0084] b. Select a representative amax value, amax_representative, from state_gp_adr[amax_history]. For example, you could take the maximum value within the history window, or more robustly, take a high percentile (99.5%) of the amax values ​​within the history window, or use the Delayed Scaling recipe in NVIDIA Transformer Engine (e.g., take the maximum value from the amax history).

[0085] c. Select the target FP8 format. The default setting is E4M3. Based on the gradient characteristic analysis in step 1.1, if there are many gradient outliers or the range is too large, the GAFS mechanism can temporarily switch to E5M2. Set FP8_max_representable to the maximum normal value that the selected FP8 format can represent.

[0086] d. Calculate the scaling factor used for quantization: The margin is a coefficient less than 1 (e.g., 0.98) used to provide a certain dynamic range margin and reduce peak clipping. To prevent... If it's too small (when amax_representative is extremely small), a minimum allowed value needs to be set. Values ​​(e.g., the smallest normal number based on FP8).

[0087] Step 1.3: Gradient scaling and quantization to FP8:

[0088] a. Scale the gradient values ​​to be quantized: .

[0089] b. Quantization is performed to the target FP8 format (e.g., E4M3). This process involves converting the floating-point number to its binary representation in FP8 and applying rounding rules (e.g., stochastic rounding, or round-to-nearest-even) to obtain the FP8 gradient. Random rounding helps reduce quantization bias.

[0090] Step 1.4: Descaling factor transfer: (or its reciprocal, defined according to subsequent usage habits) and It is passed to subsequent modules to restore the FP8 gradient to an approximate original scale when needed.

[0091] Output: Preprocessed FP8 gradient tensor Its corresponding (inverse) scaling factor , and the updated state_gp_adr.

[0092] 2. Specific implementation of the multi-scale gradient noise filtering and signal enhancement module (GNF-SE):

[0093] Input: Output of Module 1 and its scaling factor And the module state state_gnf_se (containing the smoothed gradient from the previous time step). Smoothing coefficient of TESF wait).

[0094] Step 2.1: Initial dequantization to working accuracy: ,in, For working accuracy, For the inverse quantization function, this step restores the FP8 gradient to working precision close to the original scale but still containing FP8 quantization noise (such as BF16).

[0095] Step 2.2: Time-Series Exponential Smoothing Filter (TESF):

[0096] ;

[0097] Smoothing factor It can be dynamically adjusted: for example, in the initial stage of training, or when the rate of change of the loss function is large. It can be set to a larger value (e.g., 0.3-0.5) to quickly track gradient changes; in the later stages of training or during the stabilization phase, This can be reduced (e.g., by 0.05-0.1) to enhance noise smoothing. Adjustments can be saved in `state_gnf_sec`. The logic.

[0098] Step 2.3: (Optional advanced strategy) Gradient confidence weighting (GCW) or directional consistency enhancement:

[0099] a. Calculation and Cosine similarity (or the average direction of the past K steps of smoothed gradients) .

[0100] b. If If the gradient falls below a certain threshold (indicating a significant difference between the current gradient and historical trends, potentially indicating severe noise or a sudden change), it can be further reduced. Or to The amplitude is attenuated, or more dependent on .

[0101] c. Final filter gradient This will be output as a module. If this advanced strategy is not available, .

[0102] Output: Filtered gradient (Working precision, such as BF16), and the updated state_gnf_se.

[0103] 3. Specific implementation of the FP8-AdamWAdapt module (OCM-F8), an improved core mechanism module for FP8-friendly AdamW:

[0104] Input: Output of Module 2 (BF16), Current parameter sovereignty weight (BF16 / FP32), optimizer state state_ocm_f8 (containing first-order momentum) (BF16 / FP32), second momentum (BF16 / FP32), number of iterations ), and hyperparameters (original base learning rate) , , FP8 specific stability terms Weight decay rate ).

[0105] Step 3.1: Update first-order momentum: . Maintaining high precision (BF16 / FP32) to accumulate historical information.

[0106] Step 3.2: Update second-order momentum (variance):

[0107] a. Calculate the squared gradient term: Because the squaring operation is sensitive to outliers, it is advisable to consider using... Beforehand, a slight Huber loss function transformation or value pruning is applied to limit the effect of extreme values ​​on second-order momentum.

[0108] b. ,in, It also maintains a high level of accuracy.

[0109] Step 3.3: Bias Correction: , .

[0110] Step 3.4: Calculate the parameter update amount:

[0111] a. Determine the learning rate for the current period. (For example, using a learning rate scheduler such as cosine annealing to obtain the original base learning rate) (Calculated).

[0112] b. FP8 specific stability terms It can be a fixed value, slightly larger than the minimum resolution of the FP8 format used (e.g., arrive between).

[0113] c. Calculate the update direction and magnitude: .

[0114] d. Applying the learning rate: .in, This is the parameter update function for the optimizer Adam.

[0115] Step 3.5: Apply weight decay (AdamW method): (Note that in AdamW, weight decay is added to the final update and is coupled with the learning rate.)

[0116] Output: Final parameter update amount (BF16 / FP32), and the updated state_ocm_f8 .

[0117] 4. Integrated application of the numerical stability and convergence guarantee enhancement module (NSC-E):

[0118] Adaptive Clipping Threshold (ACT): Obtained before step 3.1. Next, norm clipping is performed. Clipping threshold. It can be dynamically set based on the moving average or percentile of the historical gradient norm, for example. , This represents the clipping factor, typically between 2 and 10. EMA represents the exponential moving average of the corresponding element. It represents the historical gradient norm.

[0119] NaN / Inf Detection and Removal (NDR): This is done during each parameter update (i.e., calculation). After that, check immediately. All elements and the loss value of the current batch. If any NaN or Inf value is detected:

[0120] a. Record the event and rollback the model parameters. To reach a stable state (or the nearest checkpoint).

[0121] b. Set the current learning rate Temporarily reduce significantly (e.g., multiply by 0.1 or 0.5).

[0122] c. (Optional) Enhance the filtering strength of module two GNF-SE (e.g., temporarily reduce it in the following steps). ).

[0123] d. (Optional) Skip parameter updates for the current batch of data entirely, or retry the current batch with a very small learning rate.

[0124] (Optional) Optimizer State Quantization (OSQ): For extreme memory compression, it can quantize high-precision... Before storing in state_ocm_f8, the data is quantized to FP8 format using techniques similar to DRAS and GAFS in Module 1. At the start of the next iteration (before steps 3.1 and 3.2), it is dequantized back from FP8 to BF16 / FP32 for computation. This requires very careful handling of accuracy loss and potential bias accumulation during quantization and dequantization.

[0125] Through the coordinated work of the above modules, the FP8-AdamWAdapt algorithm can largely mitigate the adverse effects of the FP8 gradient, achieving more stable and efficient training.

[0126] It is worth noting that, such as Figure 2 As shown, in the multi-scale gradient noise filtering and signal enhancement module, the preprocessed FP8 gradient (dequantized to working precision) is first processed by TESF (Time-Series Exponential Smoothing), and its output is then weighted with a confidence level calculated using a certain confidence evaluation mechanism (GCW), finally outputting the filtered and enhanced gradient. The smoothing factor is indicated in the figure. and confidence level How it affects the final output.

[0127] Example 2: An adaptive optimization algorithm based on SGD that adapts to FP8 gradients (FP8-SGDMAdapt)

[0128] This embodiment improves upon the momentum-based SGD (SGDM) algorithm, which is chosen for its simplicity, good generalization performance, and advantages over Adam-type algorithms in certain scenarios (especially after fine-tuning).

[0129] 1. FP8 Gradient Adaptive Preprocessing and Dynamic Characterization Module (GP-ADR): The implementation is basically the same as described in Example 1. For SGDM, the focus may be more on preserving the high-precision characteristics of E4M3 through GAFS, since SGDM does not have second-order momentum to smooth the square of noise.

[0130] 2. Multi-scale Gradient Noise Filtering and Signal Enhancement Module (GNF-SE): The implementation is basically the same as described in Example 1. For SGDM, since its parameter updates directly depend on the gradient (after momentum accumulation), the GNF-SE module is particularly critical for gradient smoothing and denoising. Stronger filtering settings or the introduction of additional noise robustness mechanisms can be considered.

[0131] 3. Specific implementation of the FP8-SGDMAdapt module (OCM-F8) for the FP8-friendly SGDM core mechanism improvement:

[0132] Input: Output of Module 2 (BF16), Current parameter sovereignty weight (BF16 / FP32), optimizer state state_ocm_f8 (including the momentum accumulation term from the previous time step). (BF16 / FP32), number of iterations ), and hyperparameters (original base learning rate) momentum coefficient Weight decay rate ).

[0133] Step 3.1: Update the momentum accumulation term ( ):

[0134] .

[0135] Note that, unlike the standard SGDM, the learning rate here... Typically, this is applied to the final parameter update step after momentum accumulation. Maintain BF16 / FP32 accuracy.

[0136] Step 3.2: Apply learning rate and weight decay to calculate parameter updates:

[0137] Get the learning rate for the current period (via the learning rate scheduler) calculate);

[0138] If weight decay is used, the gradient is usually modified directly: (The gradient term of L2 regularization). If not used, then .

[0139] Calculate the final parameter update amount: .

[0140] Enhanced integration of dynamic learning rate adjustment strategies: Because SGDM is more sensitive to the learning rate, especially in noisy environments with FP8 gradients, more complex adaptive learning rate adjustment mechanisms can be integrated. For example:

[0141] Adjustment based on the consistency of gradient and momentum direction: if the current filter gradient Compared with the previous momentum term If the direction (cosine similarity) remains high, the learning rate can be increased appropriately; if the direction frequently reverses, the learning rate should be decreased.

[0142] Adjustments based on gradient norm or sparsity: If the gradient norm remains small or the gradient is very sparse, it may indicate that the learning has entered a plateau or that some parameters are no longer important. In this case, the learning rate can be adjusted accordingly.

[0143] Output: Final parameter update amount (BF16 / FP32), and the updated state_ocm_f8 .

[0144] 4. Integrated application of the numerical stability and convergence guarantee enhancement module (NSC-E):

[0145] Adaptive Clipping Threshold (ACT): Same as in Example 1, applied to... Or adjust the gradient of the learning rate before momentum update.

[0146] NaN / Inf detection and treatment (NDR): Same as in Example 1.

[0147] Learning Rate Warm-up & FP8 Adaptation Period: Considering the instability of SGDM in the early stages of training, especially in the FP8 environment, a "warm-up" phase can be designed. During this phase:

[0148] a. Use relatively high precision for gradient calculation and accumulation (e.g., use BF16 gradients entirely for the first K steps, or adopt a very conservative quantization strategy for FP8 gradients, such as using E5M2 fixedly, and with a large scaling factor to sacrifice some precision for range).

[0149] b. Use a very small initial learning rate and gradually warm it up to the target base learning rate.

[0150] c. After this phase, smoothly transition to a more aggressive FP8 quantization strategy and normal learning rate scheduling. This helps the model establish a relatively stable foundation in the early stages of training before gradually adapting to the challenges brought by FP8.

[0151] FP8 Quantization of the Momentum Term: Similar to the discussion of the Adam state in Example 1, if further memory compression is required, the momentum term can be... The data is quantized to FP8 storage and dequantized in the next round of computation. This also requires careful handling of the errors introduced by quantization.

[0152] FP8-SGDMAdapt aims to deliver impressive performance in the FP8 environment by enhancing gradient processing and introducing stability measures for SGDM characteristics, making it particularly suitable for scenarios with more stringent requirements for video memory and computational complexity.

[0153] It should be understood that, although Figures 1-2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0154] In one embodiment, such as Figure 3 As shown, an FP8 gradient adaptive optimization device for training large language models is provided, including: a quantization module 302, a gradient optimization module 304, and a gradient adaptive module 306, wherein:

[0155] The quantization module 302 is used in the learning iteration process of the large language model on the user input data. The processor obtains the original high-precision gradient tensor of the current iteration step and the historical information containing historical gradients stored in the memory. The processor quantizes the scaling factor and gradient value according to the target FP8 format to obtain the quantized input data. The quantized input data is stored in the memory.

[0156] The gradient optimization module 304 is used by the processor to filter and denoise the quantized input data stored in the memory, update the state of the optimizer and store it in the memory, and the processor obtains the system parameter update amount according to the base learning rate and system parameters using the learning mechanism corresponding to the optimizer.

[0157] The gradient adaptive module 306 is used by the processor to optimize the large language model according to the system parameter update amount and the preset evaluation strategy to obtain a trained large language model. The trained large language model is then used to complete the gradient adaptive optimization of the input data in the next iteration step and output high-precision data. The intermediate data in the iteration process are stored in the memory and processed by the processor.

[0158] Specific limitations regarding the FP8 gradient adaptive optimization device for training large language models can be found in the limitations of the FP8 gradient adaptive optimization method for training large language models mentioned above, and will not be repeated here. Each module in the aforementioned FP8 gradient adaptive optimization device for training large language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0159] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. An FP8 gradient self-adaptive optimization method for large language model training, characterized in that, The method comprises: In the learning iteration process of the large language model on the user input data, the processor acquires the original high-precision gradient tensor of the current iteration step and the historical information containing historical gradients stored in the memory, and scales the gradient value according to the target FP8 format quantization scaling factor by the processor to obtain quantized input data; the quantized input data is stored in the memory; After the processor filters and denoises the quantized input data stored in the memory, the state of the optimizer is updated and stored in the memory, and the processor acquires the system parameter update amount by using the learning mechanism corresponding to the optimizer according to the base learning rate and the system parameter. The processor optimizes the large language model according to the system parameter update amount and the preset evaluation strategy to obtain the trained large language model, and uses the trained large language model to complete the gradient adaptive optimization of the input data of the next iteration step, and outputs high-precision data, wherein the intermediate data in the iteration process are stored in the memory and processed by the processor.

2. The method of claim 1, wherein, In the learning iteration process of the large language model on the user input data, the processor acquires the original high-precision gradient tensor of the current iteration step and the historical information containing historical gradients stored in the memory, and scales the gradient value according to the target FP8 format quantization scaling factor by the processor to obtain quantized input data, comprising: In the learning iteration process of the large language model on the user input data, the processor acquires the original high-precision gradient tensor of the data and the historical information containing historical gradients stored in the memory, pushes the acceleration value calculated for the original high-precision gradient tensor into the circular queue of the historical information in the memory by the processor, takes the maximum value of the acceleration value in the new circular queue in the memory by the processor, and quantizes the scaling factor after format conversion according to the target FP8 format; The processor scales the original high-precision gradient tensor according to the quantized scaling factor: ； ； wherein, is the original high-precision gradient tensor, is a quantization scaling factor, is the scaled high-precision gradient value, is a dynamic range margin, is an acceleration value of the original high-precision gradient tensor, is a target FP8 format; The original high-precision gradient tensor after scaling is quantized to the target FP8 format to obtain quantized input data and store it in the memory.

3. The method of claim 2, wherein, After the processor filters and denoises the quantized input data stored in the memory, the state of the optimizer is updated and stored in the memory, comprising: The processor uses a time series exponential smoothing filtering algorithm to filter the gradient value of the dequantization to working precision of the quantized input data stored in the memory: ； wherein, is the filtered gradient value, is the gradient value dequantized to the working precision, is the smoothing factor, is the quantized input data; The processor updates the state of the optimizer according to the dequantized gradient value to working precision stored in the memory and the gradient value after filtering of the last iteration step, and stores it in the memory, and outputs the filtered quantized gradient corresponding to the current iteration step.

4. The method of claim 3, wherein, The processor acquires the system parameter update amount by using the learning mechanism corresponding to the optimizer according to the base learning rate and the system parameter, comprising: If the optimizer is Adam, the processor updates the first-order momentum and the second-order momentum of the optimizer according to the filtered quantized gradient and the system parameters stored in the memory, and stores them in the memory, and then corrects the deviation of the updated first-order momentum and the second-order momentum: ； ； ； ； ； wherein, is the updated first order momentum, is the updated second order momentum, , are different hyperparameters of the large language model, respectively, is the filtered quantized gradient, is the gradient square is the filtered quantized gradient value at the current iteration step, is the first order momentum after correcting the bias, is the second order momentum after correcting the bias; The processor calculates the system parameter update amount according to the first-order momentum after the correction of the deviation, the second-order momentum after the correction of the deviation, the main weight of the large language model, and the basic learning rate stored in the memory: ； ； ； wherein, is the system parameter update for the current iteration step, is the applied learning rate for the Adam optimizer, is the base learning rate, is the weight decay rate, is the master weight, is the FP8 specific stabilizer, is the parameter update function for the Adam optimizer.

5. The method of claim 4, wherein, After the processor calculates the system parameter update amount according to the first-order momentum after the correction of the deviation, the second-order momentum after the correction of the deviation, the main weight of the large language model, and the basic learning rate stored in the memory, the method further comprises: The processor checks the loss value of the main weight and the system parameter update amount stored in the memory using an anomaly detection strategy, and when an anomaly is detected, it rolls back to the stable main weight of the last iteration step stored in the memory after completing the parameter update of the large language model before the next iteration step.

6. The method of claim 3, wherein, The processor obtains the system parameter update amount according to the basic learning rate and the system parameters using the learning mechanism corresponding to the optimizer, comprising: If the optimizer is SGD, the processor updates the momentum accumulation item of the optimizer according to the filtered quantized gradient and the system parameters stored in the memory, and stores them in the memory: ； wherein, is the updated momentum accumulation for the current iteration step, is the momentum coefficient, is the non-updated momentum accumulation for the previous iteration step, is the filtered quantized gradient value for the current iteration step; The processor obtains the system parameter update amount according to the updated momentum accumulation item stored in the memory, the main weight of the large language model, and the basic learning rate, and stores it in the memory: ； wherein, is a system parameter update, is a base learning rate, is a momentum accumulation item updated at the current iteration step.

7. The method of claim 6, wherein, After the processor obtains the system parameter update amount according to the updated momentum accumulation item stored in the memory, the main weight of the large language model, and the basic learning rate, and stores it in the memory, the method further comprises: The processor checks the loss value of the main weight and the system parameter update amount stored in the memory using an anomaly detection strategy, and when an anomaly is detected, it rolls back to the stable main weight of the last iteration step stored in the memory after completing the parameter update of the large language model before the next iteration step.

8. An FP8 gradient self-adaptive optimization device for large language model training, characterized in that, The device comprises: A quantization module for, in the learning iteration process of a large language model on user input data, the processor obtains the original high-precision gradient tensor of the current iteration step and the historical information containing historical gradients stored in the memory, and the processor quantizes the scaling factor and the gradient value according to the target FP8 format to obtain quantized input data; the quantized input data is stored in the memory; A gradient optimization module for, after the processor filters and denoises the quantized input data stored in the memory, updating the state of the optimizer and storing it in the memory, and the processor obtains the system parameter update amount using the learning mechanism corresponding to the optimizer according to the basic learning rate and the system parameters; A gradient adaptation module is configured to update the large language model according to the system parameter update quantity and a preset evaluation strategy, obtain a trained large language model, and use the trained large language model to complete gradient adaptation optimization of input data in a next iteration step, and output high-precision data.

Images