Memory optimization method and system for large model training based on low-precision integer storage

Abstract

This invention belongs to the field of large model training technology and discloses a method and system for optimizing GPU memory for large model training based on low-precision integer storage. The method includes: acquiring state parameters during model training and storing them as integer values; using dynamic scaling factor dequantization to restore the integer values ​​to floating-point representation for calculation, resulting in an integer-stored training framework; under the integer-stored training framework, performing differentiated quantization based on different numerical distribution characteristics of the state parameters to obtain quantized state parameters; based on the quantized state parameters, managing the flow of quantized state parameters through a global gradient stack structure according to the backpropagation order, and popping state parameters sequentially from the global gradient stack structure to update model parameters, thus completing the optimization. This invention uses integer format storage, compressing GPU memory usage. Real-time dequantization of the stored integer data restores it to floating-point representation before participating in calculation and backpropagation, balancing storage compression ratio and training stability.

Description

Technical Field

[0001] This invention belongs to the field of large model training technology, and specifically relates to a method and system for optimizing GPU memory for large model training based on low-precision integer storage. Background Technology

[0002] Language models have demonstrated powerful capabilities in core areas of artificial intelligence such as natural language processing and code generation, but their application in downstream tasks relies on full parameter fine-tuning, which consumes extremely high computing resources, thus hindering the widespread adoption of the technology.

[0003] The full parameter fine-tuning process places stringent demands on the GPU memory capacity, resulting in high training costs and posing a significant challenge to effectively fine-tuning large-scale models on low- to mid-range hardware devices.

[0004] To alleviate memory pressure, existing technologies have proposed various solutions. Efficient parameter fine-tuning methods significantly reduce the number of parameters that need to be stored and optimized by updating only a small number of newly added parameters in the model or by performing low-rank parameterization. However, this method cannot optimize all parameters of the model, potentially limiting the full adaptation of model performance. Gradient checkpointing techniques trade computation time for memory space by recalculating some forward activation values ​​during backpropagation, thus avoiding storing all intermediate activations, but this significantly increases training time. Mixed-precision training utilizes 16-bit floating-point numbers for forward and backward computation to accelerate and save memory, while retaining a 32-bit floating-point master copy for parameter updates to maintain numerical stability. However, this method still requires storing model parameters, gradients, and optimizer states in 32-bit floating-point format, resulting in considerable memory consumption.

[0005] The aforementioned existing solutions all have significant limitations: efficient parameter fine-tuning sacrifices the potential for full parameter optimization; gradient checkpointing and mixed-precision training fail to fundamentally address the floating-point storage overhead of core states (weights, gradients, optimizer states). Furthermore, while low-precision integer quantization techniques have been successfully applied to compression and acceleration during model inference, directly transferring them to the training phase presents serious challenges, including the accumulation of numerical errors during iteration, convergence difficulties caused by gradient instability, and the dependence of existing deep learning training frameworks' automatic differentiation mechanisms on floating-point tensors. This results in high GPU memory usage during integer storage and computation, instability in integer training, and significant loss of quantization precision. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for optimizing GPU memory during large model training based on low-precision integer storage, so as to solve the problems of high GPU memory usage, unstable integer training, and large loss of quantization accuracy during the fine-tuning of full parameters of existing large models.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for optimizing GPU memory for training large models based on low-precision integer storage, including:

[0009] Obtain the state parameters during model training and store them as integer values ​​in integer format;

[0010] By using dynamic scaling factor dequantization to restore integer values ​​to floating-point values ​​for calculation, a training framework with integer storage is obtained.

[0011] Within the training framework of integer storage, differentiated quantization is performed based on the different numerical distribution characteristics of the state parameters to obtain the quantized state parameters;

[0012] Based on the quantized state parameters, the flow of quantized state parameters is managed through a global gradient stack structure according to the backpropagation order. State parameters are popped out of the global gradient stack structure in sequence to update model parameters, thus completing the optimization.

[0013] Furthermore, the acquisition of state parameters during model training, stored as integer values ​​in integer format, includes:

[0014] The state parameters during model training include weight parameters, gradient information, and the first and second momentum variables of the optimizer, and all model states are stored in integer format.

[0015] Furthermore, the training framework that uses dynamic scaling factor dequantization to restore integer values ​​to floating-point values ​​for calculation, resulting in integer storage, includes:

[0016] Before each calculation, the integer value is dequantized by a dynamic scaling factor to restore it to a floating-point representation for forward and reverse operations. After the calculation is completed, it is requantized and written back to the video memory in integer form.

[0017] Furthermore, within the training framework of integer storage, differentiated quantization is performed to obtain quantized state parameters based on different numerical distribution characteristics of the state parameters, including:

[0018] For gradients and optimizer momentum with a concentrated distribution, a uniform quantizer is used for quantization. For model weights with long-tailed distributions and outliers, a dense-sparse quantizer is used for quantization. The dense-sparse quantizer first decomposes the weights into dense and sparse parts, performs uniform quantization on the dense part, and stores the sparse part in a compressed sparse row format.

[0019] Furthermore, quantization is performed using a uniform quantizer:

[0020] Quantization and dequantization are mathematically defined as follows:

[0021]

[0022] in, It is a floating-point vector. For quantized integer vectors, This represents the floor function, where b∈N is the quantization bit width; and These are the quantization scale and the zero point, respectively. The quantization scale and the zero point are determined by... The lower and upper limits are determined as follows:

[0023]

[0024] Through linear mapping, the distribution of gradient and momentum is uniformly compressed into a finite integer range; during computation, it is dequantized into floating-point numbers in real time according to the scaling factor for forward and backward propagation;

[0025] The quantization is performed using a dense-sparse quantizer:

[0026] First, we analyze the distribution density of the weights, decomposing the weight matrix W into two complementary sub-matrices: one dense part D for the backbone distribution, and the other sparse part S for outliers; the formal definition is as follows:

[0027]

[0028] in, and The distribution threshold;

[0029] Subsequently, the dense matrix D is quantized and dequantized using a standard uniform quantizer, while the sparse matrix S retains the floating-point format and is stored using compressed sparse rows or an equivalent sparse storage structure.

[0030] Furthermore, based on the quantized state parameters, the flow of the quantized state parameters is managed through a global gradient stack structure according to the backpropagation order. State parameters are popped sequentially from the global gradient stack structure to update model parameters, thus completing the optimization. This includes:

[0031] During backpropagation, the flow of quantized gradients is managed through a global gradient stack structure. In the backpropagation process, the weights in integer format are dequantized into floating-point numbers to calculate the gradients. After the gradients are quantized into integers, they are pushed onto the stack. When the parameters are updated, the gradients are popped out of the global gradient stack structure in sequence to update the model parameters.

[0032] Furthermore, during the reverse computation, the integer weights stored in the video memory are first converted to floating-point format using a scaling factor, and then the conventional reverse computation is performed. After obtaining the gradient, the gradient is requantized into an integer value and stored in the video memory. A global gradient stack structure is introduced to cache the gradients of each layer. Whenever a layer completes its reverse propagation, its quantized gradient is pushed onto the stack.

[0033] The updated model parameters are:

[0034] During the parameter update phase, gradients are popped from the top of the stack in reverse order. When each layer is updated, the optimizer retrieves the quantized gradient of that layer from the stack and uses it together with the quantized first or second momentum stored in the stack for calculation. Throughout the process, the integer storage structure is restored to floating-point form during the update calculation and then quantized and written back to the video memory.

[0035] Secondly, the present invention provides a large model training memory optimization system based on low-precision integer storage, comprising:

[0036] The data acquisition module is used to acquire the state parameters during model training and store them as integer values ​​in integer format.

[0037] The framework building module is used to convert integer values ​​back to floating-point values ​​through dynamic scaling factor dequantization, resulting in a training framework with integer storage.

[0038] The quantization module is used to perform differentiated quantization on the different numerical distribution characteristics of the state parameters under the training framework of integer storage to obtain the quantized state parameters.

[0039] The update and optimization module is used to manage the flow of quantized state parameters through a global gradient stack structure, based on the quantized state parameters and following the backpropagation order. State parameters are popped out of the global gradient stack structure in sequence to update model parameters, thus completing the optimization.

[0040] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the large model training memory optimization method based on low-precision integer storage.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the large model training memory optimization method based on low-precision integer storage.

[0042] Compared with the prior art, the present invention has the following technical effects:

[0043] This invention proposes a unified training framework for low-precision integer storage. At the storage level, all training state data, including weights, gradients, and momentum, are stored in integer format, significantly reducing GPU memory usage. During computation, the stored integer data is dequantized in real time, restoring it to floating-point representation before participating in forward and backward computation, effectively balancing storage compression and training stability.

[0044] This invention presents a distribution-aware hybrid feature quantizer that adaptively distinguishes between key sparse and dense features by analyzing the distribution characteristics of parameters. It employs a hybrid quantization strategy: high precision is used to protect key features, while linear quantization is used for dense features. This approach effectively maintains the numerical stability of model updates and training performance without increasing additional GPU memory overhead.

[0045] This invention is based on an integer gradient flow scheme with a stack structure. It uses the stack structure to manage the gradients of each layer in sequence, realizes the orderly accumulation and backpropagation of gradients, and ensures the correct transmission and stable update of gradient flow in backpropagation.

[0046] This invention adopts a modular encapsulation architecture, decoupling the integer storage, quantizer, optimizer adaptation and gradient flow management modules to form a plug-and-play component that can be directly integrated with mainstream deep learning frameworks. Attached Figure Description

[0047] Figure 1 This is a comparison diagram between the present invention and the traditional QAT.

[0048] Figure 2 This is the distribution of training state parameters for this invention.

[0049] Figure 3 This invention relates to the integer gradient flow and backpropagation mechanism.

[0050] Figure 4 This is a comparison of the training loss curves for this invention.

[0051] Figure 5 This is a radar chart showing the capabilities of the model of this invention.

[0052] Figure 6 This is a flowchart of the present invention.

[0053] Figure 7 This is a system structure diagram of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings:

[0055] Explanation of relevant parameters:

[0056] GPU: Graphics Processing Unit

[0057] QAT: Quantization-aware Training

[0058] INT: Integer data type;

[0059] INT8: 8-bit integer data;

[0060] CSR: Compressed Sparse Row;

[0061] LLaMA2-7B: An open-source large language model;

[0062] ShareGPT: A platform for sharing ChatGPT conversations;

[0063] Lion Optimizer: A deep learning optimization algorithm developed through symbolic program search.

[0064] Example 1, please refer to Figure 6 This invention provides a method for optimizing GPU memory for training large models based on low-precision integer storage, including:

[0065] Obtain the state parameters during model training and store them as integer values ​​in integer format;

[0066] By using dynamic scaling factor dequantization to restore integer values ​​to floating-point values ​​for calculation, a training framework with integer storage is obtained.

[0067] Within the training framework of integer storage, differentiated quantization is performed based on the different numerical distribution characteristics of the state parameters to obtain the quantized state parameters;

[0068] Based on the quantized state parameters, the flow of quantized state parameters is managed through a global gradient stack structure according to the backpropagation order. State parameters are popped out of the global gradient stack structure in sequence to update model parameters, thus completing the optimization.

[0069] This invention can efficiently store and update model weights, gradients, and optimizer states in a unified integer format while maintaining training stability and performance. This significantly reduces training memory usage and supports full-parameter training of large models on low- to mid-range GPUs with limited memory, improving training accessibility and cost-effectiveness.

[0070] Example 2: This invention provides a method for optimizing GPU memory for large model training based on low-precision integer storage, including:

[0071] This invention systematically quantizes and integerizes the core states (including model weights, gradients, optimizer states, etc.) throughout the training process, and dynamically dequantizes them through a scaling factor during the computation phase, thereby achieving collaborative optimization of storage compression and computation without compromising model performance.

[0072] The entire technical solution mainly includes the following three core parts:

[0073] (1) The training framework for integer storage is constructed by using INT8 storage and dynamic dequantization mechanism to achieve compression of training state storage and preservation of computational accuracy;

[0074] (2) The distribution-aware hybrid feature quantization method identifies key parameter ranges by modeling feature distribution and adopts a hybrid precision quantization strategy to balance performance and compression rate;

[0075] (3) Integer gradient flow and backpropagation mechanism: Through a stack-based integer gradient flow scheme, the orderly storage and backpropagation of gradients are realized.

[0076] Specific

[0077] (1) Construction of the training framework for integer storage

[0078] Existing quantization methods mainly focus on accelerating the model inference stage, aiming to reduce inference computation and memory usage. However, during the training stage, a large number of floating-point parameters, gradients, and optimizer states still need to be saved, thus contributing little to training memory optimization.

[0079] Traditional Quantization-Aware Training (QAT) methods insert "pseudo-quantization nodes" into floating-point parameters, allowing the training process to still take place in floating-point space. Therefore, its memory overhead is almost the same as the original floating-point training scheme, and it does not essentially reduce the training storage burden.

[0080] To address the aforementioned issues, the proposed integer storage training framework directly stores all model states in INT8 format during training, including weight parameters, gradient information, and the first and second momentum variables of the optimizer. Unlike QAT, this scheme does not rely on pseudo-quantization nodes. Instead, before each calculation, it uses dynamic scaling factor dequantization to restore integer values ​​to floating-point representations for forward and backward operations. After the calculation is complete, it requantizes and writes the integer values ​​back to memory.

[0081] This collaborative mechanism of "storing as integers and calculating in floating points" strikes a balance between memory utilization and numerical precision.

[0082] During the storage phase, all training states are compressed and stored in INT8 format, reducing GPU memory usage by approximately 75%.

[0083] During the computation phase, controllable inverse quantization is achieved through a scaling factor to ensure the continuity of floating-point calculations and the stability of training.

[0084] The system updates the scaling factor at the end of each training step to achieve dynamic range adaptation and avoid saturation and overflow.

[0085] like Figure 1 As shown, through this mechanism, the present invention achieves systematic compression of training memory, enabling full parameter fine-tuning of large models to be completed on a single card or low-cost GPU device, greatly improving the accessibility and economy of training.

[0086] (2) Hybrid feature quantization method based on distribution perception

[0087] To maintain the stability and accuracy of model training under low-precision integer storage conditions, this invention further systematically analyzes the numerical distribution characteristics of model weights, gradients, and optimizer momentum, as follows: Figure 2 As shown.

[0088] Different training state variables exhibit significant differences in their distribution patterns: gradients and momentum typically show a concentrated symmetrical distribution, while weights exhibit obvious long-tail characteristics and outlier phenomena. Therefore, adopting a uniform quantization strategy will inevitably lead to excessive truncation or information loss of some parameters. To address this, this invention proposes a distribution-aware hybrid feature quantization method, designing differentiated quantizers for different state variables to achieve an optimal balance between memory compression and numerical fidelity.

[0089] Uniform quantizer for gradient and momentum

[0090] Gradient and momentum are variables that change frequently and have a concentrated distribution during training. Their probability density typically exhibits a bell-shaped distribution with approximately zero mean, and almost no significant outliers. Therefore, this invention employs a lightweight uniform quantizer for these two types of variables, which can significantly compress GPU memory while ensuring efficient quantization and dequantization processes without imposing additional burden on training throughput. The uniform quantizer includes two basic operations: quantization and dequantization, whose mathematical definitions are as follows:

[0091]

[0092] in, It is a floating-point vector. For quantized integer vectors, This represents the floor function, where b∈N is the quantization bit width. and These are the quantization scale and zero point, respectively. For the sake of fast calculation, they are directly derived from... The lower and upper limits are determined as follows:

[0093]

[0094] Through this linear mapping, the distribution of gradient and momentum is uniformly compressed into a finite integer range. During computation, the values ​​are dequantized into floating-point numbers in real time according to the scaling factor for forward and backward propagation, which not only ensures numerical reversibility but also avoids the computational overhead of complex nonlinear mapping.

[0095] Dense-Sparse Quantizer for Weights

[0096] Compared to gradients and momentum, model weights exhibit a larger dynamic range and stronger non-uniformity. Statistical results show that the range of weight values ​​is typically three orders of magnitude larger than that of momentum, and they exhibit a significant long-tailed distribution and sparse outliers. This characteristic makes it easy for direct uniform quantization to cause outlier overflow and amplify approximation errors, thus severely affecting training convergence.

[0097] To address this, this invention designs a dense-sparse decomposition quantizer. This quantizer first analyzes the distribution density of the weights and then decomposes the weight matrix W into two complementary sub-matrices: a dense portion D for the backbone distribution and a sparse portion S for outliers. Its formal definition is as follows:

[0098]

[0099] in, and The distribution threshold can be adaptively determined as a percentage of the overall range (e.g., 20%). This decomposition effectively separates extreme outliers, ensuring that quantization operations only apply to the core regions of the distribution set. Subsequently, the dense matrix D undergoes INT8 quantization and dequantization using a standard uniform quantizer, while the sparse matrix S retains its floating-point format and is stored using a compressed sparse row format (CSR) or an equivalent sparse storage structure, thereby minimizing its GPU memory usage. Since the proportion of outliers is typically less than 1%, this hybrid quantization strategy significantly reduces overall storage overhead with almost no loss of training accuracy.

[0100] (3) Integer gradient flow and backpropagation mechanism

[0101] like Figure 3 As shown, in a training framework with low-precision integer storage, traditional floating-point-based automatic differentiation mechanisms cannot be directly applied. Mainstream deep learning frameworks all assume that model parameters are in floating-point format (32-bit floating-point numbers / 16-bit floating-point numbers), and only floating-point tensors possess gradient properties. However, when model weights, gradients, and optimizer states are all stored in integer form (such as INT8), the system cannot automatically trace the computation graph, generate backpropagation paths, or save gradient results. This limitation becomes a key obstacle to achieving fully integer-based training.

[0102] In summary, this invention proposes a stack-based integer gradient flow and backpropagation mechanism to achieve accurate gradient calculation, caching, and updating within a collaborative framework of integer storage and floating-point computation.

[0103] Backpropagation of quantization parameters

[0104] During backpropagation, the final task loss is passed forward sequentially from the output layer, and the gradient of each layer's parameters needs to be calculated. Since integer weights themselves do not possess gradient properties, the system cannot directly calculate the gradient using automatic differentiation.

[0105] To address this, this invention designs a gradient flow scheme for integer weights. Whenever backpropagation is performed, the system first converts the integer weights stored in GPU memory to floating-point format using a scaling factor for regular backpropagation. After obtaining the gradient, the gradient is requantized into integer values ​​and stored in GPU memory, achieving a closed loop of "dequantization computation, quantization storage".

[0106] Meanwhile, this invention introduces a global gradient stack structure to cache gradients from each layer. Whenever a layer completes backpropagation, its quantized gradient is pushed onto the stack. The stack's LIFO (Last-In, First-Out) characteristic naturally matches the backpropagation direction of the network, ensuring that gradient access always maintains a constant time complexity of O(1), eliminating the need for additional indexing operations and thus significantly saving GPU memory while maintaining efficiency.

[0107] Update of quantization parameters

[0108] During backpropagation, gradients from each layer are pushed onto the stack in the order from the output layer to the input layer; while in the parameter update phase, gradients are popped from the top of the stack layer by layer in the reverse order, realizing an update process that corresponds one-to-one with the network layers.

[0109] During each layer update, the optimizer retrieves the quantized gradient of that layer from the stack and uses it in conjunction with the quantized first or second momentum stored in the stack for computation. The entire process maintains an integer storage structure, only briefly reverting to floating-point form during update computation, and then quantizing and writing it back to GPU memory. Since stack access has a constant time complexity of O(1), the entire update phase has extremely high execution efficiency and is unaffected by model depth.

[0110] The core principle of this invention lies in constructing a collaborative training framework that "stores data as integers and performs calculations using floating-point numbers." It drastically compresses GPU memory usage by systematically storing the core states (including model weights, gradients, and optimizer states) throughout the entire training process of a large model using low-precision integer quantization. When actual calculations are required, these integer values ​​are temporarily restored to floating-point numbers using a dynamic scaling factor to ensure computational accuracy and training stability. This achieves a balance between storage efficiency and numerical precision, thereby reducing the reliance on GPU memory resources for full-parameter training without significantly compromising model performance.

[0111] The implementation of this scheme relies primarily on the orderly collaboration of three key methods. First, it constructs an integer-stored training framework, where weights, gradients, and optimizer momentum are consistently stored in GPU memory as 8-bit integers (INT8) during training, with dequantization only performed before computation via dynamically updated scaling factors. Second, it introduces a distribution-aware hybrid feature quantization method, applying different quantization strategies to address the varying data distribution characteristics of different states: a computationally efficient uniform quantizer is used for gradients and momentum with concentrated distributions; a dense-sparse quantizer is used for weights with outliers, decomposing them into a dense main body that can be uniformly quantized and a sparse outlier part that requires special storage. Finally, it designs an integer gradient flow and backpropagation mechanism, introducing a global gradient stack structure to manually manage gradient computation, caching, and update processes that cannot be automatically handled by the framework under integer storage conditions. This ensures that gradients flow in an orderly manner in quantized form and work collaboratively with the integer-quantized optimizer state during parameter updates.

[0112] The significant reduction in GPU memory usage achieved by this invention, using INT8 storage, is expected to reduce memory usage by approximately 75% compared to traditional floating-point storage (such as 32-bit floating-point numbers). This makes it possible to perform full-parameter fine-tuning of large-scale models on a single or a small number of low-to-mid-range GPUs, greatly improving the accessibility and cost-effectiveness of training. Secondly, through a distribution-aware hybrid quantization strategy, this scheme effectively overcomes the numerical error accumulation and model convergence problems that may arise from simple uniform quantization, ensuring training accuracy and stability while achieving high compression ratios. Finally, the proposed integer gradient flow mechanism successfully solves the dependency of existing deep learning frameworks on floating-point tensors, providing a feasible engineering implementation path for complete and systematic integer storage during the training phase, ensuring efficient execution of the entire training process.

[0113] Example 3: A full-parameter fine-tuning experiment was conducted on a large language model with 7 billion parameters:

[0114] Pre-trained model: LLaMA2-7B

[0115] Training dataset: ShareGPT (including 94.1K samples)

[0116] Optimizer: Lion Optimizer

[0117] GPU server: NVIDIA RTX A6000 GPU.

[0118] The differences between our method and the traditional full-precision training scheme in terms of memory usage, training stability, and model performance were compared and analyzed.

[0119] 1. Algorithm Implementation

[0120] Integer storage mechanism application

[0121] At the beginning of training, all model parameters are initialized to INT8 storage format using a uniform scaling factor. Within each training step, the parameters are dynamically dequantized to floating-point for forward and backward calculations, updated, and then requantized and written back, thus ensuring both numerical accuracy and memory compression.

[0122] Hybrid quantizer design

[0123] Lightweight uniform quantizers are used for gradients and momentum to achieve low-latency compression;

[0124] A dense-sparse hybrid quantizer is used for the weights, retaining only 1% of the outliers as floating-point values, while the rest are quantized to integers, which significantly improves storage efficiency.

[0125] Integer backpropagation mechanism

[0126] The Lion optimizer uses a global gradient stack to record the quantized gradient information during backpropagation. Each time the Lion optimizer is updated, it retrieves the gradient of the corresponding layer from the stack in O(1) complexity, thus achieving efficient parameter updates and weight write-back.

[0127] 2. Training memory usage assessment

[0128] This embodiment systematically reports the overall picture of video memory usage during the training process, covering the memory distribution characteristics at different stages, and also provides a comprehensive statistical and comparative analysis of total allocated video memory and peak allocated video memory.

[0129] Table 1 Comparison of memory usage between traditional methods and the present invention

[0130]

[0131] 3. Training performance evaluation

[0132] Figure 5 The training loss curves for different full parameter fine-tuning are shown. Figure 4As can be seen, the method proposed in this invention can maintain considerable training performance while significantly reducing GPU memory usage.

[0133] like Figure 5 As shown, this invention also provides a radar chart covering eight capability dimensions to visually demonstrate the model's overall performance across various metrics. Experimental results show that, compared to the pre-trained LLaMA-2 baseline model, the proposed quantization full-parameter fine-tuning method achieves comprehensive and consistent performance improvements across all evaluation dimensions. Compared to traditional full-parameter fine-tuning methods, the proposed method maintains a comparable level of overall performance and outperforms in some key dimensions. For example, it demonstrates a significant advantage in mathematical computation metrics, further validating the effectiveness and superiority of this invention under low-precision storage conditions.

[0134] Please see Figure 7 In another embodiment of the present invention, a large model training memory optimization system based on low-precision integer storage is provided, which can be used to implement the above-mentioned large model training memory optimization method based on low-precision integer storage. Specifically, the system includes:

[0135] The data acquisition module is used to acquire the state parameters during model training and store them as integer values ​​in integer format.

[0136] The framework building module is used to convert integer values ​​back to floating-point values ​​through dynamic scaling factor dequantization, resulting in a training framework with integer storage.

[0137] The quantization module is used to perform differentiated quantization on the different numerical distribution characteristics of the state parameters under the training framework of integer storage to obtain the quantized state parameters.

[0138] The update and optimization module is used to manage the flow of quantized state parameters through a global gradient stack structure, based on the quantized state parameters and following the backpropagation order. State parameters are popped out of the global gradient stack structure in sequence to update model parameters, thus completing the optimization.

[0139] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0140] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for operating a large model training memory optimization method based on low-precision integer storage.

[0141] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the large model training memory optimization method based on low-precision integer storage in the above embodiments.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for optimizing GPU memory for training large models based on low-precision integer storage, characterized in that, include: Obtain the state parameters during model training and store them as integer values ​​in GPU memory; By dequantizing the integer value using a dynamic scaling factor, the integer value is restored to floating point for calculation. After the calculation is completed, it is requantized and written back to the display memory in integer form, thus obtaining a training framework with integer storage. Within the training framework of integer storage, differentiated quantization is performed based on the different numerical distribution characteristics of the state parameters to obtain the quantized state parameters; Based on the quantized state parameters, the flow of quantized state parameters is managed through a global gradient stack structure according to the backpropagation order. State parameters are popped out of the global gradient stack structure in sequence to update model parameters, thus completing the optimization. Under the training framework with integer storage, differentiated quantization is performed based on different numerical distribution characteristics of the state parameters to obtain quantized state parameters, including: For gradients and optimizer momentum, a uniform quantizer is used for quantization. For model weights, a dense-sparse quantizer is used for quantization. The dense-sparse quantizer first decomposes the weights into dense and sparse parts, performs uniform quantization on the dense part, and stores the sparse part in a compressed sparse row format. The quantization is performed using a uniform quantizer: Quantization and dequantization are mathematically defined as follows: in, It is a floating-point vector. For quantized integer vectors, This represents the floor function, where b∈N is the quantization bit width; and These are the quantization scale and the zero point, respectively. The quantization scale and the zero point are determined by... The lower and upper limits are determined as follows: Through linear mapping, the distribution of gradient and momentum is uniformly compressed into a finite integer range; during computation, it is dequantized into floating-point numbers in real time according to the scaling factor for forward and backward propagation; The quantization is performed using a dense-sparse quantizer: First, we analyze the distribution density of the weights, decomposing the weight matrix W into two complementary sub-matrices: one dense part D for the backbone distribution, and the other sparse part S for outliers; the formal definition is as follows: in, and The distribution threshold; The dense matrix D is quantized using a standard uniform quantizer, while the sparse matrix S retains the floating-point format and is saved using compressed sparse rows.

2. The method for optimizing large model training memory based on low-precision integer storage according to claim 1, characterized in that, The acquisition of state parameters during model training, stored as integer values ​​in integer format, includes: The state parameters during model training include weight parameters, gradient information, and the first and second momentum variables of the optimizer, and all model states are stored in integer format.

3. The method for optimizing large model training memory based on low-precision integer storage according to claim 1, characterized in that, The training framework, which uses dynamic scaling factor dequantization to restore integer values ​​to floating-point values ​​for calculation, and obtains integer storage, includes: Before each calculation, the integer value is dequantized by a dynamic scaling factor to restore it to a floating-point representation for forward and reverse operations. After the calculation is completed, it is requantized and written back to the video memory in integer form.

4. The method for optimizing large model training memory based on low-precision integer storage according to claim 2, characterized in that, Based on the quantized state parameters, the flow of quantized state parameters is managed through a global gradient stack structure according to the backpropagation order. State parameters are popped sequentially from the global gradient stack structure to update model parameters, thus completing the optimization. This includes: During backpropagation, the flow of quantized gradients is managed through a global gradient stack structure. In the backpropagation process, the weights in integer format are dequantized into floating-point numbers to calculate the gradients. After the gradients are quantized into integers, they are pushed onto the stack. When the parameters are updated, the gradients are popped out of the global gradient stack structure in sequence to update the model parameters.

5. The method for optimizing large model training memory based on low-precision integer storage according to claim 4, characterized in that, When performing reverse calculation, the integer weights stored in the video memory are first converted to floating-point format using a scaling factor before performing conventional reverse calculation. After obtaining the gradient, the gradient is requantized into an integer value and stored in the video memory; A global gradient stack structure is introduced to cache the gradients of each layer. Whenever a layer completes backpropagation, its quantized gradient is pushed onto the stack. The updated model parameters are as follows: During the parameter update phase, gradients are popped from the top of the stack layer by layer in reverse order. When updating each layer, the optimizer retrieves the quantized gradient of that layer from the stack and uses it together with the quantized first or second momentum stored in the stack for calculation. The integer storage structure is maintained throughout the process. When updating the calculation, the gradient is restored to floating-point form and then quantized and written back to the video memory.

6. A large model training memory optimization system based on low-precision integer storage, characterized in that, include: The data acquisition module is used to acquire the state parameters during the model training process and store them as integer values ​​in the video memory. The framework building module is used to dequantize integer values ​​to restore them to floating-point values ​​for calculation through dynamic scaling factor dequantization. After the calculation is completed, the values ​​are requantized and written back to the display memory in integer form to obtain a training framework with integer storage. The quantization module is used to perform differentiated quantization on the different numerical distribution characteristics of the state parameters under the training framework of integer storage to obtain the quantized state parameters. The update and optimization module is used to manage the flow of quantized state parameters through a global gradient stack structure according to the backpropagation order, based on the quantized state parameters. State parameters are popped out of the global gradient stack structure in order to update model parameters and complete the optimization. Under the training framework with integer storage, differentiated quantization is performed based on different numerical distribution characteristics of the state parameters to obtain quantized state parameters, including: For gradients and optimizer momentum, a uniform quantizer is used for quantization. For model weights, a dense-sparse quantizer is used for quantization. The dense-sparse quantizer first decomposes the weights into dense and sparse parts, performs uniform quantization on the dense part, and stores the sparse part in a compressed sparse row format. The quantization is performed using a uniform quantizer: Quantization and dequantization are mathematically defined as follows: in, It is a floating-point vector. For quantized integer vectors, This represents the floor function, where b∈N is the quantization bit width; and These are the quantization scale and the zero point, respectively. The quantization scale and the zero point are determined by... The lower and upper limits are determined as follows: Through linear mapping, the distribution of gradient and momentum is uniformly compressed into a finite integer range; during computation, it is dequantized into floating-point numbers in real time according to the scaling factor for forward and backward propagation; The quantization is performed using a dense-sparse quantizer: First, we analyze the distribution density of the weights, decomposing the weight matrix W into two complementary sub-matrices: one dense part D for the backbone distribution, and the other sparse part S for outliers; the formal definition is as follows: in, and The distribution threshold; The dense matrix D is quantized using a standard uniform quantizer, while the sparse matrix S retains the floating-point format and is saved using compressed sparse rows.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the large model training memory optimization method based on low-precision integer storage as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the large model training memory optimization method based on low-precision integer storage as described in any one of claims 1 to 5.

Images