A rate constraint optimization training method and system for large language models

Abstract

This application relates to the field of data processing technology, and in particular to a rate-constrained optimization training method and system for large language models. The method includes: first, statistically modeling the model parameters to generate a parameter distribution model; then, constructing a joint optimization loss function and training the parameter distribution model with code rate constraints; next, index mapping the trained model to obtain a quantized index; and finally, using exponential Golomb coding to entropy compress the quantized index to achieve parameter encoding. This application introduces distribution prior constraints and information rate regularization terms during the model training phase, enabling the model to adaptively form a compressible parameter structure during learning. After training, it combines exponential Golomb coding to achieve efficient parameter entropy compression, thereby significantly reducing model size and transmission costs while ensuring model performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a rate-constrained optimization training method and system for large language models. Background Technology

[0002] With generative large language models such as GPT and Llama demonstrating remarkable generalization capabilities in natural language processing and multimodal modeling, deep learning technology has entered a new phase of exponential growth in parameter size. However, this dramatic expansion of parameter size is a double-edged sword. While it enhances model intelligence, it also directly leads to a surge in computational resource consumption during model deployment. In resource-constrained applications such as mobile internet, embedded devices, and edge computing, the massive number of parameters not only causes severe inference latency but also incurs high network distribution and storage bandwidth costs. This has become a core technical bottleneck restricting the deep penetration of artificial intelligence technology into various industries.

[0003] However, current mainstream model compression techniques are still largely limited by the passive paradigm of "compressing after training." This model treats training optimization and size reduction as two independent and isolated stages, resulting in a lack of forward-looking awareness of the final deployment scale during parameter updates. This makes it difficult to proactively balance accuracy preservation and resource conservation from the underlying algorithmic logic. Meanwhile, existing compression schemes often rely excessively on the idealized prior assumption that parameters follow a Gaussian distribution. This is severely disconnected from the complex characteristics of real large-scale model weight distributions, such as asymmetry, spikes, and long tails. This directly limits the efficiency of entropy coding techniques like Huffman coding when dealing with real distributions, making it difficult to extract higher compression gains. This deep disconnect between training and compression, along with the inaccuracy of statistical modeling, has become a core bottleneck restricting large models from achieving extremely high compression ratios and extremely low accuracy loss in resource-constrained environments. Therefore, the industry urgently needs to build a systematic framework that deeply integrates compression objectives into the entire training cycle. By introducing more universal modeling methods such as the generalized Gaussian distribution, precise control and redundancy removal of parameter distribution can be achieved at the source of model evolution.

[0004] In recent years, large language model compression techniques have made some progress in eliminating redundant neural connections, reducing the accuracy of weight representations, and alleviating computational and storage pressure by introducing techniques such as pruning, quantization, and low-rank decomposition. However, when faced with extreme deployment scenarios where the parameter scale grows exponentially, these traditional methods increasingly expose their inherent limitations. Existing technologies, when dealing with extremely complex nonlinear feature distributions within the model, often lack accurate capture of deep semantic relationships. This leads to the inevitable destruction of the model's inherent structured information while significantly compressing parameters, directly manifesting as a significant degradation in the model's multi-task generalization ability and a precipitous drop in performance when performing complex logical reasoning tasks. This negative correlation between compression ratio and cognitive ability makes it difficult for existing methods to simultaneously achieve high model efficiency and superior intelligence in resource-constrained edge computing environments.

[0005] In summary, although existing compression techniques have achieved certain results for specific tasks, they still suffer from problems such as a disconnect between training and compression, inaccurate modeling of the real distribution, and limited compression gain. These limitations make it difficult to meet the dual demands of large language models for extremely high compression ratios and extremely low accuracy loss in diverse application environments. To address these limitations, this application introduces a rate-constrained optimization training framework based on generalized Gaussian distribution modeling, enabling precise control and redundancy removal of model parameter distribution during the training phase. Summary of the Invention

[0006] This application provides a rate-constrained optimization training method and system for large language models. By jointly modeling the storage requirements of model parameters and prediction accuracy, it addresses the pain points of large-scale pre-trained models when deployed to resource-constrained terminals or edge devices, such as high storage overhead, high computational resource consumption, and low distribution efficiency.

[0007] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a rate-constrained optimization training method for large language models, comprising the following steps: First, statistically modeling the model parameters to generate a parameter distribution model; then, constructing a joint optimization loss function and training the parameter distribution model with a code rate constraint; next, index mapping the trained model to obtain a quantization index; finally, using exponential Golomb coding to entropy compress the quantization index to achieve parameter encoding.

[0008] In some exemplary embodiments, statistical modeling of the model parameters includes: extracting all model parameters and performing statistical distribution analysis through training nodes on the server side; incorporating the information rate and task error of the model parameters into the training objective function; and using a generalized Gaussian distribution to accurately fit the trained parameters.

[0009] In some exemplary embodiments, a generalized Gaussian distribution is used to accurately fit the trained parameters, including: introducing shape parameters. v To adjust the kurtosis and tail characteristics of the distribution; when v When = 2, it degenerates into a Gaussian distribution. v When the value is 1, it degenerates into a Laplace distribution.

[0010] In some exemplary embodiments, during the bitrate-constrained training phase, the constructed loss function consists of a distortion term and an information rate term; wherein, the distortion term is used to measure the difference between the model output and the true label, and the information rate term represents the parameter complexity in the form of a discretized generalized Gaussian rate.

[0011] In some exemplary embodiments, an adjustable precision-complexity tradeoff is achieved through Lagrange multipliers, enabling the model to automatically converge to the optimal information encoding state during training; the expression for the loss function is:

[0012] in, These are the model's input, output, and weights, respectively. Let be the forward propagation function of the model. For the task loss function, It is the total number of parameters in the model. These are Lagrange multipliers, used to control the balance between model performance and compression ratio. In some exemplary embodiments, during the dynamic execution of training, a shape parameter adaptive estimation module is set to ensure that the bitrate constraint remains consistent with the actual parameter distribution; because v This reflects the steepness of the parameter distribution. v The values ​​change as the model converges during training; to achieve adaptive updates, the server node calculates the expected values ​​of the second and first moments of the model parameters at the beginning of each training batch, according to the following relationship:

[0013] Estimate the shape parameters of the current batch v And based on shape parameters v Update the DGGR (information rate) term in the loss function; this process ensures that the bit rate constraint is consistent with the actual parameter distribution, so that the training process always conforms to the generalized Gaussian prior in a statistical sense.

[0014] In some exemplary embodiments, a soft gradient pruning mechanism is introduced during the gradient optimization phase of model training to prevent gradient instability; when When the gradient is close to zero, it may tend to infinity, leading to training oscillations; a constant is introduced into the DGGR term. The original expression is rewritten to suppress gradient explosion when the parameters are close to zero, thus significantly improving training stability; the rewritten expression is:

[0015] This mechanism is executed automatically during the server's backpropagation phase, requiring no manual intervention.

[0016] In some exemplary embodiments, the trained model is indexed to obtain a quantized index, including: performing fixed-point processing on the model parameters in the server-side quantization module, discretizing the continuous weights into quantized values ​​with a fixed step size; the quantized parameter set is sent to the frequency statistics module, which counts and sorts all possible values, and establishes a one-to-one mapping table between parameter values ​​and indices to realize subsequent index encoding.

[0017] In some exemplary embodiments, exponential Golomb coding is used to entropy compress the quantization index, including: selecting different EG parameters based on different shape parameters. k Encode, when k A smaller file size can achieve better compression results; the EG encoding is performed by a compression server, and the output is a compressed bitstream file and its corresponding code table.

[0018] Secondly, this application also provides a rate-constrained optimization training system for large language models. This system is implemented using the rate-constrained optimization training method for large language models described in the above embodiments. The system includes: a parameter statistical modeling module, a rate-constrained training module, and an encoding module connected in sequence. The parameter statistical modeling module is used to statistically model the model parameters and generate a parameter distribution model. The rate-constrained training module is used to construct a joint optimization loss function and perform rate-constrained training on the parameter distribution model. The encoding module is used to perform index mapping on the trained model to obtain a quantization index. Exponential Golomb coding is used to entropy compress the quantization index to achieve parameter encoding.

[0019] The technical solution provided in this application has at least the following advantages: This application provides a rate-constrained optimization training method and system for large language models. The method includes the following steps: First, statistical modeling of model parameters is performed to generate a parameter distribution model; then, a joint optimization loss function is constructed, and the parameter distribution model is trained with code rate constraints; next, the trained model is indexed to obtain a quantized index; finally, exponential Golomb coding is used to entropy compress the quantized index to achieve parameter encoding. This application addresses the high redundancy parameter problem in the storage, communication, and deployment stages of very large language models (LLMs) by proposing an efficient parameter encoding method based on generalized Gaussian distribution modeling and code rate constraint training. This scheme introduces distribution prior constraints and information rate regularization terms during model training, enabling the model to adaptively form a compressible parameter structure during learning. After training, exponential Golomb coding is combined to achieve efficient parameter entropy compression, thereby significantly reducing model size and transmission costs while ensuring model performance. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0021] Figure 1 This is a flowchart illustrating a rate-constrained optimization training method for large language models, provided as an embodiment of this application. Detailed Implementation

[0022] As can be seen from the background technology, existing technologies suffer from problems such as disconnect between training and compression, inaccurate modeling of the real distribution, and limited compression gain, making it difficult to meet the dual requirements of large language models for extremely high compression ratio and extremely low precision loss in diverse application environments.

[0023] The closest existing technology is a regularized neural network compression and sparsification method. This technique introduces the sum of the absolute values ​​of parameters as a penalty term into the loss function during model training, aiming to induce the model parameters to evolve towards sparsity. Its technical principle lies in utilizing the geometric characteristics of peaks at the coordinate axes using L1 and L2 regularization techniques, causing many weight parameters that contribute little to the prediction results to be compressed to zero or close to zero during gradient descent optimization. This method can improve the model's generalization performance while achieving structured or unstructured pruning by removing redundant connections, thereby reducing model storage volume and computational complexity. In practice, the system typically pre-sets a fixed Lagrange multiplier, applying uniform contraction pressure to all weights during backpropagation. Existing techniques often perform non-zero parameter compression at the posterior stage and exhibit significant modeling bias. However, due to its computational simplicity and direct integration into existing deep learning training frameworks, this method has become one of the core techniques for achieving neural network weight sparsity and preliminary model compression.

[0024] While regularized neural network compression and sparsification methods have achieved some success in reducing model size and computational complexity, they still face several technical limitations when dealing with the exponential growth in parameter size and the increasing complexity of distribution characteristics in large language models. First, existing methods typically presuppose that model parameters follow an idealized Laplace distribution, using this as the theoretical basis for weight sparsification. However, in real-world large-scale neural networks, parameter distributions often exhibit dynamic and diverse characteristics, displaying not only significant asymmetry and peakedness but also long-tail effects. This idealized prior assumption fails to effectively characterize the real distribution, resulting in a lack of specificity in the model parameter shrinkage strategy during sparsification, thus leading to problems such as limited compression efficiency and model performance degradation.

[0025] Secondly, existing regularization-based sparsification methods fail to establish a mathematical coupling relationship between the cost of parameter sparsification and the efficiency of entropy coding from the perspective of rate-distortion optimization. This fragmented optimization process results in a lack of forward-looking awareness of the final compression target during the training phase, making it difficult to dynamically balance storage overhead and performance loss during model updates, thus failing to approach the compression limit in the information theory sense.

[0026] Furthermore, existing regularization schemes typically perform posterior encoding only after training convergence, failing to adaptively adjust based on the real-time shape of parameter distribution. The compression process lacks dynamic feedback and theoretical guidance, limiting encoding efficiency to a fixed prior model and making it difficult to meet the dual requirements of maximum storage efficiency and real-time communication efficiency when deploying large language models on terminal devices.

[0027] Existing L1 regularization-based compression methods primarily focus on the sparsity constraints during the training phase, mainly concerned with the compression effect of parameter magnitudes, without forming a closed-loop linkage with system-level aspects such as entropy coding, communication bandwidth allocation, and storage strategies. Their compression process remains at the offline processing or post-hoc optimization level, lacking collaborative awareness and real-time adaptation between model training dynamics and deployment requirements. Therefore, it is difficult to achieve adaptive compression and efficient transmission based on parameter distribution evolution in the actual deployment of large language models.

[0028] To address the problem that existing large-scale language models fail to simultaneously balance model performance and parameter compression during training, resulting in enormous computational and storage overhead during the inference and deployment phases, this application proposes a training-time compression technique based on rate-distortion optimization constraints. This scheme introduces an information rate constraint mechanism during the model training phase, fundamentally achieving joint optimization of model accuracy and complexity, thus overcoming the bottlenecks of the traditional "train first, compress later" paradigm in terms of performance preservation and structural constraints.

[0029] The core idea of ​​this application is to incorporate the information rate of model parameters and task error into the training objective function, and to achieve an adjustable precision-complexity tradeoff through Lagrange multipliers, enabling the model to automatically converge to the optimal information encoding state during training. Unlike existing methods that only perform post-processing compression on fixed-structure models, this application introduces a discrete generalized Gaussian rate constraint term during the training phase to dynamically estimate the shape parameters of the model parameter distribution, thereby adaptively adjusting the sparsity and entropy structure of the model parameters, making the training results directly compressible and robust.

[0030] At the model encoding level, this application proposes for the first time to introduce the exponential Golomb encoding mechanism into the entropy encoding process of large language model parameters. Taking into account the statistical characteristics of model parameters following a generalized Gaussian distribution and having a shape parameter less than 2, a unified encoding scheme is designed that eliminates the need for adaptive table construction and achieves encoding efficiency close to the entropy limit. This method demonstrates significant code rate advantages across various models and tasks, reducing parameter storage by an average of 60%–90% without sacrificing accuracy, and significantly reducing communication and energy consumption overhead during inference deployment.

[0031] Unlike existing fixed regularization methods based on the Gaussian assumption, the discrete generalized Gaussian information rate proposed in this application achieves adaptive distribution adjustment through dynamic estimation of shape parameters. This allows the statistical characteristics of the model's parameters to be effectively characterized at different training stages, thus theoretically unifying and extending the definition paradigm of traditional regularization terms. This mechanism not only improves the model's convergence speed and robustness but also maintains the model's expressive power and generalization performance under high sparsity conditions.

[0032] Through the above-mentioned innovative components, the technical solution of this application realizes a paradigm shift from post-processing of parameter compression to intrinsic compression of training. While ensuring model accuracy, it significantly reduces model complexity and provides a unified, universal and hardware-friendly training framework for the efficient deployment of large language models in edge computing, mobile terminals and distributed environments.

[0033] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0034] This application provides a rate-constrained optimization training method for large language models, comprising the following steps: First, statistical modeling of model parameters to generate a parameter distribution model; then, constructing a jointly optimized loss function and training the parameter distribution model with code rate constraints; next, index mapping of the trained model to obtain a quantization index; finally, entropy compression of the quantization index using exponential Golomb coding to achieve parameter encoding. This application provides a rate-constrained optimization training method and system for large language models, which addresses the pain points of large-scale pre-trained models when deployed to resource-constrained terminals or edge devices, such as high storage overhead, high computational resource consumption, and low distribution efficiency, by jointly modeling the storage requirements and prediction accuracy of model parameters.

[0035] First, in the parametric statistical modeling stage, this application performs full extraction and statistical distribution analysis of model parameters through server-side training nodes. Unlike traditional initialization methods that assume parameters follow a Gaussian distribution, this application uses a generalized Gaussian distribution to accurately fit the trained parameters. This distribution introduces a shape parameter. v To adjust the kurtosis and tail characteristics of the distribution, when v When = 2, it degenerates into a Gaussian distribution. v When the coefficient of performance is 1, it degenerates into a Laplace distribution. Extensive experiments have shown that after training convergence, the shape parameter of the parameter distribution of most language models... v All values ​​are less than 2, exhibiting peak and heavy-tailed characteristics, indicating that the generalized Gaussian distribution better characterizes the true weight distribution. This modeling process is executed by the model training server, generating the parameter distribution model and providing the probability density basis for subsequent bitrate constraint calculations.

[0036] Subsequently, during the bitrate-constrained training phase, this application constructs a jointly optimized loss function to simultaneously minimize the model's task error and parameter information rate. This loss function consists of a traditional distortion term and an information rate term, where the distortion term measures the difference between the model's output and the true label, and the information rate term represents the parameter complexity in the form of a discretized generalized Gaussian rate. The overall optimization objective can be expressed as:

[0037] in, These are the model's input, output, and weights, respectively. Let be the forward propagation function of the model. For the task loss function, It is the total number of parameters in the model. These are Lagrange multipliers used to control the balance between model performance and compression ratio. During each parameter update, the server training nodes perform forward and backward propagation based on the above objective function, gradually forming a parameter distribution with low redundancy and compressibility while maintaining accuracy. v =1 or v When =2, the regularization term degenerates into the traditional L1 and L2 regularizations respectively, and can therefore be regarded as its generalized form.

[0038] During the dynamic execution of training, this application further includes an adaptive estimation module for shape parameters. Because... v This reflects the steepness of the parameter distribution, and its value changes as the model converges during training. To achieve adaptive updates, the server node calculates the expected values ​​of the second and first moments of the model parameters at the beginning of each training batch, according to the following relationship:

[0039] Estimate the shape parameters of the current batch v This process updates the DGGR term in the loss function accordingly. This ensures that the bitrate constraint remains consistent with the actual parameter distribution, guaranteeing that the training process statistically conforms to the generalized Gaussian prior.

[0040] In the gradient optimization stage of model training, this application introduces a soft gradient pruning mechanism to prevent gradient instability. When the parameters are close to zero, the gradient may tend to infinity, leading to training oscillations. To address this, this application introduces a constant in the DGGR term. The original expression is rewritten as:

[0041] This suppresses gradient explosion when parameters are close to zero, significantly improving training stability. This mechanism is executed automatically during the backpropagation phase on the server, requiring no manual intervention.

[0042] After model training is complete, the system enters the parameter encoding stage. This application first performs fixed-point quantization on the model parameters in the server-side quantization module, discretizing the continuous weights into quantized values ​​with a fixed step size. The quantized parameter set is then sent to the frequency statistics module, where all possible values ​​are counted and sorted, and a one-to-one mapping table between parameter values ​​and indices is established to enable subsequent index encoding. At this point, the statistical distribution of the model parameters can still be approximated as a generalized Gaussian form, but it exhibits discretization characteristics after the quantization operation.

[0043] After completing the index mapping, the encoding module of this application employs exponential Golomb coding to entropy compress the quantized index. This encoding method eliminates the need to construct complex Huffman tables, maintaining a high degree of structure without sacrificing encoding efficiency. Depending on the shape parameters, this application selects different EG parameters. k Encode, when k A smaller value yields better compression results. Experiments show that for typical language models (such as BERT, GPT2, Llama3, etc.), taking a smaller value results in better compression. k Encoding efficiency is highest and average code length is shortest when the value is 0. EG encoding is performed by a compression server, and the output is a compressed bitstream file and its corresponding code table.

[0044] During the encoding process, the system first counts the frequency of each parameter value based on the quantization results, then sorts them according to frequency and assigns a corresponding index to each quantized value. The encoder inputs the index sequence into the EG encoding module to generate a continuous bitstream as the final output. This bitstream can be directly used for model storage, transmission, or cross-platform deployment. Compared with traditional Huffman coding, the EG encoding in this application does not require rebuilding the code table for each model, can support the compression of multiple model parameters with a unified structure, and is simple to implement and has low computational cost at the decoding end.

[0045] Through the above technical process, this application achieves information rate self-constrained parameter optimization during the model training stage, making the training convergence results naturally sparse and compressible. In the model output stage, it combines structured exponential Golomb coding to complete the entire compression chain from parameter distribution modeling and training optimization to entropy coding. The overall solution achieves synergistic optimization of model accuracy and storage efficiency, significantly reducing the average code length of model parameters and communication overhead. It is particularly suitable for high-efficiency transmission and long-term storage requirements in distributed computing, edge device deployment, and multi-model joint inference scenarios.

[0046] This application introduces parameter modeling based on a generalized Gaussian distribution and bitrate-constrained training. It achieves proactive control over the complexity of parameter information during the language model training phase, enabling the model to balance performance and compression ratio during optimization. This technique makes the model parameters tend towards a sparse, low-entropy distribution, providing a statistical basis for subsequent efficient encoding and effectively overcoming the bottleneck in existing technologies where post-training compression struggles to achieve an optimal rate-distortion balance.

[0047] During training, this application suppresses overfitting of redundant features and avoids gradient divergence by discretizing the generalized Gaussian rate constraint and using a soft gradient pruning mechanism, thus ensuring training stability. This mechanism enables the model to achieve a better balance between compression and performance, significantly reducing storage and communication overhead without significantly reducing accuracy.

[0048] In the parametric encoding stage, by combining the exponential Golomb coding strategy, this application achieves high compression efficiency and cross-model universality. EG coding does not require redesigning code tables for different models, and its code length distribution naturally adapts to the generalized Gaussian morphological parameters, making the coding efficiency approach the information theory limit while simplifying hardware implementation.

[0049] In summary, this application achieves end-to-end optimization of training and compression through a technical chain of "distributed modeling - rate-constrained training - dynamic shape estimation - structured entropy coding", which significantly reduces the model code length (compression rate of 70%~90%), maintains the original accuracy, and solves the problems of fragmented training and compression and limited coding efficiency in existing methods, providing technical support for efficient deployment of edge devices and low-bandwidth environments.

[0050] Furthermore, this application also provides a rate-constrained optimization training system for large language models. This system is implemented using the rate-constrained optimization training method for large language models as described in the above embodiments. The system includes: a parameter statistical modeling module, a rate-constrained training module, and an encoding module connected sequentially. The parameter statistical modeling module is used to statistically model the model parameters and generate a parameter distribution model. The rate-constrained training module is used to construct a joint optimization loss function and perform rate-constrained training on the parameter distribution model. The encoding module is used to perform index mapping on the trained model to obtain a quantization index. Exponential Golomb coding is used to entropy compress the quantization index to achieve parameter encoding.

[0051] Based on the above technical solutions, this application provides a rate-constrained optimization training method and system for large language models. The method includes the following steps: First, statistical modeling of the model parameters generates a parameter distribution model; then, a joint optimization loss function is constructed, and the parameter distribution model is trained with rate constraints; next, index mapping is performed on the trained model to obtain a quantized index; finally, exponential Golomb coding is used to entropy compress the quantized index to achieve parameter encoding. This application addresses the high redundancy parameter problem in the storage, communication, and deployment stages of very large language models (LLMs) by proposing an efficient parameter encoding method based on generalized Gaussian distribution modeling and rate-constrained training. This scheme introduces distribution prior constraints and information rate regularization terms during the model training stage, enabling the model to adaptively form a compressible parameter structure during learning. After training, exponential Golomb coding is combined to achieve efficient parameter entropy compression, thereby significantly reducing model size and transmission costs while ensuring model performance.

[0052] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A rate-constrained optimization training method for large language models, characterized in that, Includes the following steps: Perform statistical modeling on the model parameters to generate a parameter distribution model; Construct a joint optimization loss function and perform bitrate-constrained training on the parameter distribution model; The trained model is indexed to obtain a quantized index; Exponential Golomb coding is used to entropy compress the quantization index to achieve parameter encoding.

2. The rate-constrained optimization training method for large language models according to claim 1, characterized in that, Statistical modeling of the model parameters includes: The model parameters are fully extracted and statistically distributed through the training nodes on the server side; The information rate of the model parameters and the task error are incorporated into the training objective function, and the trained parameters are accurately fitted using a generalized Gaussian distribution.

3. The rate-constrained optimization training method for large language models according to claim 2, characterized in that, The parameters after training are accurately fitted using a generalized Gaussian distribution, including: Introducing shape parameters v To adjust the kurtosis and tail characteristics of the distribution; when v When = 2, it degenerates into a Gaussian distribution. v When the value is 1, it degenerates into a Laplace distribution.

4. The rate-constrained optimization training method for large language models according to claim 1, characterized in that, During the bitrate-constrained training phase, the constructed loss function consists of a distortion term and an information rate term; among which, The distortion term measures the difference between the model output and the true label, while the information rate term represents the parameter complexity in the form of a discretized generalized Gaussian rate.

5. The rate-constrained optimization training method for large language models according to claim 4, characterized in that, An adjustable precision-complexity tradeoff is achieved using Lagrange multipliers, allowing the model to automatically converge to the optimal information encoding state during training; the expression for the loss function is: in, These are the model's input, output, and weights, respectively. Let be the forward propagation function of the model. For the task loss function, It is the total number of parameters in the model. These are Lagrange multipliers used to control the balance between model performance and compression ratio.

6. The rate-constrained optimization training method for large language models according to claim 5, characterized in that, During the dynamic execution of training, an adaptive estimation module for shape parameters is set up to ensure that the bitrate constraint is consistent with the actual parameter distribution; because v This reflects the steepness of the parameter distribution. v The values ​​change as the model converges during training; to achieve adaptive updates, the server node calculates the expected values ​​of the second and first moments of the model parameters at the beginning of each training batch, according to the following relationship: Estimate the shape parameters of the current batch v And based on shape parameters v Update the DGGR information rate term in the loss function; this process ensures that the bit rate constraint is consistent with the actual parameter distribution, so that the training process always conforms to the generalized Gaussian prior in a statistical sense.

7. The rate-constrained optimization training method for large language models according to claim 6, characterized in that, In the gradient optimization stage of model training, a soft gradient pruning mechanism is introduced to prevent gradient instability. when When the parameters are close to zero, the gradient may tend to infinity, leading to training oscillations. Introduce a constant into the DGGR term. The original expression is rewritten to suppress gradient explosion when the parameters are close to zero, thus significantly improving training stability; the rewritten expression is: This mechanism is executed automatically during the server's backpropagation phase, requiring no manual intervention.

8. The rate-constrained optimization training method for large language models according to claim 1, characterized in that, The trained model is indexed to obtain a quantized index, including: In the server-side quantization module, the model parameters are processed by fixed-point quantization, and the continuous weights are discretized into quantized values ​​with a fixed step size. The quantized parameter set is sent to the frequency statistics module, which counts and sorts all possible values ​​and establishes a one-to-one mapping table between parameter values ​​and indices to enable subsequent index encoding.

9. The rate-constrained optimization training method for large language models according to claim 1, characterized in that, Entropy compression of the quantization index is performed using exponential Golomb coding, including: Choose different EG parameters based on different shape parameters. k Encode, when k A smaller size allows for better compression. The EG encoding is performed by a compression server, and the output is a compressed bitstream file and its corresponding code table.

10. A rate-constrained optimization training system for large language models, wherein the system is implemented using the rate-constrained optimization training method for large language models as described in any one of claims 1 to 9, characterized in that, The system includes: a parameter statistical modeling module, a bitrate constraint training module, and an encoding module connected in sequence; wherein, The parameter statistical modeling module is used to statistically model the model parameters and generate a parameter distribution model; The bitrate constraint training module is used to construct a joint optimization loss function and perform bitrate constraint training on the parameter distribution model. The encoding module is used to perform index mapping on the trained model to obtain the quantization index; exponential Golomb coding is used to entropy compress the quantization index to achieve parameter encoding.

Images