Mixed precision quantization method, apparatus, device, medium, and program product

Abstract

This disclosure provides a mixed-precision quantization method, apparatus, device, medium, and program product, relating to the field of deep learning technology. The method is applied to a large language model, where the feedforward network of the large language model includes a multiplicative structure configured with a first weight matrix and a second weight matrix. The method includes: determining the statistical characteristics of the multiplicative structure on each output channel of the feedforward network based on the first and second weight matrices; determining the numerical amplification risk index of the multiplicative structure on each output channel based on the statistical characteristics; and configuring a corresponding computational precision for each output channel according to the numerical amplification risk index; wherein output channels with different numerical amplification risk indices are configured with different computational precisions. This method eliminates the need for real-time scanning of input data, significantly reducing runtime overhead.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, specifically to a mixed-precision quantization method, apparatus, device, medium, and program product. Background Technology

[0002] With the continuous growth of the parameter size and training data size of large-scale language models, low-precision numerical representation has become a key technical means to reduce computational costs and storage overhead. However, during low-precision training or inference, the uneven distribution of numerical values ​​within the model, especially the presence of outliers, can easily lead to problems such as numerical overflow and decreased precision. Summary of the Invention

[0003] This disclosure presents a hybrid precision quantization method, apparatus, device, medium, and machine program product.

[0004] In a first aspect, embodiments of this disclosure propose a hybrid precision quantization method applied to a large language model. The feedforward network of the large language model includes a multiplicative structure, which is configured with a first weight matrix and a second weight matrix. The method includes: determining the statistical characteristics of the multiplicative structure in each output channel of the feedforward network based on the first weight matrix and the second weight matrix; determining the numerical amplification risk index of the multiplicative structure in each output channel based on the statistical characteristics; and configuring a corresponding computational precision for each output channel according to the numerical amplification risk index; wherein output channels with different numerical amplification risk indices are configured with different computational precisions.

[0005] In some embodiments, the statistical features include a weight norm index; determining the statistical features of the multiplicative structure on each output channel of the feedforward network based on the first weight matrix and the second weight matrix includes: for each output channel, calculating the first norm value of the first weight matrix on the output channel and the second norm value of the second weight matrix on the output channel, and determining the weight norm index of the output channel based on the first norm value and the second norm value; wherein the weight norm index is used to characterize the magnitude of the first weight matrix and the second weight matrix on the output channel.

[0006] In some embodiments, the statistical features further include a weight correlation index; determining the statistical features of the multiplicative structure on each output channel of the feedforward network based on the first weight matrix and the second weight matrix further includes: for each output channel, determining the weight correlation index of the output channel based on the correlation coefficient of the first weight matrix and the second weight matrix on the output channel; wherein the correlation coefficient is used to characterize the numerical sign consistency of the first weight matrix and the second weight matrix on the output channel.

[0007] In some embodiments, the statistical features include a weight norm index and / or a weight correlation index; based on the statistical features, determining the numerical amplification risk index of the multiplicative structure in each output channel of the feedforward network includes: for each output channel, determining a risk metric value of the output channel according to the weight norm index and / or the weight correlation index; and generating a numerical amplification risk index of the output channel according to the risk metric value and at least one preset threshold.

[0008] In some embodiments, the risk metric includes a weight norm index value and a weight correlation index value; at least one preset threshold includes a first preset threshold and a second preset threshold; generating a numerical amplification risk index for the output channel based on the risk metric and at least one preset threshold includes: comparing the weight norm index value with the first preset threshold and comparing the weight correlation index value with the second preset threshold; generating a first mask value for the output channel when the weight norm index value is greater than the first preset threshold and the weight correlation index value is greater than the second preset threshold; generating a second mask value for the output channel when the weight norm index value is less than the first preset threshold or the weight correlation index value is less than the second preset threshold; wherein the first mask value or the second mask value is a risk index for the output channel.

[0009] In some embodiments, at least one preset threshold includes a third preset threshold; determining a risk metric for the output channel based on a weight norm index and / or a weight correlation index; generating a numerical amplification risk index for the output channel based on the risk metric and at least one preset threshold, including: multiplying the weight norm index value and the weight correlation index value to obtain the risk metric for the output channel; comparing the risk metric with the third preset threshold; generating a first mask value for the output channel if the risk value is greater than the third preset threshold; generating a second mask value for the output channel if the risk value is less than the third preset threshold; wherein the first mask value or the second mask value is the risk index value of the output channel.

[0010] In some embodiments, the calculation precision includes a first precision level and a second precision level, and the calculation precision of the first precision level is higher than that of the second precision level; according to the numerical amplification risk index, the corresponding calculation precision is configured for each output channel, including: when the numerical amplification risk index is a first mask value, configuring a first precision level for the output channel corresponding to the numerical amplification risk index; when the numerical amplification risk index is a second mask value, configuring a second precision level for the output channel corresponding to the numerical amplification risk index.

[0011] In some embodiments, the method further includes: re-acquiring the updated first weight matrix and second weight matrix of the large language model according to a preset update cycle; and reconfiguring the calculation precision for each output channel based on the updated first weight matrix and second weight matrix.

[0012] In some embodiments, the preset update cycle includes: every preset number of training iterations; or in response to a trigger event for saving a large language model; wherein the trigger event includes at least one of the following: reaching a preset save interval, receiving a save instruction, or meeting a preset performance indicator threshold.

[0013] In some embodiments, the method further includes: performing feedforward network computation on the respective output channel based on the computational precision configured for each output channel.

[0014] Secondly, embodiments of this disclosure also provide a mixed-precision quantization device, comprising: a first determining module, configured to determine the statistical characteristics of a multiplicative structure in a feedforward network of a large language model on each output channel of the feedforward network based on a first weight matrix and a second weight matrix, wherein the multiplicative structure is configured with a first weight matrix and a second weight matrix; a second determining module, configured to determine the numerical amplification risk index of the multiplicative structure on each output channel of the feedforward network based on the statistical characteristics; and a configuring module, configured to configure a corresponding computational precision for each output channel according to the numerical amplification risk index; wherein output channels with different numerical amplification risk indices are configured with different computational precisions.

[0015] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the mixed-precision quantization method as described in any implementation of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the mixed-precision quantization method as described in any implementation of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the mixed-precision quantization method as described in any implementation of the first aspect.

[0018] The mixed-precision quantization method provided in this disclosure determines the statistical characteristics of each output channel based on a first weight matrix and a second weight matrix. This allows for the determination of numerical amplification risk indicators for each output channel based on these statistical characteristics. Furthermore, it enables the prediction of potential risks associated with low-precision quantization for each output channel based on these numerical amplification risk indicators. Based on the determined numerical amplification risk indicators for each output channel, an appropriate computational precision is assigned to each channel, dynamically selecting between high and low precision calculations to achieve a differentiated mixed-precision computation strategy. This disclosure eliminates the need for real-time scanning of input activation, significantly reducing runtime overhead. Simultaneously, assigning appropriate computational precision to each channel avoids numerical overflow and precision degradation issues associated with low-precision quantization, while maximizing the advantages of low-precision computation in reducing storage overhead and improving computational efficiency. Attached Figure Description

[0019] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of a hybrid precision quantization method provided in this embodiment of the disclosure; Figure 2 A structural block diagram of a hybrid precision quantization device provided in this disclosure embodiment; Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0021] Deep learning models can include Large Language Models (LLMs), computer vision models, reinforcement learning models, and more. Among these, Large Language Models refer to large-scale neural network models trained using deep learning techniques, capable of understanding and generating natural language text. For example, a Large Language Model can include a Transformer model, which includes a Multi-Layer Perceptron (MLP). A Multi-Layer Perceptron can also be simply referred to as a feedforward network.

[0022] In the feedforward network of the Transformer model, multiplicative structures such as the Swish-Gated Linear Unit (SwishGLU) are included. These structures typically consist of a first linear transformation branch and a second linear transformation branch, where the first branch is configured with a first weight matrix and the second branch with a second weight matrix. The output of the multiplicative structure is obtained by element-wise multiplying the outputs of the two linear transformation branches, and its output amplitude tends to gradually increase during training, thus adversely affecting the numerical stability during low-precision quantization.

[0023] Based on this, a mixed precision quantization scheme can be adopted during model training.

[0024] In some embodiments, a mixed precision scheme based on runtime outlier scanning can be adopted during model training.

[0025] For example, during the training process of an 8-bit integer large language model, the input data is scanned during operation to identify whether there are outliers in the input activation that exceed a preset threshold. The feature dimensions containing outliers are divided into high-precision paths, while the remaining feature dimensions are calculated using low-precision paths.

[0026] However, such methods require real-time detection and judgment of the input data during operation, which not only introduces additional online computational overhead, but also makes their quantization strategy highly dependent on the numerical distribution of the specific input data. When the distribution of the input data changes, the quantization path selection result also changes, which is not conducive to efficient and stable scheduling and control in large-scale training scenarios.

[0027] In other embodiments, the numerical amplification problem can be mitigated by introducing additional scaling factors or adjusting the computational structure.

[0028] Specifically, in feedforward networks containing multiplicative structures such as SwishGLU, the output value tends to gradually amplify during training, thus adversely affecting low-precision quantization. In this embodiment, the problem can be mitigated by modifying the original model's computational structure or operator implementation, thereby enabling outlier detection post-processing or introducing additional smoothing techniques.

[0029] However, this method is highly invasive in engineering, and due to the lack of predictive ability for the upper bound of numerical amplification, it is difficult to determine in advance whether there is a risk of low-precision quantization without scanning the input activation, resulting in a highly passive quantization control strategy.

[0030] In the two embodiments described above, due to the lack of reliable risk assessment criteria, a more conservative quantitative strategy is often adopted in practical applications. That is, when it is uncertain whether there is a risk, high-precision calculation tends to be used, thereby reducing the advantages of low-precision calculation in terms of performance and energy efficiency.

[0031] Furthermore, this disclosure also provides a mixed-precision quantization scheme that does not require runtime scanning of input data, does not require modification of the model structure, and can predict the risk of numerical amplification in advance.

[0032] Figure 1 The flowchart illustrates a mixed-precision quantization method provided in this embodiment of the disclosure. This method can be applied to large language models, specifically for training large language models or for inference based on large language models.

[0033] See Figure 1 The process specifically includes the following steps: Step 101: Based on the first weight matrix and the second weight matrix, determine the statistical characteristics of the multiplicative structure in each output channel of the feedforward network.

[0034] The feedforward network of the large language model includes a multiplicative structure, which is configured with a first weight matrix and a second weight matrix.

[0035] Specifically, for multiplicative structures in feedforward networks, such as the SwishGLU structure, it typically contains two parallel linear transformation branches, configured with a first weight matrix W1 and a second weight matrix W2, respectively, where W1 and W2 both belong to d in multiplied by d out A real matrix of columns, d in d represents the input dimension of the input data. out This refers to the output channels for output data. Each output channel i (i=1) , 2 , ...d out The i-th column vector corresponding to the first weight matrix is ​​denoted as W. 1,iEach output channel i corresponds to the i-th column vector of the second weight matrix, denoted as W. 2,i And W 1,i and W 2,i All belong to including d in A vector of n real elements.

[0036] Statistical features are calculated independently for each output channel and are used to identify the numerical distribution characteristics of the two weight matrices on that output channel, thereby reflecting the potential risk of numerical amplification in low-precision calculations for that output channel.

[0037] Furthermore, the two linear transformation branches output using their respective configured weight matrices, and the multiplicative structure obtains its output by element-wise multiplying the outputs of the two linear transformation branches. Therefore, this statistical feature is derived from the weight matrix of the multiplicative structure in the large language model and is independent of the input data. Moreover, this mixed-precision quantization method, when applied to training or inference based on a large language model, can be completed offline before training or inference, without requiring runtime input scanning.

[0038] Step 102: Based on statistical characteristics, determine the numerical amplification risk index of the multiplicative structure on each output channel.

[0039] Specifically, the statistical feature is a feature index obtained by performing channel-level statistics on the first and second weight matrices of the multiplicative structure according to the output channel dimension. This statistical feature can identify the matching degree between the weight value of a single output channel and the two weights. The numerical amplification risk index is a quantitative index constructed based on this statistical feature, which can characterize the probability and severity of numerical overflow and precision degradation of each output channel during low-precision quantization.

[0040] Specifically, for the first weight matrix W1 and the second weight matrix W2 of the multiplicative structure, they can be split according to the output channel dimension of the feedforward network to obtain two sets of weight channels that correspond one-to-one with the output channel dimension. Specifically, the first weight matrix W1 is split into C independent weight channels W 1,i (i=1) , 2 , ...C, where C is the total number of output channels), the second weight matrix W2 is split into C independent weight channels W 2,i (i=1) , 2 , ...C); After splitting, the i-th output channel uniquely corresponds to a set of weighted channel pairs (W 1,i and W 2,i Based on this set of weighted channels, the risk of numerical amplification in the i-th output channel can be determined.

[0041] It should be noted that the splitting of the weight matrix involved in this embodiment is only a dimensional division at the statistical analysis level, and does not require any modification to the storage or calculation structure of the original weight matrix of the model, so the engineering intrusion is zero.

[0042] Step 103: Configure the corresponding calculation precision for each output channel based on the numerical amplification risk index.

[0043] Among them, the numerical amplification risk indicators have different output channel configurations with varying calculation accuracies.

[0044] Specifically, the computational precision can include high-precision and low-precision levels. For example, low-precision levels can preferably use mainstream low-precision numerical representations such as 8-bit Integer (INT8) or 8-bit Floating Point (FP8). High-precision levels can preferably use numerical representations higher than low-precision levels, such as 16-bit Floating Point (FP16), 16-bit Brain Floating Point (BF16), or 32-bit Floating Point (FP32). The computational precision of each output channel is strongly correlated with the numerical amplification risk index. The magnitude of the risk index directly determines the configured computational precision level of the channel, and different risk index values ​​correspond to different computational precision configurations, achieving personalized precision matching at the channel level.

[0045] Output channels with higher numerical amplification risk are more prone to overflow or precision loss in low-precision calculations, thus requiring higher computational precision (such as FP16, BF16) to ensure numerical stability. Conversely, output channels with lower numerical amplification risk can be assigned lower computational precision (such as INT8, FP8) to reduce computational overhead and storage usage. This differentiated configuration of "high-risk, high-precision; low-risk, low-precision" ensures computational stability while minimizing computational overhead and storage usage.

[0046] In some embodiments, the numerical amplification risk index of all output channels of the feedforward network can first be classified into continuous or discrete levels, ensuring that different risk index values ​​correspond to different risk levels. The higher the risk index value, the higher the risk level of the corresponding output channel, and the higher the requirement for computational accuracy. This provides a clear criterion for subsequent accuracy configuration. Furthermore, based on the risk level classification results, a unique matching relationship between risk level and computational accuracy can be established, ensuring that output channels with different risk levels are configured with different computational accuracies. Specifically, output channels with higher risk levels are configured with higher computational accuracy to avoid overflow and accuracy degradation caused by numerical amplification through higher-precision numerical representation; output channels with lower risk levels are configured with lower computational accuracy to fully utilize the performance and energy efficiency advantages of low-precision computing.

[0047] In some embodiments, risk levels are classified based on the magnitude of risk indicators. This classification can be flexibly adjusted according to the quantization scenario of the large language model (such as training or inference scenarios), the type of low-precision numerical values ​​(such as INT8 or FP8), and the actual accuracy requirements. This disclosure does not impose a unique limitation on the specific method of classifying risk levels based on the numerical values ​​of risk indicators.

[0048] The mixed-precision quantization method provided in this disclosure determines the statistical characteristics of each output channel based on a first weight matrix and a second weight matrix. This allows for the determination of numerical amplification risk indicators for each output channel based on these statistical characteristics. Furthermore, it enables the prediction of potential risks associated with low-precision quantization for each output channel based on these numerical amplification risk indicators. By analyzing the statistical characteristics of the weight channels, this method can predict the upper bound of the output values ​​without relying on specific input data. It eliminates the need for real-time scanning of input activation, significantly reducing runtime overhead. Moreover, based on the determined numerical amplification risk indicators for each output channel, this method assigns an appropriate computational precision to each channel, dynamically selecting between high and low precision calculations. This achieves a differentiated mixed-precision computation strategy, avoiding numerical overflow and precision degradation issues associated with low-precision quantization while maximizing the advantages of low-precision computation in reducing storage overhead and improving computational efficiency. This addresses the technical shortcomings of conservative quantization in existing technologies, which often result in insufficient utilization of low-precision capabilities. Additionally, applying this method to model training or inference can improve the efficiency of training or inference.

[0049] Furthermore, in some embodiments, the mixed precision quantization method not only includes steps 101-103 as described above, but may also include: performing feedforward network calculations on the corresponding output channels based on the calculation precision configured for each output channel.

[0050] Specifically, different computational precisions correspond to different data formats, which in turn have different numerical ranges, precisions, and storage sizes. Lower precision data occupies less memory space, reducing memory bandwidth pressure, while different precisions have different dynamic ranges, and lower precision is more prone to overflow.

[0051] This disclosure achieves a precise match between computational precision and channel requirements by configuring each output channel independently based on differentiated configurations of "high-risk, high-precision" and "low-risk, low-precision". This ensures computational stability while minimizing computational overhead, memory bandwidth, and storage usage, thereby improving overall throughput.

[0052] In some embodiments, the statistical features include the weight norm index.

[0053] Furthermore, the step of determining the statistical characteristics of the multiplicative structure in each output channel of the feedforward network based on the first weight matrix and the second weight matrix includes: for each output channel, calculating the first norm value of the first weight matrix in the output channel and the second norm value of the second weight matrix in the output channel, and determining the weight norm index of the output channel according to the first norm value and the second norm value; wherein, the weight norm index is used to characterize the magnitude of the first weight matrix and the second weight matrix in the output channel.

[0054] Specifically, by performing statistical analysis on the two weight matrices in the multiplicative structure, quantitative features reflecting the potential numerical amplification risk of each output channel under low-precision computation are extracted. Since the output of the multiplicative structure is obtained by element-wise multiplication of the results of two linear transformations, its magnitude depends not only on the amplitude of the input activation but also on the amplitudes of the column vectors of the two weight matrices in the corresponding channels. Therefore, it is first necessary to characterize the amplitude of the weight vector in each channel, i.e., the weight norm index.

[0055] For each output channel of the feedforward network multiplicative structure, the weight parameters of the corresponding output channel can be extracted from the first and second weight matrices. The magnitude characteristics of the single-channel weights are quantified by calculating the norm value, thereby obtaining the weight norm index of each output channel. The specific form of the norm can be selected according to actual needs, including but not limited to the first norm, the second norm, etc. Preferably, since the second norm has a good mathematical correlation with the numerical range of subsequent matrix multiplications and is computationally stable, the magnitude characteristics of the single-channel weights can be quantified primarily using the second norm.

[0056] Furthermore, the first norm value and the second norm value can be multiplied to obtain the weight norm index of the corresponding output channel, which can be specifically expressed as: N i =||W 1,i || 2· ||W 2,i ||2; where N i ||W represents the weight norm index value of the i-th output channel. 1,i ||2 represents the first norm value of the first weight matrix W1 on output channel i, ||W 2,i ||2 represents the second norm value of the second weight matrix W2 on output channel i. The first and second norm values ​​can be calculated using different methods, and this disclosure does not limit their application.

[0057] The weight norm index, as a statistical feature of each output channel of the multiplicative structure, can accurately characterize the magnitude of the numerical values ​​of the two weight matrices in the corresponding output channels.

[0058] This embodiment only performs channel-level static statistical analysis on the first and second weight matrices of the multiplicative structure. The entire process does not require calling or scanning any input data, nor does it require any modification to the storage structure of the original weight matrix or the computational logic of the feedforward network. This completely avoids the computational overhead and engineering intrusion problems caused by relying on input data and modifying the model structure in related solutions. Moreover, all calculations are completed based on the inherent parameters of the weight matrix, and the statistical results are stable and will not change with the distribution of input data. This can provide objective and reliable weight-based basis for subsequent risk prediction.

[0059] Furthermore, the output of the multiplicative structure is affected not only by the magnitude of the column vectors of the two weight matrices in the corresponding channels, but also by the directional relationship between the column vectors of the two weight matrices in the corresponding channels.

[0060] In some embodiments, the statistical features also include weighted correlation metrics.

[0061] Furthermore, based on the first weight matrix and the second weight matrix, the statistical characteristics of the multiplicative structure in each output channel of the feedforward network are determined, which also includes: for each output channel, determining the weight correlation index of the output channel based on the correlation coefficient of the first weight matrix and the second weight matrix in the output channel; wherein the correlation coefficient is used to characterize the numerical sign consistency of the first weight matrix and the second weight matrix in the output channel.

[0062] Specifically, the weighted correlation index is used to measure the consistency of the numerical direction of two corresponding weight channels. The higher the correlation, the more consistent the weighting direction of the two channels, and the more significant the amplification effect of the multiplication.

[0063] In some embodiments, the weighted relevance index can be calculated using the absolute value of the normalized inner product or cosine similarity. The weighted relevance index can be expressed as: C i ||W represents the weight correlation index value of the i-th output channel. 1,i || 2· ||W 2,i ||2 represents the weight norm index of the i-th output channel, <W 1,i W 2,i > represents the inner product operation of two weighted channels on output channel i, and ε represents a parameter constant.

[0064] The weight norm index is used to measure the numerical intensity of numerical amplification; the weight correlation index measures the directional fit between two linear branches for numerical amplification. Based on both the weight norm index and the weight correlation index, this embodiment of the present disclosure can more accurately identify the numerical distribution characteristics of the two weight matrices on each output channel, thereby more precisely reflecting the potential risk of numerical amplification in low-precision calculations for each output channel.

[0065] In some embodiments, the statistical features include a weight norm index and / or a weight correlation index; the step of determining the numerical amplification risk index of the multiplicative structure in each output channel of the feedforward network based on the statistical features includes: for each output channel, determining a risk metric value of the output channel according to the weight norm index and / or the weight correlation index; and generating a numerical amplification risk index of the output channel according to the risk metric value and at least one preset threshold.

[0066] Specifically, for each output channel, a risk metric that comprehensively reflects the risk level of that output channel can first be determined based on the weight norm index and / or weight correlation index. This risk metric can be in scalar form, and its determination method can be based on the actual use of the weight norm index or the weight correlation index, or a combination of the two indices. The risk metric is positively correlated with both the weight norm index and the weight correlation index; that is, the larger the weight norm index, the higher the amplitude base of the two weight matrices, and the greater the risk of numerical amplification; the larger the weight correlation index, the stronger the directional consistency of the two weight matrices, the more significant the synergistic amplification effect after multiplication, and the greater the cumulative risk of numerical amplification. The fusion of the two indices can comprehensively reflect the overall numerical amplification risk of the output channel.

[0067] Secondly, to achieve a hierarchical classification of risk metrics, at least one preset threshold can be set as the dividing standard for risk level determination, based on the stability requirements of low-precision quantization in large language models and the need to utilize low-precision computing power. The number of preset thresholds can be flexibly set according to the actual precision configuration requirements. The setting of preset thresholds must balance quantization stability and low-precision utilization, avoiding situations where setting the preset threshold too high results in high-risk channels not being protected, or setting it too low results in conservative quantization and wasted low-precision capabilities.

[0068] Finally, for each output channel, the risk metric is compared with one or more preset thresholds. Based on the comparison results, a numerical amplification risk index for that output channel is generated. This numerical amplification risk index can also be represented by a level label or a quantified value, and it is positively correlated with the severity of the numerical amplification risk, thus bridging the gap between risk measurement and precision configuration.

[0069] This disclosure constructs a risk metric to comprehensively assess the risk level of each output channel under low-precision calculations. Then, it maps this metric to discrete or continuous risk indicators through threshold comparison, facilitating subsequent precision allocation operations. Furthermore, this disclosure flexibly selects and integrates weight norm indicators and / or weight correlation indicators to characterize numerical amplification risk from two core dimensions: amplitude basis and / or synergistic amplification effect. The judgment method can be flexibly adjusted according to the model quantization scenario, adapting to the needs of different low-precision types and different precision configuration levels, resulting in judgment results that better reflect the actual risk characteristics of the model.

[0070] In some embodiments, the risk metric includes a weight norm index value and a weight correlation index value; at least one preset threshold includes a first preset threshold and a second preset threshold; generating a numerical amplification risk index for the output channel based on the risk metric and at least one preset threshold includes: comparing the weight norm index value with the first preset threshold and comparing the weight correlation index value with the second preset threshold; generating a first mask value for the output channel when the weight norm index value is greater than the first preset threshold and the weight correlation index value is greater than the second preset threshold; generating a second mask value for the output channel when the weight norm index value is less than the first preset threshold or the weight correlation index value is less than the second preset threshold; wherein the first mask value or the second mask value is a risk index for the output channel.

[0071] Specifically, the weight norm index and weight correlation index can be used as risk measures. A preset threshold can be set for each index, and the risk of the output channel under that index is determined by comparing the index value with the corresponding preset threshold. Then, the numerical amplification risk index of the output channel is obtained based on the values ​​of the two indices. Since each index corresponds to a preset threshold, the comparison results include two cases: greater than or less than or equal to. Therefore, the corresponding comparison results can be represented using a mask.

[0072] If the weight norm index value is greater than a first preset threshold and the weight correlation index value is greater than a second preset threshold, it indicates that the numerical amplitude of the two weight matrices in the corresponding output channel is greater than the preset value, and the numerical directions of the two weight matrices in the corresponding output channel are consistent. This further indicates that the more consistent the weighting directions of the two channels are, the more significant the numerical amplification effect after multiplication will be. In this case, high-precision calculation can be used to ensure the stability of the calculation. Therefore, when the numerical amplification effect after multiplication is significant, a first mask value can be generated for this output channel to identify that the corresponding output channel can be calculated using high-precision calculation.

[0073] Conversely, if at least one of the weight norm index and the weight correlation index is less than the corresponding preset threshold, it indicates that the numerical amplitude of the two weight matrices in the corresponding output channel is less than the preset threshold, meaning the amplification effect after multiplication is not significant. In this case, low-precision calculation can be used to reduce computational and storage costs. Alternatively, if the numerical directions of the two weight matrices in the corresponding output channel are inconsistent, even if the numerical amplitudes of the two weight matrices in the corresponding output channel are large, the amplification effect after multiplication is still not significant due to the inconsistent directions. Low-precision calculation can also be used to reduce computational and storage costs. Therefore, if the amplification effect after multiplication is not significant, a second mask value can be generated for the output channel to indicate that low-precision calculation can be used for the corresponding output channel.

[0074] To improve the efficiency of determining numerical amplification indicators and reduce the number of preset thresholds, this disclosure can first determine a risk metric for the output channel based on the weight norm indicator and the weight correlation indicator, and then obtain the numerical amplification risk indicator through threshold comparison. A specific implementation method is given below.

[0075] In some embodiments, at least one preset threshold includes a third preset threshold; determining a risk metric for the output channel based on a weight norm index and / or a weight correlation index; generating a numerical amplification risk index for the output channel based on the risk metric and at least one preset threshold, including: multiplying the weight norm index value and the weight correlation index value to obtain the risk metric for the output channel; comparing the risk metric with the third preset threshold; generating a first mask value for the corresponding output channel if the risk metric is greater than the third preset threshold; generating a second mask value for the corresponding output channel if the risk metric is less than the third preset threshold; wherein the first mask value or the second mask value is the numerical amplification risk index value of the output channel.

[0076] Specifically, for each output channel i, the weight norm index value N is... i Correlation index value C with weight i Multiplying them together yields the risk metric S for that output channel. i The risk metric can be expressed as: S i =N i ×C i Among them, the risk metric S of the output channel i This means that the output magnitude of a multiplicative structure is approximately proportional to the product of the norms of the two weighted column vectors, while directional consistency determines whether this product can be effectively converted into actual numerical amplification. When the two vectors are in the same direction (i.e., C...), the output magnitude is proportional to the product of the norms of the two weighted column vectors. i When the result of element-wise multiplication is close to 1, the result of element-wise multiplication is close to the magnitude product, and the numerical amplification effect is significant; when the directions are inconsistent (i.e., C... i When the magnitude is small (or even negative), even if the magnitude is large, the output may decrease due to mutual cancellation. Therefore, the product N i ×C i It can simultaneously capture the synergistic effect of amplitude and direction on numerical amplification. The larger the risk metric value, the higher the probability of numerical overflow in low-precision calculations.

[0077] Furthermore, a third preset threshold can be pre-set based on factors such as empirical values, model size, and target quantization accuracy. Typically, once set, this third preset threshold remains unchanged in subsequent risk assessments. For each output channel, the risk metric is compared one-to-one with the third preset threshold. Based on the comparison result, a corresponding mask value is generated for each output channel; this mask value represents the numerical amplification risk index value for the output channel. Specifically, if the risk metric is less than the third preset threshold, it indicates that the numerical amplification is insignificant, and a first mask value can be generated to indicate that the numerical amplification of the corresponding output channel is insignificant. Conversely, if the risk metric is greater than the third preset threshold, it indicates that the numerical amplification is significant, and a second mask value can be generated to indicate that the numerical amplification of the corresponding output channel is significant.

[0078] In this embodiment, for each output channel, the product of the weight norm index value and the weight correlation index value is used as the risk metric value, and a binary numerical amplification risk index (i.e., risk mask) is generated by comparing it with a single threshold. This implementation is simple and efficient, and is particularly suitable for scenarios that require rapid identification of high-risk channels and allocation of two calculation precisions.

[0079] In some embodiments, the calculation precision includes a first precision level and a second precision level, and the calculation precision of the first precision level is higher than that of the second precision level; the step of configuring the corresponding calculation precision for each output channel according to the numerical amplification risk index includes: configuring the first precision level for the output channel corresponding to the numerical amplification risk index when the numerical amplification risk index is a first mask value; configuring the second precision level for the output channel corresponding to the numerical amplification risk index when the numerical amplification risk index is a second mask value.

[0080] Specifically, based on the comparison and determination results of the risk measurement value with the first preset threshold, the second preset threshold, or the third preset threshold in the above embodiments, a unique discrete mask value is assigned to each output channel. The high-risk channel corresponds to the first mask value, and the low-risk channel corresponds to the second mask value. The first mask value and the second mask value together serve as a concrete expression of the numerical amplification risk index and directly serve as the basis for calculating the accuracy of the output channel configuration.

[0081] Specifically, for output channels identified as having a high risk of numerical amplification, a first mask value is generated for them, which indicates that the channel needs to be configured with a high-precision calculation path; for output channels identified as having a low risk of numerical amplification, a second mask value is generated for them, which indicates that the channel can be configured with a low-precision calculation path.

[0082] The embodiments provided in this disclosure can quickly and accurately allocate appropriate calculation precision to each output channel based on the mask value corresponding to each output channel, thereby reducing calculation and storage overhead while ensuring calculation accuracy.

[0083] In some embodiments, the first precision level includes a 16-bit floating-point format, and the second precision level includes an 8-bit floating-point format.

[0084] Of course, more preset thresholds can be set, allowing for more precise calculations at various levels of accuracy. In this case, other forms of representation can be used to amplify the risk indicators, such as binary representation, and this disclosure does not limit this approach.

[0085] In some embodiments, the method includes not only steps 101-103 as described above, but also: according to a preset update cycle, re-acquire the updated first weight matrix and second weight matrix of the large language model; and based on the updated first weight matrix and second weight matrix, reconfigure the calculation precision for each output channel.

[0086] Specifically, this method is applied to scenarios involving the training of large language models. Each time the model is updated and iterated, the weight matrix of the multiplicative structure in the large language model is also updated. Since training large language models typically requires a large number of iterations, the weight matrix is ​​continuously updated with gradient descent, and its statistical characteristics (such as norm and correlation) evolve accordingly. If the precision configuration is determined and fixed only at the beginning of training, as training progresses, previously identified high-risk channels may no longer be high-risk, while previously low-risk channels may become high-risk due to weight changes, leading to a mismatch between the precision configuration and the actual situation, and consequently, numerical overflow or precision loss. However, since the update magnitude of the weight matrix is ​​small with each update, to improve computational efficiency, it is not necessary to re-obtain the corresponding first and second weight matrices for each iteration. Instead, the corresponding first and second weight matrices are obtained based on a certain period.

[0087] This disclosure introduces a dynamic update mechanism to enable the configuration of computational precision to adapt to the continuous changes in model weights during training, thereby ensuring the timeliness of numerical amplification risk prediction and the effectiveness of precision configuration. This disclosure re-executes precision configuration according to a preset update cycle, ensuring long-term stable operation of mixed precision training.

[0088] In some embodiments, the preset update cycle includes: every preset number of training iterations; or in response to a trigger event for saving a large language model; wherein the trigger event includes at least one of the following: reaching a preset save interval, receiving a save instruction, or meeting a preset performance indicator threshold.

[0089] Specifically, the preset period can be flexibly selected based on factors such as training strategy, hardware resources, and model size.

[0090] Every preset number of training iterations means that an update is triggered every preset number of training iterations. The number of steps can be determined based on update overhead and configuration timeliness. For example, a smaller number of steps can make the configuration more timely, but increases the computational overhead; conversely, a larger number of steps reduces the update frequency, but may cause configuration lag.

[0091] In some embodiments, the update cycle can also be dynamically adjusted based on the training phase.

[0092] Specifically, the weights change drastically in the early stages of training, potentially requiring more frequent updates; while in the later stages, the weights stabilize, allowing for a reduction in update frequency. Therefore, the update cycle can be dynamically adjusted based on the training progress. For example, updates could be performed every 50 steps for the first 10% of training steps, every 200 steps for the middle 40%, and every 500 steps for the last 50%. Optionally, the training stage can be determined by the ratio of the current iteration count to the total number of iterations; this disclosure does not impose any limitations on this.

[0093] Furthermore, updates can be based on events triggered by model saving. Specifically, during model training, checkpoints are typically saved at specific times, such as reaching a preset save interval, receiving a save command, or meeting a preset performance threshold; checkpoints can also be saved at the end of each epoch, or when validation set performance improves. This disclosure uses these save events as update triggers, recalculating the accuracy configuration while saving the model, and saving the configuration information along with the model weights. This allows the latest configuration to be used directly when resuming training or deploying inference later.

[0094] In some embodiments, adaptive updates can also be performed through risk change monitoring.

[0095] Specifically, by continuously monitoring the actual values ​​of each output channel to amplify the changes in risk indicators, an update is triggered when a certain proportion of output channels show a change in their risk level. This disclosed method is more intelligent.

[0096] This disclosure addresses the issue of numerical instability caused by low-precision quantization in large language models. The proposed hybrid precision quantization method is based on a technique that predicts the risk of numerical amplification based on the statistical characteristics of the weight channels. The core of this hybrid precision quantization method lies in its ability to predict the risk of numerical amplification in the output of multiplicative structures, such as SwishGLU, without relying on input activation data. This method directly characterizes the upper bound of numerical amplification in the output of multiplicative structures at the level of weight statistical characteristics, achieving pre-prediction of the risk of low-precision quantization and avoiding real-time detection of input activation.

[0097] In related technologies, quantization strategies often change with the distribution of input data. Different input samples may trigger different high-precision paths, leading to unstable system behavior. The method provided in this disclosure is based entirely on the model weights themselves, independent of specific input data, thus maintaining stable and consistent quantization behavior under different data distributions, different training stages, or different application scenarios. This technological approach breaks through the existing methods that rely on runtime outlier scanning or empirical processing, providing a predictable and data-independent solution for quantization risk assessment.

[0098] Furthermore, this disclosure employs a threshold-based channel-level mixed-precision quantization mechanism. The mixed-precision quantization method provided in this disclosure does not uniformly switch the precision for the entire operator or the entire network layer. Instead, it uses threshold determination to divide the weighted channels into high-risk channels and low-risk channels, or even more levels of risk channels, at the channel dimension, and performs calculations using different precisions such as BF16 and FP8 respectively. This channel-level mixed-precision quantization mechanism ensures numerical stability while maximizing the performance advantages of low-precision calculations.

[0099] Furthermore, the mixed-precision quantization method provided in this disclosure eliminates the need for runtime scanning or splitting of input activations during operation, thereby avoiding additional online computational overhead and dependence on input data distribution. This reduces the computational and scheduling overhead associated with mixed-precision control, making it more suitable for training and inference scenarios of large-scale language models. This feature significantly improves the scalability of the solution in large-scale training and inference scenarios.

[0100] Based on the same inventive concept as the above-described mixed precision quantization method, this disclosure also provides a mixed precision quantization device.

[0101] Figure 2 This is a structural block diagram of a hybrid precision quantization device provided in an embodiment of the present disclosure.

[0102] Further reference Figure 2 As an implementation of the methods shown in the above figures, this device embodiment is similar to... Figure 1 Corresponding to the method embodiment shown, the hybrid precision quantization device 200 of this embodiment can be applied to a large language model. The feedforward network of the modified large language model includes a multiplicative structure, which is configured with a first weight matrix and a second weight matrix.

[0103] like Figure 2 As shown, the mixed precision quantization device 200 may include: a first determining module 201, a second determining module 202, and a configuration module 203.

[0104] The first determining module 201 is used to determine the statistical characteristics of the multiplicative structure in the feedforward network of the large language model at each output channel of the feedforward network based on the first weight matrix and the second weight matrix; the multiplicative structure is configured with the first weight matrix and the second weight matrix; the second determining module 202 is used to determine the numerical amplification risk index of the multiplicative structure at each output channel of the feedforward network based on the statistical characteristics; the configuration module 203 is used to configure the corresponding calculation precision for each output channel according to the numerical amplification risk index. The output channels with different numerical amplification risk indices are configured with different calculation precisions.

[0105] In some embodiments, the statistical features include a weight norm index; the first determining module 201 is specifically used to: for each output channel, calculate the first norm value of the first weight matrix on the output channel and the second norm value of the second weight matrix on the output channel, and determine the weight norm index of the output channel based on the first norm value and the second norm value; wherein, the weight norm index is used to characterize the magnitude of the first weight matrix and the second weight matrix on the output channel.

[0106] In some embodiments, the statistical features further include a weighted correlation index; the first determining module 201 is further configured to: for each output channel, determine the weighted correlation index of the output channel based on the correlation coefficient of the first weight matrix and the second weight matrix on the output channel; wherein the correlation coefficient is used to characterize the consistency of the numerical signs of the first weight matrix and the second weight matrix on the output channel.

[0107] In some embodiments, the statistical features include a weight norm index and / or a weight correlation index; the second determining module 202 is specifically used to: for each output channel, determine a risk metric value for the output channel based on the weight norm index and / or the weight correlation index; and generate a numerical amplification risk index for the output channel based on the risk metric value and at least one preset threshold.

[0108] In some embodiments, the risk metric includes a weight norm index value and a weight correlation index value; at least one preset threshold includes a first preset threshold and a second preset threshold; the second determining module 202 is further configured to: compare the weight norm index value with the first preset threshold and compare the weight correlation index value with the second preset threshold; generate a first mask value for the output channel when the weight norm index value is greater than the first preset threshold and the weight correlation index value is greater than the second preset threshold; generate a second mask value for the output channel when the weight norm index value is less than the first preset threshold or the weight correlation index value is less than the second preset threshold; wherein the first mask value or the second mask value is a risk indicator of the output channel.

[0109] In some embodiments, at least one preset threshold includes a third preset threshold; the second determining module 202 is further configured to: perform a multiplication operation on the weight norm index value and the weight correlation index value to obtain the risk measurement value of the output channel; compare the risk measurement value with the third preset threshold; generate a first mask value for the corresponding output channel if the risk value is greater than the third preset threshold; and generate a second mask value for the corresponding output channel if the risk value is less than the third preset threshold; wherein the first mask value or the second mask value is the risk index value of the output channel.

[0110] In some embodiments, the calculation precision includes a first precision level and a second precision level, and the calculation precision of the first precision level is higher than that of the second precision level; the configuration module 203 is specifically used to: configure a first precision level for the output channel corresponding to the numerical amplification risk index when the numerical amplification risk index is a first mask value; and configure a second precision level for the output channel corresponding to the numerical amplification risk index when the numerical amplification risk index is a second mask value.

[0111] In some embodiments, the first precision level is a 16-bit floating-point format, and the second precision level is an 8-bit floating-point format.

[0112] In some embodiments, the mixed-precision quantization device 200 includes not only the first determining module 201, the second determining module 202, and the configuration module 203, but also an updating module. The updating module is specifically used to: re-acquire the updated first weight matrix and second weight matrix of the large language model according to a preset update cycle; and, based on the updated first weight matrix and second weight matrix, reconfigure the calculation precision for each output channel.

[0113] In some embodiments, the preset update cycle includes: every preset number of training iterations; or in response to a trigger event for saving a large language model; wherein the trigger event includes at least one of the following: reaching a preset save interval, receiving a save instruction, or meeting a preset performance indicator threshold.

[0114] In some embodiments, the mixed-precision quantization device 200 includes not only a first determining module 201, a second determining module 202, and a configuration module 203, but also a calculation module. The calculation module is used to: perform feedforward network calculations on the corresponding output channels based on the calculation precision configured for each output channel.

[0115] The specific implementation details and technical effects of the hybrid precision quantization device embodiments provided in this disclosure are the same as the implementation details and technical effects of the hybrid precision quantization method embodiments described above, and will not be repeated here.

[0116] This embodiment is a device embodiment corresponding to the method embodiment described above. The mixed-precision quantization device provided in this embodiment has a first determining module that determines the statistical characteristics of each output channel based on a first weight matrix and a second weight matrix. A second determining module can then determine the numerical amplification risk index for each output channel based on these statistical characteristics. This allows for the prediction in advance of whether low-precision quantization of each output channel poses a risk based on the numerical amplification risk index. The configuration module can assign an appropriate computational precision to each channel based on the determined numerical amplification risk index, thereby dynamically selecting high or low precision for computation and implementing a differentiated mixed-precision computation strategy. The device provided in this disclosure avoids the problems of numerical overflow and precision degradation under low-precision quantization while maximizing the advantages of low-precision computation in reducing storage overhead and improving computational efficiency.

[0117] The mixed-precision quantization device disclosed herein can predict the upper bound of the output value without relying on specific input data by analyzing the statistical characteristics of the weight channels. Furthermore, the device can periodically acquire the weight matrix of multiplicative structures such as SwishGLU during training or inference; perform statistical analysis on the weight channels to calculate channel-level norm and channel-level correlation indices; and determine whether the current output channel has a risk of low-precision quantization based on these indices; if a risk exists, high-precision calculation is used, and low-precision calculation is resumed when the risk decreases.

[0118] The mixed-precision quantization device provided in this disclosure does not require real-time scanning of input activation, significantly reducing runtime overhead; and the device relies only on weight statistical features when performing quantization calculations, avoiding dependence on the distribution of input data; the device can predict the risk of numerical amplification in advance, making it suitable for large-scale training scenarios.

[0119] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the mixed precision quantization method described in any of the above embodiments when executed.

[0120] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 3 As shown, the electronic device 300 of this embodiment includes a processor 301 and a memory 302; wherein, the memory 302 is used to store computer execution instructions; the processor 301 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 300 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0121] In some embodiments, the memory 302 can be either standalone or integrated with the processor 301.

[0122] When the memory 302 is set up independently, the electronic device also includes a bus 303 for connecting the memory 302 and the processor 301.

[0123] It should be understood that the processor 301 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0124] The memory 302 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0125] Bus 303 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0126] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the mixed precision quantization method in any of the above method embodiments.

[0127] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the mixed precision quantization method according to any of the above embodiments.

[0128] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0129] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0130] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The aforementioned modular unit can be implemented in hardware or in a combination of hardware and software functional units.

[0131] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0132] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0133] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0134] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0135] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A hybrid precision quantization method, characterized in that, Applied to a large language model, the feedforward network of the large language model includes a multiplicative structure, the multiplicative structure being configured with a first weight matrix and a second weight matrix, the method includes: Based on the first weight matrix and the second weight matrix, the statistical characteristics of the multiplicative structure in each output channel of the feedforward network are determined. Based on the statistical characteristics, the numerical amplification risk index of the multiplicative structure in each output channel is determined; Based on the numerical amplification risk index, a corresponding calculation precision is configured for each of the output channels; wherein, the calculation precision is configured differently for output channels with different numerical amplification risk indices.

2. The method according to claim 1, characterized in that, The statistical features include the weight norm index; The step of determining the statistical characteristics of the multiplicative structure on each output channel of the feedforward network based on the first weight matrix and the second weight matrix includes: For each output channel, calculate the first norm value of the first weight matrix on the output channel and the second norm value of the second weight matrix on the output channel, and determine the weight norm index of the output channel based on the first norm value and the second norm value. The weight norm index is used to characterize the magnitude of the first weight matrix and the second weight matrix on the output channel.

3. The method according to claim 1 or 2, characterized in that, The statistical features include weighted correlation indicators; The step of determining the statistical characteristics of the multiplicative structure on each output channel of the feedforward network based on the first weight matrix and the second weight matrix includes: For each output channel, a weighted correlation index is determined based on the correlation coefficients of the first weight matrix and the second weight matrix on the output channel. The correlation coefficient is used to characterize the consistency of the numerical signs of the first weight matrix and the second weight matrix on the output channel.

4. The method according to claim 1, characterized in that, The statistical features include weight norm indicators and / or weight correlation indicators; The step of determining the numerical amplification risk index of the multiplicative structure on each output channel based on the statistical characteristics includes: For each output channel, a risk metric is determined based on the weight norm index and / or the weight correlation index; and a numerical amplification risk index is generated for the output channel based on the risk metric and at least one preset threshold.

5. The method according to claim 4, characterized in that, The at least one preset threshold includes a third preset threshold; The risk metric of the output channel is determined based on the weight norm index and / or the weight correlation index. Based on the risk metric and at least one preset threshold, a numerical amplification risk index is generated for the output channel, including: The risk metric of the output channel is obtained by multiplying the weight norm index value and the weight correlation index value. The risk metric is compared with the third preset threshold. If the risk metric value is greater than a third preset threshold, a first mask value is generated for the output channel; If the risk metric value is less than a third preset threshold, a second mask value is generated for the output channel; Wherein, the first mask value or the second mask value is a numerical amplification risk indicator of the output channel.

6. The method according to claim 4, characterized in that, The risk metric includes a weighted norm index and a weighted correlation index; the at least one preset threshold includes a first preset threshold and a second preset threshold. The step of generating a numerical amplification risk index for the output channel based on the risk metric and at least one preset threshold includes: The weight norm index value is compared with a first preset threshold, and the weight correlation index value is compared with a second preset threshold; If the weight norm index value is greater than the first preset threshold and the weight correlation index value is greater than the second preset threshold, a first mask value is generated for the output channel. If the weight norm index value is less than the first preset threshold, or the weight correlation index value is less than the second preset threshold, a second mask value is generated for the output channel. Wherein, the first mask value or the second mask value is a numerical amplification risk indicator of the output channel.

7. The method according to claim 5 or 6, characterized in that, The calculation accuracy includes a first accuracy level and a second accuracy level, and the calculation accuracy of the first accuracy level is higher than that of the second accuracy level. The step of configuring corresponding calculation precision for each output channel based on the numerical amplification risk index includes: When the numerical amplification risk index is the first mask value, the first precision level is configured for the output channel corresponding to the numerical amplification risk index. When the numerical amplification risk index is the second mask value, configure the second precision level for the output channel corresponding to the numerical amplification risk index.

8. The method according to claim 1, characterized in that, The method further includes: According to the preset update cycle, the updated first weight matrix and second weight matrix of the large language model are re-acquired; and based on the updated first weight matrix and second weight matrix, the calculation precision of each output channel is reconfigured.

9. The method according to claim 8, characterized in that, The preset update cycle includes: Every preset number of training iterations; or The system responds to a trigger event for saving the large language model; wherein the trigger event includes at least one of the following: reaching a preset saving interval, receiving a saving instruction, or meeting a preset performance indicator threshold.

10. The method according to claim 1, characterized in that, The method further includes: The calculation of the feedforward network is performed on the corresponding output channel based on the calculation precision configured for each output channel.

11. A hybrid precision quantization device, characterized in that, The device includes: The first determining module is used to determine the statistical characteristics of the multiplicative structure in the feedforward network of a large language model on each output channel of the feedforward network based on the first weight matrix and the second weight matrix, wherein the multiplicative structure is configured with the first weight matrix and the second weight matrix. The second determining module is used to determine the numerical amplification risk index of the multiplicative structure on each output channel of the feedforward network based on the statistical characteristics. The configuration module is used to configure the corresponding calculation precision for each of the output channels according to the numerical amplification risk index; wherein the output channels with different numerical amplification risk indices are configured with different calculation precisions.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the mixed-precision quantization method according to any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the mixed precision quantization method according to any one of claims 1-10.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the mixed-precision quantization method according to any one of claims 1-10.

Images