Smooth quantization activation value transfer method, system, device and medium for trillion-parameter multi-modal large model

Abstract

The application discloses a smooth quantization activation value transfer method, system, device and medium for a trillion parameter multimodal large model, belongs to the technical field of model optimization, and comprises the following steps: collecting activation value distribution statistical information of each layer and each channel of a multimodal large model, determining abnormal value amplitude indexes of each channel, modality type information of each layer and channel condition parameters of each channel; constructing a candidate quantization scheme set of each layer; performing multiple selection knapsack solving on the candidate quantization scheme set of each layer to determine a quantization resource configuration scheme of each layer; determining a smoothing factor of each channel by using a capacity-adjustable water injection distribution processing and performing smoothing transformation; performing quantization processing to determine a compensation matrix, performing compensation processing on the quantized weight parameters, combining weight quantization errors and activation value quantization errors, and obtaining a quantized multimodal large model. The application significantly reduces quantization errors in a multimodal scene and improves the performance of the model in complex tasks.

Description

Technical Field

[0001] This invention relates to the field of model optimization technology, specifically to a method, system, device, and medium for smoothing quantization activation value transfer for multimodal large models with trillions of parameters. Background Technology

[0002] In existing technologies, post-training quantization methods for large models typically employ a uniform quantization strategy to process each layer and channel of the model. This approach fails to adequately consider the differences in activation value distribution across different modalities within a large multimodal model. Especially in scenarios where visual and text modalities coexist, the degree of activation value anomalies varies significantly across modalities, easily leading to substantial accuracy loss during quantization. Furthermore, existing technologies lack adaptive resource allocation mechanisms for different layers and channels when dealing with large-scale multimodal models. This makes it difficult to dynamically adjust quantization accuracy and smoothing intensity based on the degree of activation value anomalies, resulting in insufficient resource utilization and negatively impacting quantization performance.

[0003] Furthermore, existing technologies are primarily designed for quantization processing of single-modal models. In multimodal scenarios, there is a lack of a collaborative optimization mechanism between weights and activation values, making it difficult to balance quantization accuracy and efficiency for large multimodal models under resource constraints. Therefore, this invention aims to solve the core technical challenge faced by large multimodal models in the post-training quantization process: how to achieve efficient transfer and adaptive allocation of activation value anomalies for large multimodal models with trillions of parameters under resource-constrained conditions (limited GPU memory and computing power), thereby maximizing quantization efficiency and system throughput while maintaining model inference accuracy. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides a method, system, device and medium for smooth quantization activation value transfer for large multimodal models with trillions of parameters.

[0005] Therefore, the technical problem solved by this invention is: how to achieve efficient transfer and adaptive allocation of activation value anomalies for multimodal large models with trillions of parameters under resource-constrained conditions, so as to maximize quantization efficiency and system throughput while maintaining model inference accuracy.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a smooth quantization activation value transfer method for a multimodal large model with trillions of parameters, comprising: collecting statistical information on the activation value distribution of each channel in each layer of the multimodal large model; determining the outlier magnitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel; constructing a candidate quantization scheme set for each layer based on the modality type information of each layer and the outlier magnitude index of each channel; performing a multiple choice knapsack problem on the candidate quantization scheme set of each layer under memory budget constraints to determine the quantization resource allocation scheme for each layer; and based on the channel... The channel condition parameters and the quantization resource configuration schemes of each layer are used to determine the smoothing factor of each channel through capacity-adjustable water-filling allocation processing. Smoothing transformation is performed on the weight parameters and activation values ​​to obtain the transformed weight parameters and activation values. Quantization processing is then performed on the transformed weight parameters and activation values, followed by low-rank decomposition processing to determine the compensation matrix. Subsequently, compensation processing is performed on the quantized weight parameters based on the compensation matrix. Based on the compensated weight parameters and quantized activation values, and combining the weight quantization error and activation value quantization error, the quantized multimodal large model is obtained.

[0007] As a preferred embodiment of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters described in this invention, the step of collecting activation value distribution statistics of each layer and channel of the multimodal large model, and determining the outlier magnitude index, modality type information of each layer, and channel condition parameters of each channel includes: traversing each layer of the multimodal large model based on a calibration dataset and collecting activation value distribution statistics of each layer and channel; determining the outlier magnitude index characterizing the degree of activation value anomalies based on the activation value distribution statistics of each channel; classifying each layer's channels according to their modality affiliation to obtain modality type information of each layer; and mapping each channel to channel condition parameters according to the outlier magnitude index.

[0008] As a preferred embodiment of the smooth quantization activation value transfer method for multimodal large models with trillion parameters described in this invention, the step of constructing a candidate quantization scheme set for each layer based on the modality type information of each layer and the outlier magnitude index of each channel includes: generating candidate quantization schemes for each layer according to the modality type information of each layer; screening the candidate quantization schemes for each layer according to the outlier magnitude index of each channel; determining the accuracy loss information and resource overhead information of the screened candidate quantization schemes; and constructing the candidate quantization scheme set for each layer according to the accuracy loss information and the resource overhead information.

[0009] This preferred scheme constructs a set of candidate quantization schemes by combining modality type information and anomaly characterization information, thereby enabling the formation process of candidate quantization schemes to simultaneously possess modality difference adaptation capability and anomaly degree differentiation capability.

[0010] As a preferred embodiment of the smooth quantization activation value transfer method for multimodal large models with trillion parameters described in this invention, the step of performing a multiple choice knapsack problem on the candidate quantization scheme set of each layer under memory budget constraints to determine the quantization resource allocation scheme of each layer includes: transforming the candidate quantization scheme set of each layer into a multiple choice knapsack problem; using the resource cost information as constraint information and the accuracy loss information as optimization basis; solving the multiple choice knapsack problem based on dynamic programming algorithm according to the constraint information and optimization basis to determine the target candidate quantization scheme of each layer; and determining the quantization resource allocation scheme of each layer according to the target candidate quantization scheme of each layer.

[0011] This preferred solution transforms the set of candidate quantization schemes into a multiple choice knapsack problem, and solves it under resource overhead constraints, using accuracy loss as the optimization basis. This makes the determination process of quantization resource allocation schemes at each level solvable and comparable under global budget constraints.

[0012] As a preferred embodiment of the smoothing quantization activation value transfer method for a multimodal large model with trillion parameters described in this invention, the method involves: determining the smoothing factor of each channel based on the channel condition parameters of each channel and the quantization resource configuration scheme of each layer, using capacity-adjustable water-filling allocation processing; performing a smoothing transformation on the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values. This includes: determining the quantization bit configuration of each layer according to the quantization resource configuration scheme of each layer; determining the smoothing resource budget of each layer according to the quantization bit configuration, memory budget constraints, and channel parameter scale; determining the resource constraints of the capacity-adjustable water-filling allocation processing according to the smoothing resource budget of each layer; determining the smoothing factor of each channel based on the channel condition parameters of each channel under resource constraints using capacity-adjustable water-filling allocation processing; and performing a smoothing transformation on the weight parameters and activation values ​​according to the smoothing factors of each channel, transferring the activation values ​​between the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values.

[0013] This preferred scheme determines the smoothing resource budget based on quantization bit configuration, memory budget constraints, and channel parameter scale, and performs adjustable capacity allocation by combining channel anomaly characterization information. This enables the smoothing factor determination process to have differentiated allocation capabilities for abnormal channels, facilitating the abnormal transfer of activation values ​​during the smoothing transformation stage.

[0014] As a preferred embodiment of the smoothing quantization activation value transfer method for multimodal large models with trillion parameters described in this invention, the following steps are taken: Based on the channel condition parameters of each channel, the smoothing factor of each channel is determined using capacity-adjustable water-filling allocation under resource constraints. This includes: constructing a smoothing factor optimization problem with the objective of minimizing the total quantization error of the layer and with the constraint that the total smoothing resources do not exceed the resource constraints; constructing a Lagrange function based on the smoothing factor optimization problem; solving for the smoothing factor of each channel according to the Karouch-Kun-Tucker conditions to determine the Lagrange multipliers; and determining the smoothing factor of each channel based on the Lagrange multipliers and the channel condition parameters of each channel.

[0015] This preferred solution transforms anomaly information into equivalent condition information and allocates smoothing resources accordingly under resource constraints. This transforms channel anomaly differences into a representation that can participate in resource allocation calculations, making it easier to determine the smoothing factor based on channel requirements.

[0016] As a preferred embodiment of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters described in this invention, the method involves: performing quantization processing on the transformed weight parameters and transformed activation values; performing low-rank decomposition processing on the transformed weight parameters to determine a compensation matrix; then performing compensation processing on the quantized weight parameters based on the compensation matrix; and obtaining the quantized multimodal large model based on the compensated weight parameters and quantized activation values, combined with the weight quantization error and the activation value quantization error. This includes: performing quantization processing on the transformed weight parameters and transformed activation values ​​to obtain quantized weight parameters and quantized activation values; performing low-rank decomposition processing on the transformed weight parameters to determine a compensation matrix; combining the compensation matrix with the quantized weight parameters to obtain compensated weight parameters; determining the weight quantization error and the activation value quantization error based on the compensated weight parameters and quantized activation values; and obtaining the quantized multimodal large model based on the compensated weight parameters and the quantized activation values, combined with the weight quantization error and the activation value quantization error.

[0017] This preferred scheme determines the compensation matrix by combining low-rank compensation parameters after quantization, and determines the quantization error based on the compensated weight parameters and quantized activation values, so that the quantized multimodal large model has a joint constraint basis of quantization results and compensation results.

[0018] This invention provides a smooth quantization activation value transfer system for large multimodal models with trillions of parameters.

[0019] To address the aforementioned technical problems, this invention provides the following technical solution: a smooth quantization activation value transfer system for a multimodal large model with trillions of parameters, comprising: a statistical determination module, a scheme construction module, a configuration solving module, a smooth transformation module, and a quantization compensation module; the statistical determination module is used to collect the activation value distribution statistics of each layer and channel of the multimodal large model, and determine the outlier magnitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel; the scheme construction module is used to construct a set of candidate quantization schemes for each layer based on the modality type information of each layer and the outlier magnitude index of each channel; the configuration solving module is used to perform a multiple selection knapsack problem on the set of candidate quantization schemes for each layer under memory budget constraints. The solution involves determining the quantization resource allocation scheme for each layer. The smoothing transformation module, based on the channel condition parameters of each channel and the quantization resource allocation scheme for each layer, uses capacity-adjustable water-filling allocation processing to determine the smoothing factor for each channel, performs smoothing transformation on the weight parameters and activation values, and obtains the transformed weight parameters and transformed activation values. The quantization compensation module performs quantization processing on the transformed weight parameters and transformed activation values, performs low-rank decomposition processing on the transformed weight parameters to determine the compensation matrix, and then performs compensation processing on the quantized weight parameters according to the compensation matrix. Based on the compensated weight parameters and quantized activation values, combined with the weight quantization error and the activation value quantization error, the quantized multimodal large model is obtained.

[0020] The present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters.

[0021] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters.

[0022] The beneficial effects of this invention are as follows: This invention introduces a modality-aware smoothing mechanism, which can dynamically adjust the smoothing factor and quantization accuracy based on the activation value distribution characteristics of different modalities. Compared with the uniform processing method of existing technologies, this invention can significantly reduce quantization errors in multimodal scenarios and improve the performance of models in complex tasks.

[0023] By employing a capacity-adjustable water injection method, this invention treats the number of quantization bits or smoothing intensity as allocable resources and dynamically allocates them based on the degree of anomaly in the activation values ​​of each channel. Channels with larger anomalies are injected with more smoothing resources or allocated higher quantization bits, achieving optimal resource utilization and obtaining higher model accuracy under the same hardware conditions.

[0024] By establishing a joint optimization objective function for weight quantization error and activation value quantization error, and introducing a cross-modal compensation mechanism, this invention can find the theoretically optimal smooth transition point while ensuring inference latency constraints. Compared with existing technologies, it can achieve higher throughput and lower memory usage on trillion-parameter models, providing a feasible solution for the practical deployment of large-scale multimodal models. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The above is a flowchart of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters provided in one embodiment of the present invention.

[0027] Figure 2 This is a comparison chart of the overall performance of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters provided in an embodiment of the present invention, and traditional methods. Detailed Implementation

[0028] To make the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0029] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a smooth quantization activation value transfer method for a multimodal large model with trillions of parameters, including: S1. Collect the activation value distribution statistics of each layer and channel of the multimodal large model, and determine the outlier amplitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel.

[0030] S2. Based on the modal type information of each layer and the outlier magnitude index of each channel, construct a set of candidate quantization schemes for each layer.

[0031] S3. Under the constraint of memory budget, perform a multi-choice knapsack problem on the candidate quantization scheme set of each layer to determine the quantization resource allocation scheme of each layer.

[0032] S4. Based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer, the smoothing factor of each channel is determined by the capacity adjustable water injection allocation process. The weight parameters and activation values ​​are then subjected to smoothing transformation to obtain the transformed weight parameters and transformed activation values.

[0033] S5. Perform quantization processing on the transformed weight parameters and transformed activation values, perform low-rank decomposition processing on the transformed weight parameters, determine the compensation matrix, and then perform compensation processing on the quantized weight parameters according to the compensation matrix. Based on the compensated weight parameters and quantized activation values, and combining the weight quantization error and activation value quantization error, obtain the quantized multimodal large model.

[0034] It should be noted that multimodal large models exhibit differences in activation value distribution across different layers, channels, and modes. Under limited memory budgets, directly applying a uniform quantization method to process the model can easily lead to a lack of compatibility between quantization resource allocation and the degree of activation value anomalies, thus affecting the quantized model processing results. Therefore, this invention first determines anomaly representation information and mode type information based on the activation value statistics of each layer and channel of the multimodal large model. Then, by combining a set of candidate quantization schemes, quantization resource allocation schemes, smoothing factors, and low-rank compensation parameters, it connects resource allocation processing, smoothing transformation processing, quantization processing, and compensation processing during the model quantization process.

[0035] Multimodal large models refer to multimodal large language models that use Transformer as the main architecture and process both visual features and text tokens.

[0036] This invention introduces resource optimization algorithms from the field of communications into large-scale model quantization scenarios. By establishing an adaptive resource allocation framework that detects activation value anomalies, it achieves joint optimization of smoothing factor, quantization bit count, and low-rank compensation budget.

[0037] Example 2, an embodiment of the present invention, provides a smooth quantization activation value transfer method for a multimodal large model with trillion parameters, based on the previous embodiment, including: Furthermore, in step S1, the activation value distribution statistics of each channel in each layer of the multimodal large model are collected to determine the outlier magnitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel, including the following steps A1-A4: A1. Based on the calibration dataset, traverse each layer of the multimodal large model and collect the statistical information of the activation value distribution of each channel in each layer.

[0038] A2. Based on the statistical information of the activation value distribution of each channel, determine the abnormal value amplitude index that characterizes the degree of abnormality of activation values.

[0039] A3. Based on the modal affiliation of each layer channel, classify each layer channel to obtain the modal type information of each layer.

[0040] Anomaly detection and channel classification for multimodal large models First, collect the activation value distribution statistics of all channels in the layer. Assume this layer has a total of... The first channel, the... The activation value of each channel is denoted as Calculate the outlier magnitude index for each channel: in, Indicates the first The ratio of outlier magnitudes in each channel; For the first The maximum value among the absolute activation values ​​of each channel represents the maximum activation amplitude that the channel has ever experienced. For the first The average absolute activation value of each channel represents the average level of the overall activation amplitude of that channel. The visual modality channel set is categorized according to modality type. Text modal channel set Other modal channel sets .

[0041] A4. Based on the outlier magnitude index, map each channel to channel condition parameters.

[0042] Modality-aware channel condition mapping maps the outlier magnitudes for each channel. Mapped to equivalent channel condition parameters Drawing on the concept of channel gain in communication systems, we define: in, This is a regularization constant to prevent division by zero errors. This mapping indicates that the larger the outlier amplitude of a channel, the worse its equivalent channel conditions, requiring more smoothing resources.

[0043] Furthermore, in step S2, based on the modal type information of each layer and the outlier magnitude index of each channel, a set of candidate quantization schemes for each layer is constructed, including the following steps B1-B4: B1. Generate candidate quantization schemes for each layer based on the modality type information of each layer.

[0044] B2. Based on the outlier magnitude index of each channel, candidate quantization schemes for each layer are screened.

[0045] B3. Determine the accuracy loss information and resource overhead information for the selected candidate quantification schemes.

[0046] B4. Based on the accuracy loss information and resource overhead information, construct a set of candidate quantization schemes for each layer.

[0047] For including The layer adopts a multimodal large model with Transformer architecture, providing each layer with... Define a set of candidate quantization schemes Each candidate quantization scheme It includes three key parameters: quantization bit configuration, low-rank compensation matrix rank, and video memory usage cost.

[0048] Quantization bit configuration The number of quantization bits for weights and activation values; the rank of the low-rank compensation matrix. Cross-modal compensation for whitening based on Singular Value Decomposition (SVD); memory footprint cost This indicates that the scheme is in the... The memory overhead of the layer.

[0049] For each candidate quantization scheme The accuracy loss introduced by the calibration dataset is evaluated. The accuracy loss consists of three parts: in, For the first Layer The accuracy loss of each candidate quantization scheme, For quantization error, To smooth out errors, To compensate for low-rank residuals, These are weighting coefficients, which are adaptively adjusted according to the modality type.

[0050] Furthermore, in step S3, under the constraint of memory budget, a multi-choice knapsack problem is performed on the candidate quantization scheme set for each layer to determine the quantization resource allocation scheme for each layer, including the following steps C1-C4: C1. Transform the set of candidate quantization schemes for each layer into a multiple choice knapsack problem.

[0051] C2. Use resource overhead information as constraint information and accuracy loss information as optimization basis.

[0052] C3. Based on the constraint information and optimization criteria, solve the multiple choice knapsack problem using dynamic programming algorithm to determine the target candidate quantization schemes for each layer.

[0053] C4. Based on the target candidate quantification schemes for each layer, determine the quantification resource allocation scheme for each layer.

[0054] The hierarchical resource allocation problem is formalized as a multiple-choice knapsack problem (MCKP). Let the total video memory budget be... The goal is to select a quantization scheme for each layer while not exceeding the memory constraints, so as to minimize the total precision loss. in, For the first The number of candidate quantization schemes for the layer For the first Index of candidate quantization schemes for the layer Let be the decision variable, used to represent the _th Should the first layer be selected? One candidate quantization scheme, Indicates the first Layer selection scheme .

[0055] Constraints: in, For the first The layer adopts the first The memory overhead required for each candidate quantization scheme.

[0056] The MCKP problem is solved using dynamic programming. The state transition equation is defined, and... Indicates the preceding The layer has a video memory budget of The minimum precision loss at that time is: Initial conditions, , for By tracing back the path, the optimal quantization scheme for each layer can be obtained.

[0057] Algorithm complexity analysis: Time complexity is O(n log n). ,in This represents the average number of candidate solutions. For a trillion-parameter model, pruning strategies can control the number of candidate solutions within a reasonable range.

[0058] Furthermore, in step S4, based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer, the smoothing factor of each channel is determined using Volume Adjustable Backhaul-constrained Water-Filling (VABWF) water-filling allocation processing. A smoothing transformation is then performed on the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values, including the following steps D1-D5: D1. Determine the quantization bit configuration for each layer based on the quantization resource configuration scheme for each layer.

[0059] D2. Determine the smoothing resource budget for each layer based on the quantization bit configuration, memory budget constraints, and channel parameter scale.

[0060] D3. Based on the smoothed resource budget of each layer, determine the resource constraints for capacity-adjustable water injection allocation and treatment.

[0061] D4. Based on the channel condition parameters of each channel, the smoothing factor of each channel is determined by capacity-adjustable water injection allocation under resource constraints.

[0062] D5. Based on the smoothing factor of each channel, perform a smoothing transformation on the weight parameters and activation values, and transfer the abnormal activation values ​​between the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values.

[0063] Furthermore, in step D4, based on the channel condition parameters of each channel, the smoothing factor of each channel is determined using capacity-adjustable water injection allocation processing under resource constraints, including the following steps D41-D45: D41. With the goal of minimizing the total amount of error in this layer and with the constraint that the total amount of smoothed resources does not exceed the resource constraint, construct a smoothing factor optimization problem.

[0064] D42. Construct the Lagrangian function based on the smoothing factor optimization problem.

[0065] D43. Based on the Karouch-Kun-Tucker conditions, solve for the smoothing factor of each channel to determine the Lagrange multipliers.

[0066] D44. Determine the smoothing factor for each channel based on the Lagrange multipliers and the channel condition parameters for each channel.

[0067] Setting the first Total budget for smooth resources available at each layer This budget is constrained by video memory capacity and computational overhead. The relationship between the total smoothing resources and the number of quantization bits is as follows: in, To smooth out the total amount of resources; This is the upper limit of the video memory budget for this layer; This is the power exponent / smoothing intensity coefficient, used to control the impact of the channel smoothing factor on the memory budget and quantization error transfer intensity; For the first Number of quantization bits per channel; This represents the number of parameters for this channel.

[0068] The optimization problem is constructed using the Lagrange multiplier method. The objective function is to minimize the total scaling error of this layer, with the constraint that the total amount of smoothed resources does not exceed the budget. The optimization problem is formulated as follows: in, Let be the set of candidate smoothing factors to be solved for all channels. For the first Candidate smoothing factors to be solved for each channel For weight quantization error, This represents the quantization error of the activation value. Based on the principle of SmoothQuantization, the smoothing factor... Control the degree to which abnormal activation values ​​shift to weights. The optimal smoothing factor, obtained through KKT conditions and satisfying resource constraints, is denoted as... .

[0069] Constructing the Lagrange function: in, This is the Lagrangian function, used to optimize the smoothing factor of each channel under smoothing resource constraints; These are Lagrange multipliers used to incorporate smooth resource constraints into the optimization objective.

[0070] According to the Karush-Kuhn-Tucker (KKT) conditions, for Taking the partial derivative and setting it to zero, we obtain the closed-form solution for the optimal smoothing factor: in, For the first The optimal smoothing factor for each channel. , These are variables in the positive part of the function. Lagrange multipliers. The smooth resource constraint is satisfied by solving the problem using an iterative method: in, It represents the error scale normalization constant, used to convert the calibration set quantization error to the same dimension as the smoothing resource budget; This indicates the upper limit of the smoothed resource budget.

[0071] This formula embodies the core idea of ​​the water-filling algorithm, which is suitable for poor channel conditions. Channels with smaller outlier amplitudes are assigned a larger smoothing factor. This enables adaptive transfer of abnormal activation values.

[0072] Furthermore, in step S5, quantization processing is performed on the transformed weight parameters and transformed activation values. Low-rank decomposition processing is performed on the transformed weight parameters to determine the compensation matrix. Then, compensation processing is performed on the quantized weight parameters according to the compensation matrix. Based on the compensated weight parameters and quantized activation values, combined with the weight quantization error and activation value quantization error, the quantized multimodal large model is obtained, including the following steps E1-E5: E1. Perform quantization processing on the transformed weight parameters and transformed activation values ​​to obtain quantized weight parameters and quantized activation values.

[0073] E2. Perform low-rank decomposition on the transformed weight parameters to determine the compensation matrix.

[0074] E3. Combine the compensation matrix with the quantized weight parameters to obtain the compensated weight parameters.

[0075] E4. Based on the compensated weight parameters and the quantized activation values, determine the weight quantization error and the activation value quantization error.

[0076] E5. Based on the compensated weight parameters and quantized activation values, and combining the weight quantization error and activation value quantization error, the quantized multimodal large model is obtained.

[0077] A joint optimization framework for weight quantization error and activation value quantization error is established. By introducing modal weight coefficients and cross-modal compensation mechanisms, the theoretically optimal smooth transition point is solved.

[0078] For the Linear transformation of layers ,in, For the first The output matrix of the layer linear transformation, This is the weight matrix. This is the activation value matrix. A smoothing factor is applied. Then, it is transformed into: in, The output matrix after applying the smoothing factor, Construct a function for a diagonal matrix. The smoothed weight matrix, This is the smoothed activation value matrix.

[0079] The quantized output is ,in, Let be the quantization function. Define the joint error as: in, For joint error, and These are the weighting coefficients for modal correlation. For the square norm, since the activation anomalies are more severe, it is set to... For text modality, set .

[0080] To address the "smoothing misalignment" problem in multimodal scenarios, a cross-modal compensation (CMC) mechanism based on Singular Value Decomposition (SVD) whitening is introduced. This mechanism applies to the smoothed weight matrix. Perform singular value decomposition: in, It is a left singular vector matrix. It is a singular value diagonal matrix. It is a right singular vector matrix. For matrix The transpose of .

[0081] Before keeping Construct a low-rank compensation matrix based on the largest singular values: in, It is a low-rank compensation matrix. For the reason before A matrix composed of left singular vectors. For the reason before A diagonal matrix composed of the largest singular values. For the reason before A matrix consisting of right singular vectors.

[0082] The final quantization weights are expressed as follows: in, For the final quantization weights, The compensation intensity coefficients are adaptively determined using a calibration dataset. The local rank of the compensation matrix... It is obtained by solving the MCKP algorithm in Module 2.

[0083] For the joint error function Regarding the smoothing factor By taking the derivative and incorporating the inference delay constraint, we construct the Lagrange optimization problem: in, For inference delay function; Maximum delay; These are Lagrange multipliers used to characterize the strength of the effect of the delay constraint on the optimization objective. The KKT conditions are solved as follows: Obtain the theoretically optimal smoothing factor This method ensures that the joint quantization error of weights and activation values ​​is minimized while satisfying inference delay constraints.

[0084] The complete quantization algorithm flow of this invention is as follows: The input is a multimodal large model , calibration dataset Video memory budget Inference delay limit The output is the quantized model. .

[0085] Step 1: Model Analysis Phase.

[0086] Iterate through all layers of the model and identify the modality type of each layer, including visual, text, and other modalities; use a calibration dataset. Collect activation value statistics for each layer and each channel; calculate the outlier magnitude index for each channel. and equivalent channel conditions .

[0087] Step 2: Hierarchical scheme space construction.

[0088] For each layer Based on its modal type and outlier distribution, a set of candidate quantization schemes is generated. For each candidate solution, evaluate its accuracy loss. and video memory overhead .

[0089] Step 3: Optimize global resource allocation.

[0090] The MCKP problem is solved using dynamic programming to obtain the optimal quantization scheme for each layer; based on the selected scheme, the quantization bit configuration and the rank of the low-rank compensation matrix for each layer are determined.

[0091] Step 4: Optimize the smoothing factor layer by layer.

[0092] For each layer, calculate the smoothing resource budget based on its selected quantization scheme. The optimal smoothing factor for each channel of this layer is determined using a water-filling algorithm. Apply a smoothing transformation to obtain the adjusted weights. and activation value .

[0093] Step 5: Quantification and Compensation.

[0094] Quantization is performed on the smoothed weights and activation values; for layers requiring compensation, SVD decomposition is performed and a low-rank compensation matrix is ​​constructed. The quantized weights are then fused with the compensation matrix to obtain the final weights. .

[0095] Step 6: Joint error verification and fine-tuning.

[0096] Evaluate the accuracy loss of the quantization model on the calibration dataset; if the accuracy loss exceeds a threshold, adjust the modal weight coefficients. and compensation intensity Repeat steps 4-5 until the accuracy requirement is met or the maximum number of iterations is reached.

[0097] Step 7: Output the quantization model.

[0098] Save the quantized model parameters; record the smoothing factor, quantization configuration, and compensation matrix for each layer; generate the metadata files required by the inference engine.

[0099] Smoothing factor The value typically ranges from 0.1 to 1.0. For high-abnormal channels in the visual modality, a range of 0.6–0.9 is recommended; for normal channels in the text modality, a range of 0.3–0.5 is recommended. In extreme cases, the range can be extended to 0.05–1.0, but care should be taken to avoid values ​​that are too small. The value may lead to excessive weight quantization error.

[0100] The choice of the rank of the compensation matrix needs to balance precision and memory overhead. For a standard Transformer layer (768 hidden dimensions), the recommended rank range is: low outlier layers (O<10): (No compensation) or Middle anomaly layer (10≤O<40): or High anomaly layer (O≥40): or For larger hidden dimensions (such as 1024 or 2048), the rank can be increased proportionally, but usually does not exceed 1 / 16 of the hidden dimension.

[0101] Modal weighting coefficients and Must meet Recommended settings: Text modality: Visual modality: Audio modality: ; Fusion layer: .

[0102] Compensation strength coefficient To control the contribution of the low-rank compensation matrix, a range of 0.1-0.3 is recommended. A smaller value... Values ​​(0.1-0.15) are suitable for scenarios where high precision is not required; larger values... The value (0.2-0.3) is suitable for scenarios requiring high precision. Values ​​that are too large (>0.4) may lead to overfitting of the calibration dataset.

[0103] The size of the calibration dataset affects quantization accuracy and computational cost. Recommended sizes: Small models (<3B parameters): 500-1000 samples; Medium models (3B-10B parameters): 1000-2000 samples; Large models (10B-100B parameters): 2000-5000 samples; Very large models (>100B parameters): 5000-10000 samples. Calibration data should cover the model's main application scenarios. For multimodal models, a balanced distribution of data across modalities is necessary. 5.6 Iterative Parameter Optimization Iterative parameter settings for the joint error verification and fine-tuning stage: maximum number of iterations: 3-5; accuracy loss threshold is set according to the application scenario, usually 2%-5%; parameter adjustment step size. , Convergence is determined when the change in accuracy loss between two consecutive iterations is less than 0.5%.

[0104] Example 3 is an embodiment of the present invention, which provides a smooth quantization activation value transfer method for a multimodal large model with trillion parameters. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0105] This embodiment performs post-training quantization on a large visual-language multimodal model (7B parameter size) containing 32 Transformer layers. The model includes a visual encoder (first 12 layers) and a language decoder (last 20 layers).

[0106] Step 1: Activation value analysis.

[0107] Using a calibration dataset containing 1000 image-text pairs, activation value statistics for each layer were collected. Abnormal amplitudes of activation values ​​in the visual encoder layer were discovered. The average value is 45.3, while the language decoder layer is only 3.2, a difference of 14 times.

[0108] Step 2: Candidate solution design.

[0109] Five candidate schemes are designed for the visual encoder layer: Solution V1: W8A8, no compensation, 1.0x video memory usage, 12.3% precision loss.

[0110] Scheme V2: W8A8, rank r=16 compensation, video memory usage 1.2x, precision loss 6.8%.

[0111] Solution V3: W8A16, no compensation, 1.5x memory usage, 4.2% precision loss.

[0112] Scheme V4: W8A8, rank r=32 compensation, video memory usage 1.4x, precision loss 3.1%.

[0113] Solution V5: W16A16, no compensation, 2.0x video memory usage, 0.8% precision loss.

[0114] For the language decoder layer, three candidate schemes are designed: Option T1: W8A8, no compensation, 1.0x memory usage, 2.1% precision loss.

[0115] Scheme T2: W8A8, rank r=8 compensation, video memory usage 1.1x, precision loss 1.2%.

[0116] Solution T3: W8A16, no compensation, 1.5x memory usage, 0.6% precision loss.

[0117] Step 3: Solve using MCKP.

[0118] The total video memory budget was set to 1.3 times that of the original model. Applying dynamic programming, the solution yielded the following: V4 for the first 8 layers of the visual encoder, V2 for the last 4 layers; and T1 for all 20 layers of the language decoder. Under this configuration, the total accuracy loss was 3.8%, and the video memory usage was 1.28 times the original model, satisfying the budget constraint.

[0119] Step 4: Smoothing factor optimization.

[0120] For the first layer of the visual encoder (selecting scheme V4), the smoothing resource budget is calculated. This layer has 768 channels, and the top 100 channels with the largest outlier amplitudes... The values ​​are in the range of 0.02-0.05, while the values ​​for the other channels are in the range of 0.15-0.30. A water-filling algorithm is applied to assign smoothing factors to the abnormal channels. The normal range is between 0.6 and 0.8, while the normal range is between 0.3 and 0.5.

[0121] Step 5: Quantization and compensation.

[0122] Perform W8 quantization on the smoothed weight matrix, and perform A8 quantization on the activation values. For layers that require compensation, perform SVD decomposition, retain the first 32 singular values to construct a compensation matrix. The compensation strength coefficient is determined to be 0.15 through grid search.

[0123] Step 6: Performance evaluation.

[0124] Tested on the COCO image captioning task, the CIDEr score of the quantized model is 118.3, which is 3.6% lower than that of the original model (122.7). The inference speed is increased by 2.3 times, and the video memory usage is reduced to 78% of that of the original model. Compared with directly applying SmoothQuant (CIDEr score of 108.5, a decrease of 11.6%), the present invention significantly improves the quantization accuracy.

[0125] Embodiment 4, which is an embodiment of the present invention, provides a smoothed quantization activation value transfer method for a trillion-parameter multimodal large model. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0126] In this embodiment, quantization is performed on a 1.2-trillion-parameter full-modality large model (supporting text, image, audio, and video), and the model includes 96 Transformer layers.

[0127] Step 1: Hierarchical analysis.

[0128] The 96 layers are divided into 4 groups according to modality type: text layers (layers 1-24), vision layers (layers 25-48), audio layers (layers 49-72), and fusion layers (layers 73-96). The average outlier magnitudes of each group are 3.8, 52.1, 18.6, and 28.3 respectively.

[0129] Step 2: Candidate scheme pruning.

[0130] Due to the huge scale of the model, the number of candidate schemes designed for each layer needs to be controlled. A heuristic pruning strategy is adopted: according to the outlier magnitude, the layers are divided into three categories: high outlier (O>40), medium outlier (10<O≤40), and low outlier (O≤10), with 7, 5, and 3 candidate schemes designed respectively. The total number of candidate schemes is reduced from the original 960 (96×10) to 480.

[0131] Step 3: Grouped MCKP solving.

[0132] The MCKP problem of the 96 layers is decomposed into 4 sub-problems, each group is solved separately, and then global adjustment is performed. The total video memory budget is set to 1.5 times that of the original model. The solution result is: Text layer: All text uses W8A8 without compensation (Scheme T1).

[0133] Visual layer: The first 16 layers use W8A8 rank 64 compensation (Scheme V4), and the last 8 layers use W8A8 rank 32 compensation (Scheme V3).

[0134] Audio layer: The first 12 layers use W8A8 rank 16 compensation (Scheme A2), and the last 12 layers use W8A8 without compensation (Scheme A1).

[0135] Fusion layer: The first 12 layers use W8A8 rank 32 compensation (Scheme F3), and the last 12 layers use W8A8 rank 16 compensation (Scheme F2).

[0136] Step 4: Modality-aware smoothing optimization.

[0137] Different modal weighting coefficients are used for different modal layers. Visual layer settings. Emphasis is placed on controlling the quantization error of activation values; text layer settings To balance the two; the audio layer and the blending layer are set according to their outlier distribution. .

[0138] Step 5: Distributed quantization execution.

[0139] Due to the large model size, a distributed quantization strategy was adopted. The 96 layers were distributed across 8 GPU nodes, with each node responsible for the quantization and compensation calculations of 12 layers. The smoothing factor optimization, quantization, and SVD compensation were performed in parallel on each node, and the results were finally aggregated.

[0140] Step 6: Performance evaluation.

[0141] Evaluations on multimodal understanding benchmark sets showed that the quantized model achieved an accuracy of 82.1% in image question answering (compared to 84.3% for the original model), a CIDEr score of 95.7 in video description (compared to 98.2% for the original model), and an accuracy of 88.6% in audio classification (compared to 89.8% for the original model). The average precision loss was kept below 2.5%. Inference throughput was increased by 3.1 times, and memory usage was reduced to 67% of the original model, meeting the requirements for a single-machine 8-GPU deployment.

[0142] Example 5 is an embodiment of the present invention, which provides a smooth quantization activation value transfer method for a multimodal large model with trillion parameters. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0143] This embodiment is designed for edge device deployment scenarios, and quantizes a 3B parameter visual-language model under extreme memory constraints (only 0.8 times that of the original model).

[0144] Step 1: Design of radical candidate solutions.

[0145] Due to extremely tight memory budgets, a more aggressive quantization scheme was designed: Option E1: W4A8, rank r=8 compensation, video memory usage 0.6x.

[0146] Scheme E2: W6A8, rank r=16 compensation, video memory usage 0.75x.

[0147] Solution E3: W8A8, no compensation, video memory usage 1.0x.

[0148] Scheme E4: W8A8, rank r=8 compensation, video memory usage 1.1x.

[0149] Step 2: Precision-Memory Trade-off Optimization.

[0150] Introduce a memory penalty term into the MCKP objective function and modify it as follows: in, The memory penalty coefficient is determined by binary search, so that the final memory usage exactly satisfies the 0.8x constraint.

[0151] Step 3: Hybrid bit quantization.

[0152] The solution results show that a hybrid bit quantization strategy is required: the first 6 layers of the visual encoder use scheme E2 (W6A8), and the last 6 layers use scheme E1 (W4A8); the first 10 layers of the language decoder use scheme E3 (W8A8), and the last 10 layers use scheme E1 (W4A8). The total video memory usage is 0.78x, which meets the constraints.

[0153] Step 4: Enhance smoothing and compensation.

[0154] For layers using W4 quantization, stronger smoothing and compensation are required due to the larger quantization error. The smoothing resource budget for these layers is increased by 50%, and the rank of the compensation matrix is ​​increased to 16. Simultaneously, the compensation strength coefficient is increased. Increased from 0.15 to 0.25.

[0155] Step 5: Performance evaluation.

[0156] In tests on the Visual Question Answering (VQA) task, the quantized model achieved an accuracy of 68.3% (compared to 72.1% for the original model), with a precision loss of 5.3%. Although the precision loss is greater than in Example 1, it still maintains acceptable performance under extremely resource-constrained conditions. Inference speed is improved by 3.8 times, with a peak VRAM usage of only 2.1GB, enabling smooth operation on mobile GPUs.

[0157] Example 6 is an embodiment of the present invention, which provides a smooth quantization activation value transfer method for a multimodal large model with trillion parameters. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0158] This embodiment demonstrates an online adaptive quantization scheme that can dynamically adjust the quantization strategy according to the actual inference load.

[0159] Step 1: Multi-level quantization configuration pre-calculation.

[0160] During the offline phase, three sets of quantization configurations are pre-calculated for the same model: Configuration A (High Precision): Average W8A8, critical layer W8A16, 1.4x memory, 1.8% precision loss.

[0161] Configuration B (Balanced): Average W8A8, 1.0x video memory, 3.5% precision loss.

[0162] Configuration C (High Efficiency): Hybrid W4-W8, 0.7x video memory, 6.2% precision loss.

[0163] Step 2: Runtime monitoring and switching.

[0164] During inference, the system monitors memory usage and inference latency. When memory usage exceeds 85%, it automatically switches from configuration A to configuration B; when latency exceeds a threshold, it switches to configuration C. The switching process is achieved by dynamically loading pre-calculated quantization parameters, with a latency of less than 50ms.

[0165] Step 3: Adjust the incremental smoothing factor.

[0166] After the configuration switch, the smoothing factor is adjusted incrementally based on the new number of quantization bits. Let the original smoothing factor be... The new quantization bit count is The adjusted smoothing factor is: This formula ensures that as the number of quantization bits decreases, the smoothing strength increases accordingly to compensate for the increase in quantization error.

[0167] Step 4: Performance evaluation.

[0168] In mixed-load scenarios (including simple queries and complex multimodal inference), the system can automatically switch between three configurations based on the load. The average accuracy loss is 4.1%, slightly higher than the fixed configuration B (3.5%), but peak memory usage is reduced by 28% and average inference latency is reduced by 35%, significantly improving the system's resource utilization efficiency and response speed.

[0169] Example 7, referring to Figure 2This invention provides a smooth quantization activation value transfer method for multimodal large models with trillions of parameters, as one embodiment of the present invention. To verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0170] To verify the effectiveness of the method of this invention, a comparative experiment was designed. On a large multimodal model containing 32 Transformer layers (128 channels per layer, supporting text, visual, and audio modalities), the performance of the traditional unified quantization method and the method of this invention were compared. The experiment generated 12,288 raw data records, covering quantization performance metrics for all layers, all channels, and all modalities.

[0171] The relationship between abnormal activation value amplitude and quantization error, such as Figure 2 As shown in (a), experimental data show a significant positive correlation between the magnitude of activation anomalies and quantization error. For the visual modality, when the anomaly magnitude exceeds 40, the quantization error of the traditional method rises sharply to over 15, while the method of this invention dynamically allocates smoothing resources through a water-filling algorithm, controlling the error below 6, a reduction of up to 60%. For the text modality, due to the smaller anomaly magnitude (average 3.2), the error difference between the two methods is relatively small, but the method of this invention still maintains an advantage of approximately 23%. The audio modality falls between the two, and the method of this invention achieves a 42% error reduction.

[0172] The distribution characteristics of quantization error in different modes, such as Figure 2 As shown in (b), the box plot clearly illustrates the quantization error distribution for the three modalities under the two methods. The error distribution for the visual modality is the most dispersed, with a median error of 12.3 for the traditional method, while the method of this invention reduces it to 5.2, and outliers are significantly reduced. The error distribution for the text modality is relatively concentrated, with a median error of 1.8 for the method of this invention, a 22% reduction compared to 2.3 for the traditional method. The improvement for the audio modality is the most significant, with the median error decreasing from 8.6 to 5.0, a reduction of 42%. This verifies that the method of this invention has good adaptability to different modalities.

[0173] The effectiveness of the quantization bit dynamic allocation strategy, such as Figure 2 As shown in (c), traditional methods uniformly allocate 8 bits for quantization across all modalities, while the method of this invention dynamically adjusts the quantization based on the anomalous magnitude. Experimental results show that the visual modality achieves an average of 11.2 bits (up to 12 bits), the audio modality achieves 9.8 bits, and the text modality maintains 8 bits. This adaptive allocation strategy provides higher quantization accuracy for highly anomalous modalities within the same memory budget, achieving optimal resource utilization.

[0174] The hierarchical variation pattern of inference delay is as follows: Figure 2As shown in (d), the inference latency of both methods shows a slight upward trend with the increase of the number of layers, which is due to the cumulative computational overhead of deep networks. However, the method of the present invention maintains a latency advantage of about 8% at all layers, with an average inference latency of 2.3ms / layer, which is 7.94% lower than the 2.5ms / layer of the traditional method. The green-filled area represents the computation time saved by the method of the present invention. For a 32-layer model, the total latency is reduced by about 6.4ms, which can significantly improve system performance in high-throughput scenarios.

[0175] The optimization effect of video memory usage is as follows Figure 2 As shown in (e), memory usage increases linearly with the number of layers. However, the method of this invention maintains lower memory usage across all layers through intelligent budget planning using hybrid precision quantization and low-rank compensation. The average memory usage is 167.9 MB / layer, a 7.23% reduction compared to the traditional method's 180.8 MB / layer. For a 32-layer model, this results in a total memory saving of approximately 413 MB, which is crucial for deploying trillion-parameter models, enabling models that would otherwise be undeployable to run on resource-constrained devices.

[0176] Overall performance improvement, such as Figure 2 As shown in (f), the comprehensive performance comparison demonstrates that the method of this invention achieves significant improvements in three key indicators: an average reduction of quantization error of 41.1% (23.0% for text, 58.1% for vision, and 42.1% for audio), a reduction of inference latency of 7.9%, and a reduction of video memory usage of 7.2%. This verifies that the method of this invention, through an interdisciplinary mathematical optimization framework, achieves comprehensive optimization in accuracy, speed, and resource usage, and has obvious comprehensive advantages compared to traditional methods.

[0177] Paired t-tests were performed on the experimental data, and the results showed that the improvements of the proposed method in quantization error, inference latency, and memory usage were statistically significant (p<0.001). This indicates that the observed performance improvement is not a random fluctuation, but a systematic improvement brought about by the method itself.

[0178] Experimental results fully validate the effectiveness of the proposed method. By introducing the VABWF water-filling algorithm and MCKP dynamic programming from the field of communications, this invention can adaptively allocate smoothing resources and quantization bits according to the activation value distribution characteristics of different modalities, achieving theoretically optimal resource utilization efficiency. In particular, for visual modalities with severe activation value anomalies, this invention achieves a 58.1% reduction in quantization error, significantly outperforming traditional unified quantization methods. Simultaneously, this invention also achieves approximately 7%–8% optimization in inference latency and memory usage, providing a feasible technical solution for the practical deployment of trillion-parameter multimodal large-scale models.

[0179] Example 8 is an embodiment of the present invention. This embodiment provides a smooth quantization activation value transfer system for a multimodal large model with trillion parameters, including a statistical determination module, a scheme construction module, a configuration solution module, a smooth transformation module, and a quantization compensation module.

[0180] The statistical determination module is used to collect the activation value distribution statistics of each channel in each layer of the multimodal large model, and to determine the outlier amplitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel.

[0181] The scheme construction module is used to construct a set of candidate quantization schemes for each layer based on the modal type information of each layer and the outlier magnitude index of each channel.

[0182] The configuration solver module is used to perform a multi-choice knapsack solution on the candidate quantization scheme set of each layer under the constraint of memory budget, and to determine the quantization resource configuration scheme of each layer.

[0183] The smoothing transformation module is used to determine the smoothing factor of each channel based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer. It then performs a smoothing transformation on the weight parameters and activation values ​​to obtain the transformed weight parameters and activation values.

[0184] The quantization compensation module is used to perform quantization processing on the transformed weight parameters and the transformed activation values, perform low-rank decomposition processing on the transformed weight parameters to determine the compensation matrix, and then perform compensation processing on the quantized weight parameters according to the compensation matrix. Based on the compensated weight parameters and the quantized activation values, combined with the weight quantization error and the activation value quantization error, the quantized multimodal large model is obtained.

[0185] This embodiment also provides an electronic device applicable to the smooth quantization activation value transfer method for a multimodal large model with trillion parameters, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the smooth quantization activation value transfer method for a multimodal large model with trillion parameters as proposed in the above embodiment.

[0186] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the smooth quantization activation value transfer method for a multimodal large model with trillion parameters as proposed in the above embodiment.

[0187] The storage medium proposed in this embodiment and the method for smooth quantization activation value transfer for multimodal large models with trillion parameters proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0188] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0189] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A smooth quantization activation value transfer method for multimodal large models with trillions of parameters, characterized by: include: Collect activation value distribution statistics for each layer and channel of the multimodal large model, and determine the outlier amplitude index, modality type information of each layer, and channel condition parameters of each channel; Based on the modal type information of each layer and the outlier amplitude index of each channel, a set of candidate quantization schemes for each layer is constructed; Under the constraint of memory budget, a multi-choice knapsack problem is performed on the candidate quantization scheme set of each layer to determine the quantization resource allocation scheme of each layer. Based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer, the smoothing factor of each channel is determined by capacity adjustable water injection allocation processing, and the weight parameters and activation values ​​are smoothed to obtain the transformed weight parameters and transformed activation values. Quantization is performed on the transformed weight parameters and transformed activation values. Low-rank decomposition is then performed on the transformed weight parameters to determine the compensation matrix. Subsequently, compensation is performed on the quantized weight parameters based on the compensation matrix. Based on the compensated weight parameters and quantized activation values, and combining the weight quantization error and activation value quantization error, the quantized multimodal large model is obtained.

2. The smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 1, characterized in that: The method involves collecting activation value distribution statistics for each layer and channel of a multimodal large model, determining outlier magnitude indices for each channel, modal type information for each layer, and channel condition parameters for each channel, including... Based on the calibration dataset, the multimodal large model is traversed through each layer to collect the activation value distribution statistics of each channel in each layer; Based on the statistical information of the activation value distribution of each channel, an outlier magnitude index is determined to characterize the degree of abnormality of activation values. Based on the modal affiliation of each channel, the channels are classified to obtain the modal type information of each layer; Based on the outlier magnitude index, each channel is mapped to channel condition parameters.

3. The smoothing quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 2, characterized in that: Based on the modal type information of each layer and the outlier amplitude index of each channel, a set of candidate quantization schemes is constructed for each layer, including: Based on the modality type information of each layer, candidate quantization schemes for each layer are generated; Based on the outlier magnitude index of each channel, candidate quantization schemes for each layer are screened. For the selected candidate quantization schemes, determine the accuracy loss information and resource cost information; Based on the accuracy loss information and the resource overhead information, a set of candidate quantization schemes for each layer is constructed.

4. The smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 3, characterized in that: Under the constraint of memory budget, the process involves performing a multi-choice knapsack problem on the candidate quantization scheme set for each layer to determine the quantization resource allocation scheme for each layer, including... Transform the set of candidate quantization schemes for each layer into a multiple-choice knapsack problem; The resource overhead information is used as constraint information, and the accuracy loss information is used as the optimization basis; Based on the constraint information and optimization criteria, the multi-choice knapsack problem is solved using a dynamic programming algorithm to determine the target candidate quantization schemes for each layer. Based on the target candidate quantification schemes for each layer, determine the quantification resource allocation scheme for each layer.

5. The smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 4, characterized in that: Based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer, a smoothing factor for each channel is determined using capacity-adjustable water-filling allocation processing. A smoothing transformation is then performed on the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values, including... The quantization bit configuration for each layer is determined based on the quantization resource allocation scheme for each layer. The smoothing resource budget for each layer is determined based on the quantization bit configuration, memory budget constraints, and channel parameter scale. Based on the smoothed resource budget of each layer, determine the resource constraints for capacity-adjustable water injection allocation and treatment; Based on the channel condition parameters of each channel, the smoothing factor of each channel is determined by capacity-adjustable water injection allocation under resource constraints. Based on the smoothing factor of each channel, a smoothing transformation is performed on the weight parameters and activation values, and the abnormal activation values ​​are transferred between the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values.

6. The smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 5, characterized in that: The smoothing factor for each channel is determined using capacity-adjustable water injection allocation processing based on the channel condition parameters of each channel under resource constraints, including: With the goal of minimizing the total quantification error of this layer and the constraint that the total smoothing resource amount does not exceed the resource constraint, a smoothing factor optimization problem is constructed. Construct a Lagrange function based on the smoothing factor optimization problem; Based on the Karouch-Kun-Tucker conditions, the smoothing factor of each channel is solved to determine the Lagrange multipliers; Based on the Lagrange multipliers and the channel condition parameters of each channel, the smoothing factor of each channel is determined.

7. The smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in claim 6, characterized in that: The transformed weight parameters and transformed activation values ​​are quantized, and the transformed weight parameters are decomposed into low-rank values ​​to determine the compensation matrix. Then, compensation is applied to the quantized weight parameters based on the compensation matrix. Based on the compensated weight parameters and quantized activation values, and combining the weight quantization error and the activation value quantization error, the quantized multimodal large model is obtained. include, Quantization is performed on the transformed weight parameters and transformed activation values ​​to obtain quantized weight parameters and quantized activation values; Perform low-rank decomposition on the transformed weight parameters to determine the compensation matrix; The compensation matrix is ​​combined with the quantized weight parameters to obtain the compensated weight parameters; Based on the compensated weight parameters and the quantized activation values, determine the weight quantization error and the activation value quantization error; Based on the compensated weight parameters and the quantized activation values, and combining the weight quantization error and the activation value quantization error, a quantized multimodal large model is obtained.

8. A smoothing quantization activation value transfer system for multimodal large models with trillion parameters, employing the smoothing quantization activation value transfer method for multimodal large models with trillion parameters as described in any one of claims 1 to 7, characterized in that, include: The system includes a statistical determination module, a scheme construction module, a configuration solution module, a smooth transformation module, and a quantization compensation module. The statistical determination module is used to collect the activation value distribution statistics of each channel in each layer of the multimodal large model, and to determine the outlier amplitude index of each channel, the modality type information of each layer, and the channel condition parameters of each channel. The scheme construction module is used to construct a set of candidate quantization schemes for each layer based on the modal type information of each layer and the outlier magnitude index of each channel; The configuration solving module is used to perform a multi-choice knapsack problem on the candidate quantization scheme set of each layer under the constraint of memory budget, and to determine the quantization resource configuration scheme of each layer. The smoothing transformation module is used to determine the smoothing factor of each channel based on the channel condition parameters of each channel and the quantization resource allocation scheme of each layer, and to perform smoothing transformation on the weight parameters and activation values ​​to obtain the transformed weight parameters and transformed activation values. The quantization compensation module is used to perform quantization processing on the transformed weight parameters and the transformed activation values, perform low-rank decomposition processing on the transformed weight parameters to determine the compensation matrix, and then perform compensation processing on the quantized weight parameters according to the compensation matrix. Based on the compensated weight parameters and the quantized activation values, combined with the weight quantization error and the activation value quantization error, the quantized multimodal large model is obtained.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the smooth quantization activation value transfer method for a multimodal large model with trillion parameters as described in any one of claims 1 to 7.

Images