Model compression method, device, equipment, storage medium and program product

Abstract

The application provides a model compression method, device, equipment, storage medium and program product. It relates to the field of large models. The method comprises the following steps: determining a plurality of weight matrices corresponding to a first model to be compressed; obtaining a plurality of preset compression parameter sets corresponding to the plurality of weight matrices respectively, wherein the plurality of preset compression parameter sets comprise a plurality of compression levels; determining target compression levels corresponding to the plurality of weight matrices respectively, so that when the plurality of weight matrices are compressed according to the corresponding target compression levels, the difference between the total error reduction benefit of the plurality of weight matrices and the preset global benefit is less than or equal to a first preset value, and the total error reduction benefit is less than or equal to the preset global benefit; and compressing the plurality of weight matrices according to the target compression levels respectively to obtain a target model after compression. The compression performance of the model is improved.

Description

Technical Field

[0001] This application relates to the field of large models, and more particularly to a method, apparatus, device, storage medium, and program product for compressing models. Background Technology

[0002] With the increasing demand for deploying large language models on edge devices (such as smartphones and edge servers), model compression technology has become a key means to improve computing efficiency and reduce storage and energy consumption.

[0003] In related technologies, model compression can be achieved through additive codebook quantization, and weight blocks can be reconstructed additively from multiple learnable codebooks, thus achieving several times compression while maintaining good model performance.

[0004] However, in the above method, using a uniform codebook configuration for all weight matrices leads to insufficient protection of sensitive matrices and wasted bits for redundant matrices, resulting in poor compression performance. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for compressing a model to solve the problem of poor compression performance.

[0006] Firstly, this application provides a method for compressing a model, comprising:

[0007] Determine multiple weight matrices corresponding to the first model to be compressed; these multiple weight matrices are used to support the operation of the first model.

[0008] Obtain a set of preset compression parameters corresponding to multiple weight matrices. The set of preset compression parameters includes multiple compression levels, each of which corresponds to a different compression bit. Each compression level includes the number of codebooks, the number of codewords, the input group, and the output group of the first codebook used to compress the weight matrix.

[0009] Determine the target compression level corresponding to each of the multiple weight matrices, so that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain. The total error reduction gain is used to indicate the sum of the ratios of the output error of the multiple weight matrices to the bit overhead.

[0010] The multiple weight matrices are compressed according to the target compression level to obtain the compressed target model.

[0011] In one possible design, multiple weight matrices are determined to correspond to the target compression levels, including:

[0012] Among the multiple compression levels within each preset compression parameter set, the first compression level is determined, and the first compression level has the smallest compression bits.

[0013] Based on the first compression level corresponding to each of the multiple weight matrices, determine the first error reduction benefit of each weight matrix;

[0014] Based on the first error reduction gain of each weight matrix, determine the first total error reduction gain of multiple weight matrices;

[0015] Based on the first total error reduction gain of multiple weight matrices, the target compression level corresponding to each of the multiple weight matrices is determined.

[0016] In one possible design, the target compression levels corresponding to the multiple weight matrices are determined based on the first total error reduction gain of the multiple weight matrices, including:

[0017] Determine whether the first difference between the first total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and whether the first total error reduction benefit is less than or equal to the preset global benefit;

[0018] If so, then the first compression level within each preset compression parameter set is determined as the target compression level corresponding to each weight matrix;

[0019] If not, then in each weight matrix, determine the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix, and the first error reduction benefit of the first weight matrix is ​​maximized;

[0020] Based on the first weight matrix and the first preset compression parameter set, the target compression levels corresponding to the multiple weight matrices are determined respectively.

[0021] In one possible design, based on a first weight matrix and a first preset set of compression parameters, multiple target compression levels corresponding to different weight matrices are determined, including:

[0022] In the first preset compression parameter set, a second compression level is determined, wherein the compression bits of the second compression level are greater than the compression bits of the first compression level and less than the compression bits of the other compression levels;

[0023] The second error reduction gain of the first weight matrix is ​​determined based on the first weight matrix and the second compression level.

[0024] The second total error reduction benefit is determined based on each of the first and second error reduction benefits.

[0025] When the second difference between the second total error reduction gain and the preset global gain is less than or equal to the first preset value, and the second total error reduction gain is less than or equal to the preset global gain, the second compression level is determined as the target compression level of the first weight matrix, and the first compression level is determined as the target compression level of the remaining weight matrices.

[0026] When the second difference between the second total error reduction gain and the preset global gain is greater than the first preset value, the compression level of the weight matrix with the highest error reduction gain is replaced with the third compression level whose compression bits are higher than the previous compression level and lower than the other compression levels, and the total error reduction gain is determined until the difference between the total error reduction gain and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

[0027] In one possible design, for any second weight matrix, multiple weight matrices are compressed according to the target compression level to obtain the compressed target model, including:

[0028] Generate the initial codebook based on the second weight matrix;

[0029] Based on the initial codebook, determine the first index set, which is used to indicate the positions of multiple vector samples;

[0030] Based on the first target compression level corresponding to the first index set and the second weight matrix, the second weight matrix is ​​subjected to alternating optimization processing. The alternating optimization processing includes fixing the first index set and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first index set.

[0031] When the rate of change of the matching loss of the second weight matrix is ​​less than the second preset value, the optimized target index set and target codebook are determined. The matching loss is used to indicate the difference between the second weight matrix before and after compression.

[0032] The second weight matrix is ​​compressed based on the target index set and the target codebook.

[0033] In one possible design, after compressing multiple weight matrices according to the target compression level to obtain the compressed target model, the following steps are also included:

[0034] In response to the processing request of the target task, for any third weight matrix of the target task, a pre-training codebook corresponding to the third weight matrix is ​​generated. The pre-training codebook is the same as the target codebook corresponding to the third weight matrix.

[0035] Generate the first residual codebook, the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook;

[0036] Based on the target task and the pre-trained codebook, the first residual codebook is alternately optimized to obtain a second residual codebook that is adapted to the target task.

[0037] Based on the pre-trained codebook and the second residual codebook, the training codebook corresponding to the target task is determined so that the third weight matrix can be compressed according to the training codebook when processing the target task.

[0038] Secondly, this application provides a model compression device, comprising: a first determining module, an acquiring module, a second determining module, and a first compression module, wherein,

[0039] The first determining module is used to determine multiple weight matrices corresponding to the first model to be compressed, and the multiple weight matrices are used to support the operation of the first model;

[0040] The acquisition module is used to acquire a set of preset compression parameters corresponding to multiple weight matrices. The set of preset compression parameters includes multiple compression levels, each of which corresponds to a different compression bit. Each compression level includes the number of codebooks, the number of codewords, the input group, and the output group of the first codebook used to compress the weight matrix.

[0041] The second determining module is used to determine the target compression level corresponding to the multiple weight matrices respectively, so that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain. The total error reduction gain is used to indicate the sum of the ratios of the output error of the multiple weight matrices to the bit overhead.

[0042] The first compression module is used to compress multiple weight matrices according to the target compression level to obtain the compressed target model.

[0043] In one possible design, the second determining module is specifically used for,

[0044] Among the multiple compression levels within each preset compression parameter set, the first compression level is determined, and the first compression level has the smallest compression bits.

[0045] Based on the first compression level corresponding to each of the multiple weight matrices, determine the first error reduction benefit of each weight matrix;

[0046] Based on the first error reduction gain of each weight matrix, determine the first total error reduction gain of multiple weight matrices;

[0047] Based on the first total error reduction gain of multiple weight matrices, the target compression level corresponding to each of the multiple weight matrices is determined.

[0048] In one possible design, the second determining module is specifically used for,

[0049] Determine whether the first difference between the first total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and whether the first total error reduction benefit is less than or equal to the preset global benefit;

[0050] If so, then the first compression level within each preset compression parameter set is determined as the target compression level corresponding to each weight matrix;

[0051] If not, then in each weight matrix, determine the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix, and the first error reduction benefit of the first weight matrix is ​​maximized;

[0052] Based on the first weight matrix and the first preset compression parameter set, the target compression levels corresponding to the multiple weight matrices are determined respectively.

[0053] In one possible design, the second determining module is specifically used for,

[0054] In the first preset compression parameter set, a second compression level is determined, wherein the compression bits of the second compression level are greater than the compression bits of the first compression level and less than the compression bits of the other compression levels;

[0055] The second error reduction gain of the first weight matrix is ​​determined based on the first weight matrix and the second compression level.

[0056] The second total error reduction benefit is determined based on each of the first and second error reduction benefits.

[0057] When the second difference between the second total error reduction gain and the preset global gain is less than or equal to the first preset value, and the second total error reduction gain is less than or equal to the preset global gain, the second compression level is determined as the target compression level of the first weight matrix, and the first compression level is determined as the target compression level of the remaining weight matrices.

[0058] When the second difference between the second total error reduction gain and the preset global gain is greater than the first preset value, the compression level of the weight matrix with the highest error reduction gain is replaced with the third compression level whose compression bits are higher than the previous compression level and lower than the other compression levels, and the total error reduction gain is determined until the difference between the total error reduction gain and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

[0059] In one possible design, the first compression module is specifically used for,

[0060] Generate the initial codebook based on the second weight matrix;

[0061] Based on the initial codebook, determine the first index set, which is used to indicate the positions of multiple vector samples;

[0062] Based on the first target compression level corresponding to the first index set and the second weight matrix, the second weight matrix is ​​subjected to alternating optimization processing. The alternating optimization processing includes fixing the first index set and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first index set.

[0063] When the rate of change of the matching loss of the second weight matrix is ​​less than the second preset value, the optimized target index set and target codebook are determined. The matching loss is used to indicate the difference between the second weight matrix before and after compression.

[0064] The second weight matrix is ​​compressed based on the target index set and the target codebook.

[0065] In one possible design, a second compression module is also included.

[0066] The second compression module is used to, in response to the processing request of the target task, generate a pre-trained codebook corresponding to any third weight matrix of the target task, wherein the pre-trained codebook is the same as the target codebook corresponding to the third weight matrix.

[0067] Generate the first residual codebook, the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook;

[0068] Based on the target task and the pre-trained codebook, the first residual codebook is alternately optimized to obtain a second residual codebook that is adapted to the target task.

[0069] Based on the pre-trained codebook and the second residual codebook, the training codebook corresponding to the target task is determined so that the third weight matrix can be compressed according to the training codebook when processing the target task.

[0070] Thirdly, this application provides an electronic device, comprising: at least one processor and a memory; the memory storing computer-executable instructions; and at least one processor executing the computer-executable instructions stored in the memory, such that the at least one processor performs the compression method of the first aspect and various possible designs of the first aspect.

[0071] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the compression method of the first aspect and various possible designs of the first aspect.

[0072] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, implements the compression method of the model described in the first aspect and various possible designs of the first aspect.

[0073] The compression method, apparatus, device, storage medium, and program product provided in this application, when compressing a first model, can determine multiple weight matrices corresponding to the first model to be compressed, which are used to support the operation of the first model; obtain a preset compression parameter set corresponding to each of the multiple weight matrices, which includes multiple compression levels, each of which corresponds to different compression bits, and each compression level includes the number of codebooks, the number of codewords, input groups, and output groups of the first codebook used to compress the weight matrices; determine the target compression level corresponding to each of the multiple weight matrices, such that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain, where the total error reduction gain is used to indicate the sum of the ratios of the mean square error of the multiple weight matrices to the bit overhead; and compress the multiple weight matrices according to the target compression level to obtain the compressed target model. In this way, by using the above method, based on the multi-specification compression levels adapted to each weight matrix, and with the total error reduction benefit as a constraint, the optimal target compression level of each weight matrix can be accurately matched. This enables the allocation of bit resources to weight matrices with higher cost-effectiveness in error improvement, thus getting rid of the resource waste and precision imbalance problems of uniform compression specifications. Under the premise of controllable global bit overhead, the overall compression accuracy is maximized, resulting in a low-bit, high-fidelity compression model, thereby improving the performance of model compression. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0075] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application;

[0076] Figure 2 A flowchart illustrating a model compression method provided in an embodiment of this application;

[0077] Figure 3 This is a schematic diagram illustrating the process of determining the target compression levels corresponding to multiple weight matrices, as provided in an embodiment of this application.

[0078] Figure 4 A schematic diagram illustrating the process of compressing any second weight matrix as provided in an embodiment of this application;

[0079] Figure 5 A schematic diagram illustrating the task-oriented adaptation process for compressing the target model, as provided in the embodiments of this application;

[0080] Figure 6 This is a schematic diagram of the structure of a compression device for a model provided in an embodiment of this application;

[0081] Figure 7 A schematic diagram of the structure of a compression device for another model provided in an embodiment of this application;

[0082] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0083] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0084] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed herein.

[0085] The collection, storage, use, processing, transmission, provision, and disclosure of user data and other information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0086] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0087] The following is an explanation of the terms used in this application.

[0088] Transformer large-scale models: Transformer large-scale models are giant artificial intelligence models built around a self-attention mechanism. They consist of self-attention and multilayer perceptron (MLP) modules, with different modules having varying sensitivities to quantization. Transformer large-scale models abandon the sequential computation mode of traditional recurrent neural networks. Relying on a multi-head attention structure, they can simultaneously capture global long-distance relationships in input content such as text and images. By stacking dozens or hundreds of basic Transformer modules and equipping them with billions to hundreds of billions of parameters, they are pre-trained on massive amounts of image and text data. Once the scale reaches a certain level, powerful general-purpose capabilities such as reasoning, creation, and multi-task adaptation emerge. Based on their structure, they are divided into three categories: pure decoders, pure encoders, and a combination of encoder and decoder.

[0089] Weight Matrix: The weight matrix is ​​a two-dimensional numerical matrix in a neural network (including the Transformer model) that stores trainable parameters. Each value in the matrix represents the strength of the connection between two neurons. During model training, the backpropagation algorithm continuously corrects all weight values ​​in the matrix to reduce the error between the prediction result and the true label. The mapping layer and feedforward network layer in the Transformer each have their own independent weight matrix. The vast majority of the overall model parameters are composed of various weight matrices. Subsequent model compression operations such as quantization and pruning are essentially reducing the numerical precision of the weight matrix and removing invalid near-zero weight elements to reduce the size and improve inference speed.

[0090] To facilitate understanding, the following will be combined with... Figure 1 The system architecture applicable to the embodiments of this application will be described.

[0091] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application. Please refer to [link / reference]. Figure 1 This includes electronic devices, which can refer to devices with on-device computing capabilities. For example, electronic devices can include edge servers, mobile phones, mobile computers, IoT devices, etc. Large models can be deployed on electronic devices. In this embodiment of the application, electronic devices can refer to edge devices with limited storage and computing resources. When it is necessary to run large models on edge devices, it is necessary to achieve high-precision compression of the models under limited storage and computing resources.

[0092] In related technologies, model parameters are typically compressed using methods such as quantization to reduce storage footprint and computational burden when the model is deployed on the edge. Specifically, multiple weight matrices in the model are processed according to a pre-defined compression configuration, and the weight data is expressed through codebooks and grouping methods to reduce the bit overhead required for parameter representation.

[0093] However, such schemes often employ relatively fixed or convergent compression configurations for different weight matrices, failing to fully consider the differences in function, redundancy, and quantization sensitivity among the matrices within the model. As a result, under the same global bit budget, some matrices that are more sensitive to errors may not receive sufficient bit allocation, while some highly redundant matrices occupy unnecessary compression resources, leading to an unbalanced overall resource utilization.

[0094] Especially under extremely low bit conditions, the aforementioned imbalance further amplifies quantization errors, reducing model output accuracy and consequently affecting response quality, task stability, and practical availability on edge devices. Therefore, existing solutions struggle to simultaneously balance compression ratio, accuracy preservation capabilities, and the stringent resource consumption requirements of edge deployment.

[0095] To address the aforementioned technical problems, in this embodiment, when a first model needs to be compressed, multiple weight matrices corresponding to the first model to be compressed can be determined, and these multiple weight matrices are used to support the operation of the first model; a preset compression parameter set corresponding to each of the multiple weight matrices is obtained, the preset compression parameter set including multiple compression levels, each compression level corresponding to different compression bits, and each compression level including the number of codebooks, the number of codewords, input groups, and output groups of the first codebook used to compress the weight matrices; a target compression level corresponding to each of the multiple weight matrices is determined, such that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain, the total error reduction gain being used to indicate the sum of the ratios of the mean square error of the multiple weight matrices to the bit overhead; and the multiple weight matrices are compressed according to the target compression level to obtain the compressed target model. In this way, by using the above method, based on the multi-specification compression levels adapted to each weight matrix, and with the total error reduction benefit as a constraint, the optimal target compression level of each weight matrix can be accurately matched. This enables the allocation of bit resources to weight matrices with higher cost-effectiveness in error improvement, thus getting rid of the resource waste and precision imbalance problems of uniform compression specifications. Under the premise of controllable global bit overhead, the overall compression accuracy is maximized, resulting in a low-bit, high-fidelity compression model, thereby improving the performance of model compression.

[0096] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0097] Figure 2 This is a schematic flowchart illustrating a model compression method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 As shown, the method may include the following steps:

[0098] S201. Determine the multiple weight matrices corresponding to the first model to be compressed.

[0099] The execution entity in this application embodiment can be an edge electronic device or a model compression device installed in the edge electronic device. The model compression device can be implemented by software or by a combination of software and hardware. The edge electronic device can be a device with on-device computing capabilities, such as an edge server, tablet, computer, or mobile phone.

[0100] The first model can refer to the original high-precision large model to be compressed, such as a full-precision Transformer large language model, a visual Transformer model, etc.

[0101] The weight matrix can refer to a two-dimensional matrix composed of all trainable parameters in the first model. It is the core computational carrier for the first model to complete vector transformation, attention calculation, feature dimensionality increase and decrease, and word probability prediction. Understandably, multiple weight matrices can be used to support the computation of the first model. Without weight matrices, the model cannot complete inference computation.

[0102] For example, the weight matrix may include an attention query matrix, an attention key matrix, an attention value matrix, an attention fusion output matrix, a feedforward neural network dimension-up matrix, a feedforward neural network dimension-down matrix, etc.

[0103] After receiving the structure definition file and parameter file of the first model, the edge electronic device can perform structure parsing on the first model and determine multiple weight matrices from the parameters. For example, during execution, an index relationship corresponding to the multiple weight matrices can be established for subsequent steps to call.

[0104] Through structural analysis, parameter traversal, and matrix identification, multiple weight matrices supporting model operations can be identified from the first model, forming a set of matrices that can be directly called in subsequent steps. This allows the compression process to be established at the granularity of the model parameter matrices. This processing enables subsequent compression level allocation to be performed separately for different matrices, rather than using a uniform configuration for the entire model, thus providing a basis for differentiated resource allocation under global benefit constraints.

[0105] S202. Obtain the preset compression parameter set corresponding to each of the multiple weight matrices.

[0106] The preset compression parameter set can refer to the set of candidate compression configurations established for each weight matrix, which describes the multiple compression levels that can be selected for that matrix.

[0107] The preset compression parameter set can include multiple compression levels, each corresponding to different compression bits. Each compression level includes the number of codebooks, the number of codewords, the input blocks, and the output blocks of the first codebook used to compress the weight matrix.

[0108] The first codebook can be the basic structure used for codeword mapping and reconstruction during weight matrix compression, i.e., the additive codebook structure used in this layer. The number of codebooks in the first codebook can reflect the number of codebooks used during matrix compression, i.e., the number of codebooks added in parallel within a set of layers.

[0109] The number of codewords refers to how many vector codewords are stored in a single codebook. Input grouping refers to the length of the vector blocks into which each row of the weight matrix is ​​divided, with each vector block being the smallest unit for codebook reconstruction. Output grouping refers to how many rows of vectors in the weight matrix share the same codebook.

[0110] For example, the set of preset compression parameters corresponding to any weight matrix can be as shown in Table 1. Please refer to Table 1. The set of preset compression parameters can include 5 compression levels.

[0111] Table 1

[0112]

[0113] Optionally, this can be accomplished by reading pre-set compression parameter information, or by determining the corresponding set of pre-set compression parameters based on the weight matrix.

[0114] The electronic device can establish a corresponding parameter set for each weight matrix, and the parameter set contains multiple compression levels. Each compression level contains at least the following parameters: compression bits, number of codebooks, number of codewords, input blocks, and output blocks.

[0115] The input and output groups jointly determine the grouping method of the matrix. The number of codewords is related to the discreteness of the codebook, and the compressed bits are used to characterize the compression level corresponding to that compression level. In this way, the establishment and acquisition of the preset compression parameter set enables each weight matrix to have multiple candidate schemes with clear structural parameters and bit attributes. Subsequently, global coordination and selection can be performed among these discrete candidates, thereby achieving comparable configurations of different matrix compression granularities, codebook complexity, and error costs.

[0116] S203. Determine the target compression level corresponding to the multiple weight matrices respectively, so that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

[0117] The target compression level can refer to the compression configuration level ultimately selected for each weight matrix.

[0118] Preset global revenue refers to the upper limit or target benchmark of revenue allowed at the global level, i.e., the pre-set revenue ceiling, corresponding to the maximum global bit budget that the hardware can handle. Understandably, a higher preset global revenue means higher overall bit overhead and a larger model size.

[0119] The first preset value can refer to the threshold used to determine whether the current total error reduction gain is sufficiently close to the preset global gain. It can be used to allow the final total gain to approach the global gain infinitely without exceeding the upper limit of the global gain, thus avoiding an iterative infinite loop.

[0120] Error reduction benefit can refer to single-matrix single-level improvement. Error reduction benefit can be determined by the ratio of output error after upgrading level to bit overhead after upgrading level. That is, error reduction benefit can refer to how much error can be reduced by investing 1 more bit. The higher the cost-effectiveness, the higher the priority for upgrading. Here, output error can be mean square error.

[0121] In one possible implementation, the ROI of error reduction can be expressed as:

[0122]

[0123] in, This represents the decrease in error after upgrading the level, i.e., the output error. This represents the bit overhead after upgrading the level.

[0124] Total error reduction gain is used to indicate the sum of the ratios of the output error to the bit overhead of multiple weight matrices, that is, the sum of the error reduction gains of each weight matrix.

[0125] In one possible implementation, the target compression levels corresponding to multiple weight matrices can be determined as follows: First compression levels are determined among the multiple compression levels within each preset set of compression parameters; based on the first compression levels corresponding to the multiple weight matrices, the first error reduction gain of each weight matrix is ​​determined; based on the first error reduction gain of each weight matrix, the first total error reduction gain of the multiple weight matrices is determined; and based on the first total error reduction gain of the multiple weight matrices, the target compression levels corresponding to the multiple weight matrices are determined.

[0126] Among them, the first compression level has the smallest number of compressed bits, meaning that the first compression level can be used as the initial compression level for each weight matrix.

[0127] In the specific implementation, the base error and base bits at the lowest level (i.e., the first compression level) can be calculated for each weight matrix. The first error reduction gain of each matrix when moving up one level can be calculated. The first error reduction gains of all weight matrices are added together to obtain the first total error reduction gain. Based on the first total error reduction gain, it is determined whether to upgrade the compression level, thereby determining the target compression level corresponding to each of the multiple weight matrices.

[0128] In this way, by determining the target compression level of each matrix under global benefit constraints, the compression program does not adopt a convergent configuration for all matrices. Instead, it coordinates the allocation based on the differences in error sensitivity and bit cost between matrices, ensuring that the overall combination of levels satisfies the condition that the total error reduction benefit does not exceed the preset global benefit and is sufficiently close to it. Thus, under the corresponding global constraints, compression resources can be allocated among multiple weight matrices, thereby enabling the compressed model to obtain more stable error control results.

[0129] S204. Compress multiple weight matrices according to the target compression level to obtain the compressed target model.

[0130] The target model can refer to the model result formed after parameter compression is completed based on the target compression level determined by each weight matrix.

[0131] The electronic device can perform compression according to the first codebook parameters in the target compression level corresponding to each weight matrix. This includes processing the weight matrix based on input and output groups, and obtaining the compressed parameter representation based on the corresponding codebook number and codeword number. For different weight matrices, compression can be performed independently according to their respective target compression levels. After compression, the compressed target model can be generated.

[0132] In one possible implementation, consistency verification of the compression results may also be included. After all weight matrices are compressed, the compression program can backfill the parameter reference relationships according to the original model structure, generate the compressed target model, and perform a model integrity check to confirm that the model computation graph is consistent with the first model. Subsequently, forward inference can be performed on the target model using verification samples to verify that the compressed parameter file can be loaded correctly and that the output values ​​are within the preset error range.

[0133] By compressing multiple weight matrices according to the target compression level, a target model can be obtained that maintains the original network structure and computational flow while compressing the parameter representation. This target model has lower parameter usage on devices with limited storage resources, and because the compression level is determined under the constraint that the total error reduction gain is close to the preset global gain, the overall error distribution of the model is more balanced, and there is a clear correspondence between the compression result and the global resource constraints.

[0134] In this embodiment, when the first model needs to be compressed, multiple weight matrices corresponding to the first model to be compressed can be determined. The first model can refer to the original high-precision large model to be compressed, and the weight matrix can refer to a two-dimensional matrix composed of all trainable parameters in the first model. Multiple weight matrices can be used to support the operation of the first model. A preset compression parameter set corresponding to each of the multiple weight matrices is obtained. The preset compression parameter set can refer to a candidate compression configuration set established for each weight matrix, used to describe multiple compression levels that can be selected for the matrix. The preset compression parameter set can include multiple compression levels, each corresponding to a different compression bit. Each compression level includes parameters for compressing the weights. The first codebook for matrix compression includes the number of codebooks, the number of codewords, the input groups, and the output groups. The target compression levels corresponding to multiple weight matrices are determined such that when multiple weight matrices are compressed according to their respective target compression levels, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain. The target compression level can refer to the compression configuration level ultimately selected for each weight matrix. Multiple weight matrices are compressed according to their target compression levels to obtain the compressed target model. The target model can refer to the model result formed after parameter compression based on the determined target compression levels for each weight matrix.

[0135] In this way, by first performing matrix-level analysis on the first model, then establishing candidate compression levels for different weight matrices, including the number of codebooks, codewords, input groups, output groups, and compressed bits, and determining the target compression level for each matrix under a preset global benefit constraint, and finally performing differentiated compression based on their respective levels, the allocation of compression resources is transformed from a unified configuration to a globally coordinated configuration oriented towards matrix differences. Based on this processing, matrices that are more sensitive to errors and matrices with higher redundancy within the model receive different compression intensities. The compressed target model, while satisfying storage and bit constraints, achieves overall error control and resource utilization closer to the preset target, thereby improving the performance of model compression.

[0136] Based on any of the above embodiments, the following, in conjunction with Figure 3 The process of determining the target compression level corresponding to multiple weight matrices is explained in detail.

[0137] Figure 3 This is a schematic diagram illustrating the process of determining the target compression levels corresponding to multiple weight matrices, as provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0138] S301. Among the multiple compression levels in each preset compression parameter set, determine the first compression level respectively.

[0139] The first compression level has the fewest compressed bits.

[0140] It is understandable that the first compression level has the greatest compression intensity, meaning that compression through the first compression level will result in the greatest error.

[0141] In actual implementation, all weight matrices of the first model to be compressed can be traversed. In the preset compression parameter set corresponding to each weight matrix, multiple compression levels are arranged in ascending order of bit value. For any weight matrix, the first level with the lowest bit value in the set can be directly selected as the initial level, i.e. the first compression level. After compression by the first compression level corresponding to each preset compression parameter set, it can be considered that the first model has the smallest volume and the largest overall quantization error.

[0142] In this way, by uniformly initializing all matrices from the lowest compression level, the initial bit overhead of the model is reduced to the minimum, and sufficient bit budget redundancy space is reserved, thus laying the foundation for subsequent greedy upgrades and differentiated resource allocation, and avoiding the initial state from directly exceeding the hardware bit budget.

[0143] S302. Determine the first error reduction benefit of each weight matrix based on the first compression level corresponding to the multiple weight matrices.

[0144] The first error reduction benefit can refer to the error reduction benefit that the weight matrix can generate by upgrading to an adjacent higher level based on the current level (i.e., the first compression level). It can represent the cost-effectiveness of upgrading the matrix.

[0145] Understandably, upgrading to an adjacent higher level can refer to replacing the compression level with a compression level whose compression bits are greater than the current compression level's compression bits and less than the compression bits of other compression levels.

[0146] In one possible implementation, for each weight matrix at the first compression level, the output error D1 and bit overhead R1 of the current first compression level can be retrieved, and the output error D2 and bit overhead R2 of the next higher adjacent compression level can be retrieved; the error reduction value after upgrading the level can be calculated. (i.e., output error) and bit overhead after upgrading the level. ; Calculate the error reduction gain of a single-weight matrix, i.e. That is, the first error reduction benefit; store the initial upgrade cost-effectiveness of each weight matrix for subsequent global optimization.

[0147] In this way, weight matrices with higher returns or greater sensitivity to error can be assigned higher compression levels, while weight matrices with lower returns remain at lower compression levels, thus matching the target compression level with the matrix characteristics. The output of the compressed target compression level is the encoded configuration of each weight matrix, which is used to subsequently generate the compressed target model.

[0148] In this way, by quantifying the upgrade cost-effectiveness of each matrix, the fuzzy sensitivity levels are transformed into comparable values, thereby achieving data-driven allocation from the root, rather than allocating bit resources based on human experience.

[0149] S303. Determine the first total error reduction benefit of multiple weight matrices based on the first error reduction benefit of each weight matrix.

[0150] The first total error reduction gain can refer to the value obtained by summing the first error reduction gains of all weight matrices in the first model. The first total error reduction gain can represent the initial upgrade potential of the first model as a whole and can be used to compare and judge with the preset global gain.

[0151] S304. Determine whether the first difference between the first total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and whether the first total error reduction benefit is less than or equal to the preset global benefit.

[0152] The first preset value can be a value set by the user in advance. The first preset value can be a minimum convergence threshold (e.g., 0.3, 0.5). The first preset value can be used to determine whether the current first total error reduction gain has infinitely approached the budget (i.e., the preset global gain) upper limit, and serve as the stopping condition for iteration.

[0153] The first difference can refer to the difference between the first total error reduction gain and the preset global gain, and can represent the current remaining available bit resources.

[0154] Understandably, a dual safety constraint approach can be used. First, by determining whether the first total error reduction gain is less than or equal to the preset global gain, the problem of excessive model bit overhead and inability to deploy can be prevented. At the same time, by determining whether the first difference between the first total error reduction gain and the preset global gain is less than or equal to the first preset value, the timing of iteration termination can be accurately determined, ineffective iterations can be avoided, and the allocation of accuracy and computational cost can be balanced.

[0155] If so, then execute S305.

[0156] If not, then execute S306.

[0157] S305. Determine the first compression level within each preset compression parameter set as the target compression level corresponding to each weight matrix.

[0158] Understandably, when the global bit budget is extremely low and the initial first compression level already meets the convergence condition, the initial first compression level of all weight matrices can be directly determined as the final target compression level.

[0159] S306. In each weight matrix, determine the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix, and determine the target compression level corresponding to each of the multiple weight matrices based on the first weight matrix and the first preset compression parameter set.

[0160] Among them, the first weight matrix has the greatest benefit in reducing the first error. That is, the first weight matrix can refer to the weight matrix with the greatest benefit in reducing the first error in the current global range. Upgrading the first weight matrix has the highest cost performance.

[0161] Specifically, in the first preset compression parameter set, a second compression level is determined, wherein the compressed bits of the second compression level are greater than the compressed bits of the first compression level and less than the compressed bits of the other compression levels; based on the first weight matrix and the second compression level, a second error reduction gain of the first weight matrix is ​​determined; based on each of the first and second error reduction gains, a second total error reduction gain is determined; when the second difference between the second total error reduction gain and the preset global gain is less than or equal to the first preset value, and the second total error reduction gain is less than or equal to the preset global gain, the second compression level is determined as the target compression level of the first weight matrix, and the first compression level is determined as the target compression level of the other weight matrices; when the second difference between the second total error reduction gain and the preset global gain is greater than the first preset value, the compression level of the weight matrix with the highest error reduction gain is replaced with a third compression level whose compressed bits are higher than the previous compression level and less than the other compression levels, and the total error reduction gain is determined, until the difference between the total error reduction gain and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

[0162] In this case, the compression bits of the second compression level are greater than the compression bits of the first compression level and less than the compression bits of the other compression levels. It can be understood that the second compression level is the level above the current level and immediately adjacent to the current level, that is, the upgraded compression level.

[0163] The second error reduction gain can refer to the new error reduction gain obtained after upgrading the first weight matrix to the second compression level.

[0164] The second total error reduction benefit can refer to the global total error reduction benefit of the first model after upgrading the first weight matrix.

[0165] The third compression level can refer to the next higher bit level that continues upwards when the convergence condition is not met after upgrading to the second level.

[0166] Understandably, the complete loop logic is as follows: iterate through all weight matrices, select the matrix with the largest error reduction benefit as the first weight matrix (prioritizing the upgrade of the high-sensitivity attention matrix), select the second compression level that is higher in bits than the current level and closest to it, calculate the output error, bit overhead, and new error reduction benefit ROI after the upgrade of the first weight matrix, and re-accumulate to obtain the second total error reduction benefit; if the second difference between the second total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and the second total error reduction benefit is less than or equal to the preset global benefit, then the upgrade result is fixed, the remaining matrices remain at the initial first compression level, and the iteration ends; if the second difference between the second total error reduction benefit and the preset global benefit is greater than the first preset value, the weight matrix with the highest global error reduction benefit is selected again and upgraded to the next level, the third compression level, and so on, until the dual constraint conditions are met and all matrix levels are fixed.

[0167] In this way, since the benefits of each matrix are calculated from the first compression level as a unified starting point, and the hierarchical allocation is carried out based on the first total error reduction benefit, different weight matrices can obtain differentiated compression configurations under the global bit budget. Therefore, the balance of compression resource utilization can be improved, and the compressed model can better maintain the original computational accuracy.

[0168] exist Figure 3 In the illustrated embodiment, when it is necessary to determine the target compression level corresponding to multiple weight matrices, a first compression level can be determined from among the multiple compression levels within each preset compression parameter set, where the first compression level has the smallest compression bits. Based on the first compression level corresponding to each of the multiple weight matrices, the first error reduction benefit of each weight matrix is ​​determined. The first error reduction benefit can refer to the error reduction benefit that the weight matrix can generate by upgrading to an adjacent higher level based on the current level (i.e., the first compression level), which can represent the cost-effectiveness of the matrix's upgrade level. Based on the first error reduction benefit of each weight matrix, the first total error reduction benefit of the multiple weight matrices is determined. The first total error reduction gain can refer to the value obtained by summing the first error reduction gains of all weight matrices in the first model; it is determined whether the first difference between the first total error reduction gain and the preset global gain is less than or equal to the first preset value, and whether the first total error reduction gain is less than or equal to the preset global gain. If so, the first compression level in each preset compression parameter set is determined as the target compression level corresponding to each weight matrix; if not, in each weight matrix, the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix are determined, and the target compression level corresponding to multiple weight matrices is determined according to the first weight matrix and the first preset compression parameter set.

[0169] In this way, bit resources can be allocated differently according to the sensitivity of each weight matrix in the first model, prioritizing the allocation of more bits to sensitive matrices such as attention and output heads, minimizing quantization error without exceeding the hardware bit budget. At the same time, the iterative approach of greedy upgrade can reduce computational overhead, thereby effectively balancing compression ratio and inference performance, reducing the cost of deploying large models at the edge, and thus improving the performance of model compression.

[0170] Based on any of the above embodiments, the following, in conjunction with Figure 4 The process of compressing multiple weight matrices according to the target compression level to obtain the compressed target model is explained in detail.

[0171] Figure 4 This is a schematic diagram illustrating the compression process for any second weight matrix provided in an embodiment of this application. Please refer to... Figure 4 The method may include:

[0172] S401. Generate the initial codebook based on the second weight matrix.

[0173] The initial codebook can refer to the original codeword library that has not undergone gradient optimization and is only used for quickly initializing the quantization structure.

[0174] In one possible implementation, the initial codebook can be determined as follows: determine the first target compression level corresponding to the second weight matrix and the first input group within the first target compression level; divide the second weight matrix according to the first input group to obtain multiple vector samples; and perform clustering on the multiple vector samples to obtain the initial codebook.

[0175] Among them, vector samples can refer to the equal-length one-dimensional vectors obtained after dividing the input into groups according to a fixed input length. They are the basic samples for clustering and quantization.

[0176] Clustering processing can refer to processing using the K-Means clustering algorithm.

[0177] Specifically, the first target compression level of the second weight matrix to be compressed can be read, the first input group can be determined according to the first target compression level, each row parameter of the second weight matrix can be evenly divided according to the length of the first input group, all the divided vector samples can be summarized to form multiple vector samples, and all vector samples within the same output group range can share a set of codebooks.

[0178] Optionally, if the number of codebooks K=1, then K-Means clustering is directly performed on the sample set to generate S cluster centers as initial codebooks; if the number of codebooks K≥2, the same clustering method is used on the difference matrix, and after fitting the first set of codebooks, the residual matrix between the original samples and the reconstructed samples is calculated, and the residual matrix is ​​split and clustered again to generate K sets of initial codebooks in sequence.

[0179] S402. Determine the first index set based on the initial codebook.

[0180] The first set of indices can be used to indicate the location of multiple vector samples.

[0181] The first index set can refer to the set of integer subscripts generated for all vector samples. Each vector sample corresponds to a set of indices, which are used to select the corresponding codeword from multiple codebooks.

[0182] An index can refer to a non-continuous discrete parameter, which can only be an integer between 0 and S-1, where S is the number of cluster centers.

[0183] In one possible implementation, the initial codebook generated above can be fixed, and the codeword values ​​cannot be modified. Each vector sample is traversed, and beam search is used to traverse all index combinations. The index combination with the smallest error to the original vector is selected according to the addition reconstruction formula. The optimal indexes corresponding to all vector samples are summarized to form a complete first index set.

[0184] S403. Based on the first index set and the first target compression level, perform alternating optimization on the second weight matrix.

[0185] Alternating optimization processes may include fixing the first set of indices and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first set of indices.

[0186] Understandably, alternating optimization can refer to optimizing the codebook and scaling parameters with a fixed index, then re-searching for the optimal index with a fixed codebook, and repeating this process to solve the problem of continuous and discrete parameters not being synchronized.

[0187] In one possible implementation, the process of fixing the first index set and optimizing the initial codebook can refer to freezing the first index set of each vector sample, with the quantization weights determined only by the codebook and scaling parameters, inputting the calibration dataset for forward inference, obtaining the quantized Transformer block activation output and calculating the matching loss with the original model output; and updating all codebook codewords, scaling factors, and offset factors through the backpropagation algorithm to minimize the error under the current index conditions.

[0188] In one possible implementation, the process of fixing the optimized initial codebook and optimizing the first index set can refer to freezing the updated codebook, scaling and offset parameters, traversing each vector sample, re-searching for the optimal index combination in the beam search, and replacing the original old indexes, thereby further reducing the block output matching loss corresponding to the quantization weights.

[0189] S404. When the rate of change of the matching loss of the second weight matrix is ​​less than the second preset value, determine the optimized target index set and target codebook.

[0190] Matching loss can be used to indicate the difference between the second weight matrix before and after compression.

[0191] Specifically, the matching loss can refer to the mean squared error of the output activation per Transformer block, which can be used to indicate the difference between the quantized model activation output and the original dense model activation output.

[0192] The rate of change of the matching loss can be used to indicate the magnitude of the decrease in the matching loss between two adjacent iterations, and the rate of change of the matching loss can be used to indicate whether the first model has converged.

[0193] The second preset value can be a value set by the user, that is, a convergence threshold set in advance to determine the termination condition of the iteration.

[0194] The target index set and target codebook can refer to the quantization parameters that are finally finalized and permanently fixed after iterative convergence.

[0195] In one possible implementation, after each round of alternating optimization, the block output matching loss of the current round is recorded; the rate of change of the matching loss between two adjacent rounds of alternating optimization is calculated. If the rate of change of the matching loss is greater than a second preset value, it means that the first model still has room for optimization, and alternating optimization continues; if the rate of change of the matching loss is less than or equal to the second preset value, the training is determined to be converged, the iteration is stopped immediately, and the current codebook, scaling parameters and index set are fixed.

[0196] S405. Compress the second weight matrix based on the target index set and the target codebook.

[0197] This allows for lightweight compression of individual weight matrices, significantly reducing the storage size of the weight matrix and the memory usage for inference. At the same time, the inference logic is simple and highly compatible, requiring no modification to the inference engine, and can be directly deployed to edge devices and low-computing-power servers.

[0198] In this way, during compression, samples are first formed by grouping and segmentation, and then initial quantization centers are established through clustering algorithms. The stable compressed expression is approximated by alternating updates of the codebook and index, so that the difference between the final compressed result and the original weight matrix remains within the convergence threshold.

[0199] exist Figure 4 In the illustrated embodiment, when multiple weight matrices need to be compressed according to the target compression level to obtain the compressed target model, the process of compressing any second weight matrix may include: generating an initial codebook based on the second weight matrix, where the initial codebook may refer to the original codeword library without gradient optimization; determining a first index set based on the initial codebook, where the first index set may be used to indicate the position of multiple vector samples, and the first index set may refer to the set of integer subscripts generated for all vector samples, with each vector sample corresponding to a set of indices used to select the corresponding codeword from multiple codebooks; performing alternating optimization processing on the second weight matrix based on the first index set and the first target compression level, where alternating optimization processing may include fixing the first index set and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first index set; when the rate of change of the matching loss of the second weight matrix is ​​less than a second preset value, determining the optimized target index set and target codebook, where the matching loss may be used to indicate the difference between the second weight matrix before and after compression, and the target index set and target codebook may refer to the quantization parameters that are finally finalized and permanently fixed after iterative convergence; and compressing the second weight matrix based on the target index set and target codebook. In this way, the above method can be used to perform fine-grained vector quantization compression on a single weight matrix that has been allocated the target compression level. The weight block division and codebook initialization are completed based on the determined target compression level. The activation-oriented matching loss is used as the optimization objective. The differentiable codebook parameters and discrete index parameters are updated separately through an alternating optimization strategy. This effectively solves the problem that continuous and discrete parameters cannot be trained together in additive codebook quantization. Without increasing the deployment difficulty, it effectively reduces the model's GPU memory usage and storage cost, improves accuracy, and thus adapts to various terminals and lightweight deployment scenarios on the edge.

[0200] Based on any of the above embodiments, the following, in conjunction with Figure 5 After obtaining the compressed target model, the process of adapting the compression of the target model to specific target tasks is explained in detail.

[0201] Figure 5 This is a schematic diagram illustrating the task-oriented adaptation process for compressing the target model, as provided in an embodiment of this application. Please refer to... Figure 5 The method may include:

[0202] S501. In response to the processing request of the target task, generate a pre-trained codebook corresponding to any third weight matrix for processing the target task.

[0203] The target task can refer to a downstream-specific task that requires additional adaptation from the first model. It can include text classification, question answering, text summarization, machine translation, sentiment analysis, etc., which are different from the general generation tasks of the first model.

[0204] The pre-trained codebook is the same as the target codebook corresponding to the third weight matrix.

[0205] Understandably, a pre-trained codebook can refer to a copy of the target codebook corresponding to the third weight matrix. The pre-trained codebook is a general quantization codebook for the model, used only to carry general basic features, and does not have task-specific adaptation capabilities.

[0206] S502, Generate the first residual codebook.

[0207] The first residual codebook can refer to a dedicated initial residual codebook adapted to the target task, and the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook.

[0208] In one possible implementation, the codeword dimension of the first residual codebook can be set to be consistent with the codeword dimension of the pre-trained codebook, thereby ensuring vector format compatibility.

[0209] Set the hyperparameters of the first residual codebook so that the number of codebook sets in the first residual codebook is less than the number of codebook sets in the pre-trained codebook, and the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook, thereby structurally limiting the residual size.

[0210] Based on the target task-specific dataset, the deviation vector between the original dense model output and the quantized model output reconstructed solely from the pre-trained codebook is calculated. The deviation vector is then subjected to K-Means clustering to generate the first residual codebook of the initial state, which is used to fit the task-specific quantization deviation.

[0211] In this way, by adopting the first residual codebook design with the same source and lightweight, there is no need to modify the original quantization inference architecture and it can be adapted to the existing addition codebook decoding logic. The task error compensation is constructed with a very low number of parameters, which can not only make up for the accuracy defects of the general quantization model under specific tasks, but also keep the storage overhead of a single task at a low level.

[0212] S503. Based on the target task and the pre-trained codebook, perform alternating optimization on the first residual codebook to obtain a second residual codebook that is adapted to the target task.

[0213] Alternating optimization can refer to fixing the main intervention training codebook, alternately optimizing the continuous parameters of the residual codebook and the discrete index of the residual, and obtaining the optimal second residual codebook that fits the current task after iterative convergence.

[0214] Among them, continuous parameters can refer to codewords within the first residual codebook, residual-specific scaling factors, and offset factors, which can support gradient backpropagation updates. The discrete residual index can refer to the set of integer indices corresponding to the residual codebook, used to select residual codewords, and can be used when there is no gradient and direct differentiation is not possible.

[0215] In one possible implementation, a dedicated training dataset for the target task can be loaded, with the activation output of the original dense model as the ground truth label. The complete quantization weights consist of the backbone and residuals superimposed. The residual indices are fixed, and continuous parameters are optimized, i.e., all residual indices are frozen, and only the residual codebook codewords, residual scaling, and offset parameters are updated via backpropagation to minimize the block activation matching loss between the quantized model and the original model. The residual codebook is fixed, and discrete indices are optimized, i.e., the updated residual codewords and scaling parameters are frozen, and all residual index combinations are traversed through beam search to select the index with the lowest error to replace the old index. The optimization process is repeated cyclically until the rate of change of the matching loss in adjacent iterations is less than a preset threshold, at which point the iteration stops, and the parameters are fixed to obtain the second residual codebook.

[0216] S504. Based on the pre-trained codebook and the second residual codebook, determine the training codebook corresponding to the target task so that the third weight matrix can be compressed according to the training codebook when processing the target task.

[0217] In one possible implementation, the frozen main intervention training codebook is combined with the optimized second residual codebook to form a composite training codebook adapted to the current target task. During the inference phase, the weights are reconstructed based on the composite training codebook. During the inference process, the reconstructed weights are weighted and superimposed through the composite training codebook to complete the lightweight inference specific to the downstream task.

[0218] exist Figure 5 In the illustrated embodiment, after obtaining the compressed target model, when performing a specific target task on the target model, if task-specific adaptation of the target model compression is required, a pre-training codebook corresponding to any third weight matrix of the target task can be generated in response to the processing request of the target task. The target task may refer to a downstream specific task that requires additional adaptation of the first model. The pre-training codebook is the same as the target codebook corresponding to the third weight matrix. A first residual codebook is generated. The first residual codebook may refer to a specific initial residual codebook adapted to the target task, and the number of codewords in the first residual codebook is less than the number of codewords in the pre-training codebook. Based on the target task and the pre-training codebook, the first residual codebook is alternately optimized to obtain a second residual codebook adapted to the target task. Based on the pre-training codebook and the second residual codebook, the training codebook corresponding to the target task is determined so that the third weight matrix can be compressed based on the training codebook when processing the target task.

[0219] This approach provides the first model with lightweight multi-task adaptability. By freezing all main intervention training codebooks, the model's general basic capabilities are guaranteed. A small residual codebook is used to specifically compensate for the quantization biases specific to different downstream tasks, and a mature alternating optimization algorithm is reused to iteratively update residual parameters, thereby improving downstream task adaptability. Furthermore, since the pre-trained codebook is identical to the target codebook, it can directly reuse the basic expressive capabilities of the compressed target model. The smaller number of codewords in the first residual codebook compared to the pre-trained codebook concentrates the task adaptation process within a smaller compensation space. Simultaneously, training and storing only lightweight residuals is sufficient to adapt to multiple downstream tasks, improving not only inference speed but also adaptation accuracy for downstream scenarios, while reducing the training computational power, storage space, and scheduling costs of lightweight deployment on the edge.

[0220] Figure 6 This is a schematic diagram of the structure of a compression device for a model provided in an embodiment of this application. Please refer to [link / reference]. Figure 6 The model compression device 10 includes: a first determining module 11, an acquiring module 12, a second determining module 13, and a first compression module 14, wherein,

[0221] The first determining module 11 is used to determine multiple weight matrices corresponding to the first model to be compressed, and the multiple weight matrices are used to support the operation of the first model.

[0222] The acquisition module 12 is used to acquire a set of preset compression parameters corresponding to multiple weight matrices. The set of preset compression parameters includes multiple compression levels, each of which corresponds to a different compression bit. Each compression level includes the number of codebooks, the number of codewords, the input group, and the output group of the first codebook used to compress the weight matrix.

[0223] The second determining module 13 is used to determine the target compression level corresponding to the multiple weight matrices respectively, so that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain. The total error reduction gain is used to indicate the sum of the ratios of the output error of the multiple weight matrices to the bit overhead.

[0224] The first compression module 14 is used to compress multiple weight matrices according to the target compression level to obtain the compressed target model.

[0225] The compression device for a model provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0226] In one possible design, the second determining module 13 is specifically used for,

[0227] Among the multiple compression levels within each preset compression parameter set, the first compression level is determined, and the first compression level has the smallest compression bits.

[0228] Based on the first compression level corresponding to each of the multiple weight matrices, determine the first error reduction benefit of each weight matrix;

[0229] Based on the first error reduction gain of each weight matrix, determine the first total error reduction gain of multiple weight matrices;

[0230] Based on the first total error reduction gain of multiple weight matrices, the target compression level corresponding to each of the multiple weight matrices is determined.

[0231] In one possible design, the second determining module 13 is specifically used for,

[0232] Determine whether the first difference between the first total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and whether the first total error reduction benefit is less than or equal to the preset global benefit;

[0233] If so, then the first compression level within each preset compression parameter set is determined as the target compression level corresponding to each weight matrix;

[0234] If not, then in each weight matrix, determine the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix, and the first error reduction benefit of the first weight matrix is ​​maximized;

[0235] Based on the first weight matrix and the first preset compression parameter set, the target compression levels corresponding to the multiple weight matrices are determined respectively.

[0236] In one possible design, the second determining module 13 is specifically used for,

[0237] In the first preset compression parameter set, a second compression level is determined, wherein the compression bits of the second compression level are greater than the compression bits of the first compression level and less than the compression bits of the other compression levels;

[0238] The second error reduction gain of the first weight matrix is ​​determined based on the first weight matrix and the second compression level.

[0239] The second total error reduction benefit is determined based on each of the first and second error reduction benefits.

[0240] When the second difference between the second total error reduction gain and the preset global gain is less than or equal to the first preset value, and the second total error reduction gain is less than or equal to the preset global gain, the second compression level is determined as the target compression level of the first weight matrix, and the first compression level is determined as the target compression level of the remaining weight matrices.

[0241] When the second difference between the second total error reduction gain and the preset global gain is greater than the first preset value, the compression level of the weight matrix with the highest error reduction gain is replaced with the third compression level whose compression bits are higher than the previous compression level and lower than the other compression levels, and the total error reduction gain is determined until the difference between the total error reduction gain and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

[0242] In one possible design, the first compression module 14 is specifically used for,

[0243] Generate the initial codebook based on the second weight matrix;

[0244] Based on the initial codebook, determine the first index set, which is used to indicate the positions of multiple vector samples;

[0245] Based on the first target compression level corresponding to the first index set and the second weight matrix, the second weight matrix is ​​subjected to alternating optimization processing. The alternating optimization processing includes fixing the first index set and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first index set.

[0246] When the rate of change of the matching loss of the second weight matrix is ​​less than the second preset value, the optimized target index set and target codebook are determined. The matching loss is used to indicate the difference between the second weight matrix before and after compression.

[0247] The second weight matrix is ​​compressed based on the target index set and the target codebook.

[0248] The compression device for a model provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0249] Figure 7 This is a schematic diagram of the structure of a compression device for another model provided in an embodiment of this application. Figure 6 Based on this, please refer to Figure 7 The compression device 10 of the model also includes a second compression module 15.

[0250] The second compression module 15 is used to, in response to the processing request of the target task, generate a pre-training codebook corresponding to any third weight matrix of the target task, wherein the pre-training codebook is the same as the target codebook corresponding to the third weight matrix.

[0251] Generate the first residual codebook, the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook;

[0252] Based on the target task and the pre-trained codebook, the first residual codebook is alternately optimized to obtain a second residual codebook that is adapted to the target task.

[0253] Based on the pre-trained codebook and the second residual codebook, the training codebook corresponding to the target task is determined so that the third weight matrix can be compressed according to the training codebook when processing the target task.

[0254] The compression device for a model provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0255] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 20 may include: a transceiver 21, a processor 22, and a memory 23.

[0256] Processor 22 executes computer execution instructions stored in memory, causing processor 22 to perform the scheme in the above embodiments. Processor 22 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0257] The memory 23 is connected to the processor 22 via the system bus and completes communication between them. The memory 23 is used to store computer program instructions.

[0258] Transceiver 21 can be used to obtain the task to be run and the configuration information of the task to be run.

[0259] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0260] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0261] This application also provides a chip for executing instructions, which is used to execute the compression method of the model in the above embodiments.

[0262] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the compression method of the model described in the above embodiments.

[0263] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the compression method of the model in the above embodiments.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0265] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0266] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0267] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0268] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0269] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0270] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0271] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0272] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0273] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0274] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for compressing a model, characterized in that, include: Determine multiple weight matrices corresponding to the first model to be compressed, the multiple weight matrices being used to support the operation of the first model; Obtain a preset compression parameter set corresponding to each of the multiple weight matrices. The preset compression parameter set includes multiple compression levels, each of which corresponds to a different compression bit. Each compression level includes the number of codebooks, the number of codewords, the input group, and the output group of the first codebook used to compress the weight matrices. The target compression level corresponding to each of the multiple weight matrices is determined such that when the multiple weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the multiple weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain. The total error reduction gain is used to indicate the sum of the ratios of the output error to the bit overhead of the multiple weight matrices. The multiple weight matrices are compressed according to the target compression level to obtain the compressed target model; After compressing the multiple weight matrices according to the target compression level to obtain the compressed target model, the process further includes: In response to the processing request of the target task, for any third weight matrix of the target task, a pre-trained codebook corresponding to the third weight matrix is ​​generated. The pre-trained codebook is the same as the target codebook corresponding to the third weight matrix. The target task is a downstream-specific task that is additionally adapted to the first model. The target task includes text classification task, question answering task, text summarization task, machine translation task, sentiment analysis task, and the target task is different from the general generation task of the first model. Generate a first residual codebook, wherein the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook; Based on the target task and the pre-trained codebook, the first residual codebook is alternately optimized to obtain a second residual codebook adapted to the target task; Based on the pre-trained codebook and the second residual codebook, a training codebook corresponding to the target task is determined so that the third weight matrix can be compressed according to the training codebook when processing the target task.

2. The method according to claim 1, characterized in that, Determining the target compression level corresponding to the plurality of weight matrices includes: Among the multiple compression levels within each preset compression parameter set, a first compression level is determined, wherein the first compression level has the smallest compression bits. Based on the first compression level corresponding to the multiple weight matrices, determine the first error reduction benefit of each weight matrix; Based on the first error reduction gain of each weight matrix, determine the first total error reduction gain of the multiple weight matrices; Based on the first total error reduction gain of the multiple weight matrices, the target compression level corresponding to each of the multiple weight matrices is determined.

3. The method according to claim 2, characterized in that, Based on the first total error reduction gain of the plurality of weight matrices, the target compression level corresponding to each of the plurality of weight matrices is determined, including: Determine whether the first difference between the first total error reduction benefit and the preset global benefit is less than or equal to the first preset value, and whether the first total error reduction benefit is less than or equal to the preset global benefit; If so, then the first compression level within each preset compression parameter set is determined to be the target compression level corresponding to each weight matrix; If not, then among the weight matrices, determine the first weight matrix and the first preset compression parameter set corresponding to the first weight matrix, where the first error reduction benefit of the first weight matrix is ​​the greatest; Based on the first weight matrix and the first preset compression parameter set, the target compression level corresponding to each of the multiple weight matrices is determined.

4. The method according to claim 3, characterized in that, Based on the first weight matrix and the first preset compression parameter set, the target compression levels corresponding to the plurality of weight matrices are determined, including: In the first preset compression parameter set, a second compression level is determined, wherein the compression bit of the second compression level is greater than the compression bit of the first compression level and less than the compression bit of the other compression levels; Based on the first weight matrix and the second compression level, determine the second error reduction benefit of the first weight matrix; The second total error reduction benefit is determined based on each of the first error reduction benefits and the second error reduction benefits; When the second difference between the second total error reduction gain and the preset global gain is less than or equal to the first preset value, and the second total error reduction gain is less than or equal to the preset global gain, the second compression level is determined as the target compression level of the first weight matrix, and the first compression level is determined as the target compression level of the remaining weight matrices. When the second difference between the second total error reduction gain and the preset global gain is greater than the first preset value, the compression level of the weight matrix with the highest error reduction gain is replaced with a third compression level whose compression bits are higher than the previous compression level and lower than the other compression levels, and the total error reduction gain is determined until the difference between the total error reduction gain and the preset global gain is less than or equal to the first preset value, and the total error reduction gain is less than or equal to the preset global gain.

5. The method according to any one of claims 1-4, characterized in that, For any given second weight matrix; compress the plurality of weight matrices according to the target compression level to obtain the compressed target model, including: Generate the initial codebook based on the second weight matrix; Based on the initial codebook, a first index set is determined, which is used to indicate the positions of multiple vector samples; Based on the first target compression level corresponding to the first index set and the second weight matrix, the second weight matrix is ​​subjected to alternating optimization processing. The alternating optimization processing includes fixing the first index set and optimizing the initial codebook, and fixing the optimized initial codebook and optimizing the first index set. When the rate of change of the matching loss of the second weight matrix is ​​less than the second preset value, the optimized target index set and target codebook are determined, wherein the matching loss is used to indicate the difference between the second weight matrix before and after compression. The second weight matrix is ​​compressed based on the target index set and the target codebook.

6. A compression device for a model, characterized in that, include: The system comprises a first determining module, an acquiring module, a second determining module, and a first compression module, wherein, The first determining module is used to determine multiple weight matrices corresponding to the first model to be compressed, the multiple weight matrices being used to support the operation of the first model; The acquisition module is used to acquire a preset compression parameter set corresponding to the plurality of weight matrices respectively. The preset compression parameter set includes a plurality of compression levels, and the plurality of compression levels correspond to different compression bits respectively. The compression level includes the number of codebooks, the number of codewords, the input group and the output group of the first codebook used to compress the weight matrix. The second determining module is used to determine the target compression level corresponding to the plurality of weight matrices respectively, so that when the plurality of weight matrices are compressed according to the corresponding target compression level, the difference between the total error reduction gain of the plurality of weight matrices and the preset global gain is less than or equal to a first preset value, and the total error reduction gain is less than or equal to the preset global gain, wherein the total error reduction gain is used to indicate the sum of the ratios of the output error to the bit overhead of the plurality of weight matrices; The first compression module is used to compress the plurality of weight matrices according to the target compression level to obtain the compressed target model; The first compression module is further configured to, in response to a processing request for a target task, generate a pre-trained codebook corresponding to any third weight matrix for processing the target task, wherein the pre-trained codebook is the same as the target codebook corresponding to the third weight matrix, the target task being a downstream-specific task additionally adapted by the first model, the target task including text classification, question answering, text summarizing, machine translation, and sentiment analysis, and the target task being different from the general generation task of the first model; generate a first residual codebook, wherein the number of codewords in the first residual codebook is less than the number of codewords in the pre-trained codebook; perform alternating optimization processing on the first residual codebook according to the target task and the pre-trained codebook to obtain a second residual codebook adapted to the target task; and determine a training codebook corresponding to the target task according to the pre-trained codebook and the second residual codebook, so that the third weight matrix is ​​compressed according to the training codebook when processing the target task.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images