Model compression method, apparatus and system

Abstract

The present application relates to the technical field of computers, and provides a model compression method, apparatus and system. The method comprises: performing a quantization operation on an artificial intelligence model on the basis of quantization operation parameters, so as to convert the format of model parameters of the artificial intelligence model; and generating sparsification operation parameters on the basis of the quantized model parameters of the artificial intelligence model, and performing a sparsification operation on the quantized model parameters of the artificial intelligence model on the basis of the sparsification operation parameters, so as to obtain a compressed model. In this way, a correlation between the sparsification operation and the quantization operation is established, enabling both quantization and sparsification of the model, thereby avoiding a conflict between quantization and sparsification, and ensuring that model performance of a compressed model is good.

Description

Model compression method, device and system

[0001] This application claims priority to the Chinese patent application No. 202411119626.4, filed on August 13, 2024, entitled "Model compression method, device and system", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, in particular to a model compression method, device and system. BACKGROUND

[0003] Computing devices can run artificial intelligence models (referred to as models) to complete various data processing tasks. With the development of computer technology, the size of the model can be large, and large-scale models can process complex data processing tasks. However, the larger the size of the model, the more storage resources and computing resources are required to store and run the model.

[0004] Model compression can reduce the amount of parameter data of the model, and reduce the storage resource consumption and computing resource consumption of the model. For example, model compression can quantize model parameters to reduce the data width of model parameters (also referred to as the bit width of parameters). Or, model compression can sparsify model parameters, for example, pruning model parameters that are less important to model performance. However, respectively quantizing or sparsifying the model, the results of quantization and sparsification will conflict, for example, a certain parameter needs to be pruned when sparsifying, but quantization changes the importance of the model parameter to the model performance, the importance of the parameter is higher after quantization, and pruning the parameter leads to loss of model performance. SUMMARY

[0005] The present application provides a model compression method, device and system, which can solve the problem of difficult to better reduce the resource consumption of the model while maintaining the performance of the model.

[0006] In a first aspect, a model compression method is provided, the method comprising performing a quantization operation on an artificial intelligence model according to a quantization operation parameter, the quantization operation being used to convert the format of a model parameter of the artificial intelligence model, and further generating a sparsification operation parameter according to the model parameter of the artificial intelligence model after quantization, and performing a sparsification operation on the model parameter of the artificial intelligence model after quantization according to the sparsification operation parameter, to obtain a compressed model.

[0007] Therefore, the model compression method provided by the embodiments of the present application generates the sparsification operation parameter according to the quantized model parameter, that is, generates the sparsification operation parameter by referring to the influence of the quantization operation on the model parameter. The sparsification operation is performed on the basis of the quantized model parameter according to the sparsification operation parameter, that is, the sparsification operation is associated with the quantization operation, the model is quantized and sparsified, and the conflict between quantization and sparsification is avoided (for example, the important parameters in the quantized model parameter are pruned during sparsification), so that the model performance of the compressed model is ensured.

[0008] In some implementations, the model parameter includes a plurality of parameters, and the method generates the sparsification operation parameter according to the importance of the plurality of parameters in the quantized model parameter.

[0009] The importance of the parameter (for example, the weight) can reflect the importance of the parameter to the model performance. The importance of the parameter in the quantized model parameter of the model may change compared with the original model parameter. Therefore, the sparsification operation parameter is generated according to the importance of the parameter in the quantized model parameter, and when the quantized model parameter is sparsified according to the sparsification operation parameter, the quantized parameter with low importance can be pruned more accurately.

[0010] In other implementations, the method further includes updating the quantization operation parameter according to the results of processing the sample data by the compressed model and the artificial intelligence model respectively.

[0011] The results of processing the sample data by the compressed model and the artificial intelligence model respectively can reflect the performance of the compressed model. Therefore, the quantization operation parameter can be updated according to the performance of the compressed model, and the model can be quantized more accurately according to the updated quantization operation parameter, and the quantization effect is better.

[0012] In other implementations, the method includes processing the sample data by the compressed model to obtain a first processing result of the sample data, processing the sample data by the artificial intelligence model to obtain a second processing result of the sample data, and updating the quantization operation parameter according to the difference between the first processing result and the second processing result.

[0013] The difference between the first processing result and the second processing result can accurately reflect the difference between the performance of the compressed model and the performance of the original model. Therefore, the quantization operation parameter can be updated more accurately, and the effect of quantizing the model according to the updated quantization operation parameter is better.

[0014] In another implementation, the method further includes quantizing the compressed model according to the updated quantization operation parameter, and updating the sparsification operation parameter according to the quantized model parameter of the compressed model.

[0015] After the quantization operation is performed on the compressed model again according to the updated quantization operation parameter, the model parameter further changes. Based on this, the sparsification operation parameter is updated with reference to the further change of the model parameter of the reference model, and the model can be sparsified more finely according to the updated sparsification operation parameter, and the sparsification effect is better.

[0016] In some other implementations, the quantization operation parameter includes an affine transformation parameter of the model parameter of the artificial intelligence model.

[0017] In some other implementations, performing the quantization operation on the artificial intelligence model according to the quantization operation parameter includes performing a linear transformation on the model parameter of the artificial intelligence model according to the affine transformation parameter, and performing a format transformation on the model parameter of the artificial intelligence model after the linear transformation.

[0018] The affine transformation parameter is used to perform a linear transformation on the model parameter, and can adjust the size and distribution of the model parameter during model compression (for example, adjust the model parameter range and parameter value during each compression process). The format transformation is performed on the model parameter after the linear transformation according to the affine transformation parameter, which can effectively control the parameter change during the model quantization and the sparsification after the quantization, and better guarantee the model performance of the compressed model.

[0019] In some other implementations, the sparsification operation parameter is used to indicate the to-be-pruned parameter in the quantized model parameter, and performing the sparsification operation on the quantized model parameter of the artificial intelligence model according to the sparsification operation parameter includes pruning the to-be-pruned parameter in the quantized model parameter of the artificial intelligence model according to the sparsification operation parameter.

[0020] In this way, the sparsification operation parameter accurately indicating the to-be-pruned parameter in the quantized model parameter is generated with reference to the influence of the quantization operation on the model parameter, the quantized model can be accurately pruned according to the sparsification operation parameter, the effect of the sparsification operation is better, and the model performance of the compressed model is better guaranteed.

[0021] In some other implementations, the model parameter includes a parameter of a linear layer of the artificial intelligence model, the quantization operation parameter includes a quantization operation parameter of the parameter of the linear layer, and the sparsification operation parameter includes a sparsification operation parameter of the parameter of the linear layer.

[0022] The parameters of different linear layers of the model are different. By performing separate compression processing on each linear layer, the quantization operation parameter and the sparsification operation parameter are applied to each linear layer respectively, the model compression is performed according to the parameter characteristics of different linear layers, and the parameter compression effect of each linear layer is better, which better guarantees the model performance of the compressed model.

[0023] In some implementations, the compressed model is used to process at least one of text, image, and audio data to obtain a processing result.

[0024] In a second aspect, a model compression apparatus is provided. The apparatus includes a quantization operation module and a sparsification operation module.

[0025] The quantization operation module is configured to perform a quantization operation on the artificial intelligence model according to a quantization operation parameter, the quantization operation being configured to convert a format of model parameters of the artificial intelligence model. The sparsification operation module is configured to generate a sparsification operation parameter according to the quantized model parameters of the artificial intelligence model, and perform a sparsification operation on the quantized model parameters of the artificial intelligence model according to the sparsification operation parameter to obtain a compressed model.

[0026] In some implementations, the model parameters include a plurality of parameters, and the sparsification operation module is further configured to generate the sparsification operation parameter according to importance of the plurality of parameters in the quantized model parameters.

[0027] In some implementations, the quantization operation module is further configured to update the quantization operation parameter according to a result of processing sample data by the compressed model and the artificial intelligence model, respectively.

[0028] In some implementations, the quantization operation module is further configured to obtain a first processing result of the sample data by processing the sample data according to the compressed model, obtain a second processing result of the sample data by processing the sample data according to the artificial intelligence model, and update the quantization operation parameter according to a difference between the first processing result and the second processing result.

[0029] In some implementations, the quantization operation module is further configured to perform the quantization operation on the compressed model according to the updated quantization operation parameter.

[0030] In some implementations, the sparsification operation module is further configured to update the sparsification operation parameter according to the quantized model parameters of the compressed model.

[0031] In some implementations, the quantization operation parameter includes an affine transformation parameter of the model parameters of the artificial intelligence model.

[0032] In some implementations, the quantization operation module is further configured to perform a linear transformation on the model parameters of the artificial intelligence model according to the affine transformation parameter, and perform a format conversion on the model parameters of the artificial intelligence model after the linear transformation.

[0033] In some implementations, the sparsification operation parameter is configured to indicate a to-be-pruned parameter in the quantized model parameters, and the sparsification operation module is further configured to prune the to-be-pruned parameter in the quantized model parameters of the artificial intelligence model according to the sparsification operation parameter.

[0034] In some implementations, the model parameters comprise parameters of a linear layer of the artificial intelligence model, the quantization operation parameters comprise quantization operation parameters of the parameters of the linear layer, and the sparsification operation parameters comprise sparsification operation parameters of the parameters of the linear layer.

[0035] In some implementations, the compressed model is used to process at least one of text, image, and audio to obtain a processing result.

[0036] In a third aspect, a model compression system is provided, which comprises at least one apparatus configured to perform the model compression method according to the first aspect.

[0037] In a fourth aspect, a computing device is provided, which comprises a processor and a memory; wherein the processor is configured to execute instructions stored in the memory to cause the computing device to perform the model compression method according to the first aspect.

[0038] In a fifth aspect, a computing device cluster is provided, which comprises at least one computing device, wherein each computing device comprises a processor and a memory; and the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the model compression method according to the first aspect.

[0039] In a sixth aspect, a computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the model compression method according to the first aspect is provided.

[0040] In a seventh aspect, a computer-readable storage medium comprising computer program instructions which, when executed by a computing device, cause the computing device to perform the model compression method according to the first aspect is provided.

[0041] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners.

[0042] In the following description, more specific details about the implementation manners of the aspects described above are included. BRIEF DESCRIPTION OF DRAWINGS

[0043] FIG. 1 is an architecture schematic diagram of a model compression system provided by an embodiment of the present application;

[0044] FIG. 2 is a flow schematic diagram of a model compression method provided by an embodiment of the present application;

[0045] FIG. 3 is a flow schematic diagram of a model compression method provided by an embodiment of the present application;

[0046] FIG. 4 is a flow schematic diagram of a model compression method provided by an embodiment of the present application;

[0047] FIG. 5 is a structural schematic diagram of a model compression device provided by an embodiment of the present application;

[0048] FIG. 6 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0049] FIG. 7 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] First, the terms or concepts involved in the present application are introduced.

[0051] The artificial intelligence model can be referred to as a model, which can refer to a model obtained by abstractly modeling things in the real world through mathematics and algorithms. The artificial intelligence model includes various types, which can include, for example, language models, image recognition models, speech recognition models, and the like.

[0052] The artificial intelligence model can be used to process various data such as text, images, and audio, to obtain a processing result of the data.

[0053] For example, the language model refers to a model obtained by abstractly modeling language through mathematics according to language objective facts, and can be used to understand and generate natural language.

[0054] As an example, the language model can be used to process text to obtain an inference result corresponding to the text (for example, the text includes a question, a requirement, etc., and the language model obtains a reply text of the text, such as an answer to the question, a feedback result of the requirement, etc.). As another example, the language model can be used to process image, audio / speech, etc. data to generate text corresponding to the data (for example, text description of the image, text description of the audio / speech, etc.).

[0055] The language model can include various types, which can include, for example, a language model constructed by a neural network such as a Transformer.

[0056] As another example, the image recognition model can be used to analyze an image to obtain an inference result of the image (for example, an image classification result, image feature information, etc.).

[0057] As another example, the speech recognition model can be used to analyze audio / speech to obtain an inference result of the audio / speech (for example, a speech classification result, speech feature information, etc.).

[0058] A large-scale model includes a large number of parameters and a large amount of parameter data. The model parameters can include multiple weights and other parameters of the model. For example, a large language model has a more complex or deeper network structure and a larger number of parameters than a general language model. As an example, a large language model can include a Generative Pre-trained Transformer (GPT), a Bidirectional Encoder Representation from Transformers (BERT), and the like.

[0059] Computing devices running large-scale models can better complete complex data processing tasks. For example, a large language model can complete various complex Natural Language Processing (NLP) tasks such as machine translation, text summarization, and question answering systems.

[0060] A model needs to be stored in a storage medium for retrieval and use when needed. The larger the model, the more storage resources (such as the capacity of the storage medium) are required to store the model.

[0061] Running a model (such as model training and model inference) by a computing device requires consumption of computing resources. The larger the model, the greater the amount of computation required to run the model, and thus the more computing resources (such as the computing power, memory, and the like of the computing device) are required to run the model.

[0062] Model compression is a technique for reducing the amount of parameter data of a model, and there are various implementations, such as model quantization and model sparsification.

[0063] Model quantization is used to change the format of the model parameters, thereby reducing the data width of the model parameters.

[0064] As an example, the format of the model parameters is Single-precision floating-point (FP32). Model quantization converts the format of the model parameters from FP32 to 8-bit integer (INT8). Alternatively, model quantization converts the format of the model parameters from FP32 to another data format with lower precision than FP32, such as Half-precision floating-point (FP16), 16-bit integer (INT16), and the like.

[0065] Wherein, FP32 refers to a floating point number represented by 32 bits of binary. FP32 can include a sign bit, an exponent bit, and a fraction bit.

[0066] The length of the sign bit is 1 bit. The value of the sign bit is 0 for a positive number and 1 for a negative number.

[0067] The length of the exponent bit of FP32 is 8 bits, and the bias value is 127. The exponent bit represents the integer part in FP32. In addition, in order to enable the exponent bit to represent negative numbers, a bias value is introduced. The bias value is a fixed number, and the bias value is added to the actual exponent.

[0068] The length of the fraction bit of FP32 is 23 bits. The fraction bit represents the decimal part in FP32.

[0069] FP16 refers to a floating point number represented by 16 bits of binary. FP16 and FP32 are similar and can also include a sign bit, an exponent bit, and a fraction bit. The length of the exponent bit of FP16 is 5 bits, the bias value is 15, and the length of the fraction bit is 10 bits.

[0070] INT8 refers to an integer represented by 8 bits of binary. INT16 refers to an integer represented by 16 bits of binary.

[0071] The higher the precision, the larger the data volume, and the more storage space occupied. The precision of FP32 is higher than that of FP16, INT16, and INT8. FP32 occupies more storage space than FP16, INT16, and INT8. Using FP32 to represent data ensures calculation accuracy, but occupies more storage space. Using FP16, INT16, and INT8 to represent data can reduce the occupied storage space and improve the calculation efficiency.

[0072] Model sparsification is used to prune model parameters, thereby reducing the number of model parameters.

[0073] As an example, the model includes a plurality of network layers, the plurality of network layers include a plurality of (e.g., tens of thousands of) parameters, and the model sparsification prunes part of the network layers or part of the parameters of the part of the network layers. Wherein, the pruning can be setting the model parameters to be pruned to 0 or removing the model parameters to be pruned, etc.

[0074] In addition, the pruned parameters can be parameters with a large sparsity priority. In some embodiments, the sparsity priority of the model parameters can be determined according to the values of the model parameters. For example, the smaller the value of the model parameter, the greater the sparsity priority. The sparsity priority of the model parameter can reflect the importance of the model parameter to the model performance (the model performance can include the accuracy of model inference), and the greater the sparsity priority, the lower the importance of the model parameter to the model performance.

[0075] In some embodiments, the storage device or the available resources of the computing device are less (for example, mobile terminals, low-power devices, single computing devices, etc.), and it is difficult to store and run large-scale models. Model compression can solve this problem.

[0076] For example, after quantization, the model parameter precision of the model is reduced, the parameter data volume of the model is reduced, the storage resources consumed by storing the model are reduced, and the computational complexity when running the model is reduced, the computing resources consumed by running the model are reduced, and the computing speed is also faster.

[0077] For another example, after sparsification, the number of model parameters of the model is reduced, the storage resources consumed by storing the model are reduced, and the amount of computation when running the model is reduced, the computing resources consumed by running the model are reduced, and the computing speed is also faster.

[0078] In some application scenarios, it is necessary to greatly reduce the parameter data volume of the model. For example, the parameter data volume of the model is too large, and the storage resources of the storage device or the computing resources of the computing device are simultaneously less. Separate model quantization or model sparsification cannot greatly reduce the parameter data volume of the model.

[0079] In addition, the results of quantization and sparsification will conflict. For example, quantization changes the importance of model parameters to model performance. A parameter is determined to be a pruning parameter in sparsification according to the original value of the parameter, but the importance of the parameter is higher after quantization, and the parameter should not be pruned. Therefore, when the model is quantized and sparsified at the same time, the performance of the model may be lost (for example, the accuracy of model inference decreases).

[0080] The embodiments of the present application provide a model optimization method, which comprises: performing a quantization operation on an artificial intelligence model according to a quantization operation parameter, to convert the format of the model parameters of the artificial intelligence model; and generating a sparsification operation parameter according to the model parameters of the artificial intelligence model after quantization, performing a sparsification operation on the model parameters of the artificial intelligence model after quantization according to the sparsification operation parameter, to obtain a compressed model.

[0081] Through the model compression method provided by the embodiments of the present application, the sparsification operation parameter is generated according to the model parameters after quantization, that is, the influence of the quantization operation on the model parameters is referred to generate the sparsification operation parameter. The sparsification operation is performed on the basis of the model parameters after quantization according to the sparsification operation parameter, that is, the sparsification operation and the quantization operation are associated, the model is quantized and sparsified, the conflict between quantization and sparsification is avoided (for example, the important parameters in the quantized model are not pruned in sparsification), and the performance of the compressed model is guaranteed.

[0082] The model compression method provided in the embodiments of the present application can be applied to compress various models, such as language models, image recognition models, speech recognition models, and the like. The embodiments of the present application mainly take a language model (such as a large language model) as an example for description.

[0083] In some implementations, the model compression method provided in the embodiments of the present application can be applied to compress a trained model.

[0084] The system architecture provided in the embodiments of the present application is introduced below.

[0085] FIG. 1 is a schematic diagram of the architecture of a model compression system provided in the embodiments of the present application. As shown in FIG. 1, the model compression system 100 can include a compression processing device 110, a storage medium 120, and an execution device 130.

[0086] The compression processing device 110 is configured to compress a model. Specifically, the compression processing device 110 can be configured to execute the model compression method provided in the embodiments of the present application to compress a target model and obtain a compressed model. In the embodiments of the present application, the artificial intelligence model to be compressed is referred to as a target model. The target model can be a trained model and can be used to process any one or more of text, images, and audio to obtain a processing result.

[0087] For example, the compression processing device 110 can be configured to perform a quantization operation on the target model according to quantization operation parameters to convert the format of the model parameters of the target model. In addition, the compression processing device 110 can also be configured to generate sparsification operation parameters according to the quantized model parameters of the target model, and perform a sparsification operation on the quantized model parameters of the target model according to the sparsification operation parameters to obtain the compressed model. The specific content of the model compression method performed by the compression processing device 110 can be found in FIG. 2 and FIG. 3 and the related descriptions thereof.

[0088] In some embodiments, the target model can be stored in an external storage device, and the compression processing device 110 can obtain the target model from the storage device and compress the target model. Alternatively, the target model can be stored in the storage medium of the compression processing device 110, and the compression processing device 110 can directly compress the target model.

[0089] The compression processing device 110 can include a server or a cloud, or can include a terminal such as a tablet computer, a notebook computer, and the like. In some embodiments, the compression processing device 110 can be a single processing device, or can be a device cluster composed of multiple processing devices.

[0090] The compressed model obtained by the compression processing device 110 performing compression processing on the target model can be stored in the storage medium 120, so as to obtain or use the compressed model when needed. The storage medium for storing the compressed model, such as the storage medium 120, can also be referred to as a target storage medium.

[0091] The storage medium 120 can include a hard disk, a magnetic tape, and various storage media that can be used for data storage.

[0092] The compressed model obtained by the compression processing device 110 performing compression processing on the target model can be applied to different systems or devices, such as the execution device 130.

[0093] In some embodiments, the execution device 130 can read or call the compressed model in the storage medium 120, so as to apply the compressed model on the execution device 130.

[0094] The execution device 130 can include a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an AR / VR, a vehicle-mounted terminal, and the like, and can also include a server or a cloud, and the like.

[0095] As an example, the execution device 130 in FIG. 1 is configured with an I / O interface 132 for data interaction with an external device. A user can input data to the I / O interface 132 through a client device 140, and the input data in the embodiments of the present application can include text, image, audio, and the like. The computing module 131 of the execution device 130 is configured to input the input data received by the I / O interface 132 into the compressed model for processing, to obtain a processing result. The I / O interface 132 can also output the processing result to the client device 140, so as to provide it to the user.

[0096] Further combined with an application scenario, as an example, the execution device 130 can be a mobile terminal such as a smart phone, or an embedded device such as an Internet of Things device. The resources of the mobile terminal or the embedded device cannot meet the needs of storing the target model and running the target model. After the compression processing device 110 performs compression processing on the target model by using the model compression method provided in the embodiments of the present application to obtain the compressed model, the parameter data amount of the compressed model is reduced, and the compressed model can be stored in the storage medium of the mobile terminal and the embedded device (for example, the storage medium 120 is arranged in the execution device 130), and the mobile terminal or the embedded device can run the compressed model to process a data processing task, such as processing a speech recognition task, an image classification task, and the like.

[0097] As a further example, the execution device 130 can be a vehicle terminal of an autonomous vehicle. The computing resource of the vehicle terminal cannot meet the requirement of running the target model. After the compression processing device 110 performs the model compression method provided in the embodiments of the present application to compress the target model to obtain the compressed model, the compressed model can be stored in the storage medium 120. In addition, the vehicle terminal can call the compressed model stored in the storage medium 120 and run the compressed model, input the external environment information (for example, road, pedestrian, obstacle, etc.) obtained by the sensor of the vehicle in the driving process into the compressed model, and obtain the next action trajectory of the vehicle.

[0098] FIG. 1 is only a schematic diagram of a model compression system architecture provided in the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.

[0099] For example, the storage medium 120 in FIG. 1 is an external component of the compression processing device 110, and in some cases, the storage medium 120 can also be arranged in the compression processing device 110.

[0100] For another example, the storage medium 120 in FIG. 1 is an external component of the execution device 130, and in some cases, the storage medium 120 can also be arranged in the execution device 130.

[0101] For another example, the compression processing device 110 and the execution device 130 in FIG. 1 are different devices, and in some cases, the compression processing device 110 and the execution device 130 can also be one device.

[0102] The model compression method provided in the embodiments of the present application will be described in detail below. The model compression method provided in the embodiments of the present application can be implemented on the model compression system 100 shown in FIG. 1.

[0103] FIG. 2 is a flowchart of a model compression method provided in the embodiments of the present application. The model compression method shown in FIG. 2 can be executed by a computing device. For example, it can be executed by the compression processing device 110 in the model compression system 100.

[0104] As shown in FIG. 2, the method includes the following steps.

[0105] Step 210: performing a quantization operation on the artificial intelligence model according to a quantization operation parameter.

[0106] The quantization operation parameter refers to an operation parameter required for the quantization operation.

[0107] As described above, the quantization operation is used to convert the format of the model parameters of the artificial intelligence model. The quantization operation can reduce the bit width of the model parameters.

[0108] For example only, a quantization operation can convert the format of a model parameter from FP32 to INT8, in which case the bit width of the model parameter is reduced from 32 bits to 8 bits. Or, a quantization operation can convert the format of a model parameter from FP16 to INT8, in which case the bit width of the model parameter is reduced from 16 bits to 8 bits. A quantization operation can also be used to implement other types of format conversion of a model parameter, which can be configured as needed.

[0109] For example, FIG. 2 shows a diagram of model parameters of an artificial intelligence model, at 212. The model parameters of the model include a weight matrix, in which each element can be represented as W ij And, performing a quantization operation on the model can include performing a quantization operation on the model parameters of the model (which can include each element of the weight matrix) to obtain quantized model parameters, i.e., a quantized weight matrix, in which each element can be represented as W ij

[0110] A quantization operation can be implemented in various ways, and a quantization operation parameter can be related to the way in which a quantization operation is implemented, i.e., different quantization operations can include different quantization operation parameters. A quantization operation parameter can be determined as needed.

[0111] For example, in some implementations, the way in which a quantization operation is implemented is to scale a model parameter according to a scaling coefficient, and then to truncate the scaled model parameter according to a target quantization range to obtain a quantized model parameter. The target quantization range can refer to the bit width after the format conversion of the model parameter, e.g., if the format of a model parameter is to be converted to INT8, the target quantization range can be 8 bits. For example, a certain original model parameter W 11 is represented in FP32 format as 10111111010010001010110001101111 (represented in decimal as -0.7839), and the quantized parameter obtained by performing a quantization operation on this parameter is represented in INT8 format as 11000000 (represented in decimal as -64),

[0112] In this case, the quantization operation parameter includes a scaling coefficient of the model parameter.

[0113] For another example, in other implementations, the way in which a quantization operation is implemented is to linearly transform a model parameter according to an affine transformation parameter of the model parameter, and then to perform a format conversion on the linearly transformed model parameter to obtain a quantized model parameter.

[0114] For example, the affine transformation parameter is represented as A, and linearly transforming a model parameter according to an affine transformation parameter of the model parameter can be represented as AW, and performing a format conversion on the linearly transformed model parameter can be represented as ​indicates. That is, the quantization operation on the model parameters W of the model can be represented as indicates.

[0115] At this time, the quantization operation parameter includes an affine transformation parameter of the model parameter.

[0116] More details about the affine transformation parameter and the quantization of the model parameter according to the affine transformation parameter can be found in FIG. 3 and its related description.

[0117] After the quantization operation on the model parameters (the parameters include the weights of the model, such as the weight matrix of the model) of the model, the accuracy of the model parameters changes, and the values of the model parameters may change compared with the original model parameters. For example, the model parameters change from floating-point numbers to integers, and the numerical values become smaller.

[0118] Step 220, generating a sparsification operation parameter according to the quantized model parameter of the artificial intelligence model.

[0119] The sparsification operation parameter refers to the operation parameter required by the sparsification operation.

[0120] As mentioned earlier, the sparsification operation is used to prune the model parameters, thereby reducing the number of model parameters. For example, the sparsification operation can prune part of the network layers or part of the parameters of the network layers.

[0121] The sparsification operation parameter can be used to indicate the to-be-pruned parameters in the quantized model parameters.

[0122] Since the quantized model parameters of the model change in value compared with the original model parameters of the model, the importance of the model parameters to the performance of the model may also change.

[0123] For example, the original parameters of the model include a plurality of original weights, and the importance of a certain original weight w1 in the plurality of original weights is ranked as the 220th, and the importance of a certain original weight w2 in the plurality of original weights is ranked as the 250th. It can be seen that the importance of the original weight w1 to the performance of the model is greater than that of the original weight w2.

[0124] After quantizing multiple original weights, multiple quantized weights are obtained. The values ​​of these quantized weights change compared to the original weights, and the importance ranking of the quantized weights also changes. Specifically, the quantized weight w1' of the original weight w1 ranks 240th in importance among the quantized weights, and the quantized weight w2' of the original weight w2 ranks 230th. This shows that the importance of quantized weight w1' to model performance is reduced, and it is less important than quantized weight w2'. Step 220 realizes the influence of the reference quantization operation on the importance of model parameters, generating sparsity operation parameters. This avoids conflicts between quantization and sparsity during subsequent sparsification stacking, ensuring good model performance after compression.

[0125] In some implementations, specifically, sparsification operation parameters can be generated based on the importance of multiple weights in the quantized model parameters.

[0126] The importance of a weight can refer to its impact on model performance. The importance of each weight in the quantized model parameters can be determined in various ways.

[0127] In some embodiments, the importance of the weights in the quantized model parameters can be determined based on the weights in the quantized model parameters and the sample data of the input model.

[0128] As an example, the quantized model parameters are represented by the quantized weight matrix, and the sample data input to the model is represented by the input matrix. For any weight W in the quantized model parameters... ij (Represents the weight in the i-th row and j-th column of the quantized weight matrix), the weight W can be calculated. ij The absolute value and the j-th column of the sample data X j The product of the L2 norms, and the weight W determined based on this product. ij The importance of this. For example, the weight W is determined according to the following formula. ij Importance S ij S ij =|W ij |·||X j ||2 (1)

[0129] Among them, ||X j ||2 represents the j-th column of the sample data X j The L2 norm value.

[0130] The importance of the weight in the quantized model parameter determined in this way combines the value of the weight in the quantized model parameter and the interaction between the weight in the quantized model parameter and the input data, and can accurately reflect the importance of each weight in the quantized model parameter to the model performance. This helps to generate more accurate sparsification operation parameters to better perform sparsification operations.

[0131] In some embodiments, the importance of the weight can also be determined directly according to the value of the weight in the quantized model parameter, or can also be determined by other feasible methods.

[0132] In the quantized model parameter, the greater the importance of the weight, the greater the sparsity priority of the weight can be considered.

[0133] Therefore, among the multiple weights in the quantized model parameter, the part of the weights whose importance meets the target condition can be determined as the pruned parameters, and the sparsification operation parameters are generated to indicate these pruned parameters.

[0134] The target condition can include that the importance of the weight is less than a threshold, or the importance of the weight is less than a threshold in the ordering of the multiple weights, etc.

[0135] As an example, the importance of the multiple weights in the quantized model parameter can be ordered, and the sparsity rate can be determined, and the importance threshold can be determined according to the sparsity rate and the ordering of the importance of the multiple weights. The sparsity rate is used to indicate the proportion of the pruned parameters in the multiple weights, and can be set according to the demand. And the weight whose importance is less than or equal to the importance threshold can be determined as the pruned parameter.

[0136] For example, the sparsity rate is 30%, and 30% of the multiple weights need to be pruned, and at this time the importance threshold can be determined as the 30%*Nth importance value of the multiple weights from low to high importance, N is the number of the multiple weights.

[0137] In some embodiments, the sparsification operation parameters can be represented by a sparse mask matrix. The sparse mask matrix has the same size as the quantized weight matrix of the quantized model parameter.

[0138] For example, FIG. 2 shows a schematic diagram of a sparse mask matrix, in which the element corresponding to the position of the pruned parameter is 0, and the element corresponding to the position of the non-pruned parameter is 1.

[0139] As an example, the sparse mask matrix can be generated according to the following formula:

[0140] where S ij represents the importance of the weight W ij , MS ijelements in the quantized model parameters corresponding to the positions of the to-be-pruned parameters in the sparse mask matrix. ij elements in the quantized model parameters corresponding to the positions of the to-be-pruned parameters in the sparse mask matrix.

[0141] In step 230, the quantized model parameters of the artificial intelligence model are subjected to a sparse operation according to the sparse operation parameter, to obtain a compressed model.

[0142] As described above, the sparse operation parameter can be used to indicate the to-be-pruned parameters in the quantized model parameters. Thus, the quantized model parameters can be subjected to a sparse operation according to the sparse operation parameter, to prune the to-be-pruned parameters.

[0143] The pruning can be setting the to-be-pruned model parameters to 0 or removing the to-be-pruned model parameters, etc.

[0144] For example, FIG. 2 shows a schematic diagram of the sparse model parameters. After the quantized model parameters are subjected to a sparse operation according to the sparse mask matrix, the values of the elements in the quantized model parameters corresponding to the positions of the to-be-pruned parameters in the sparse mask matrix are 0.

[0145] Through the method in FIG. 2, the model can be quantized and sparsified, and the conflict between quantization and sparsification can be avoided, to ensure that the model performance of the compressed model is good.

[0146] FIG. 3 is a flowchart of a second example of a model compression method according to an embodiment of the present application. The model compression method shown in FIG. 3 can be executed by a computing device. For example, the compression processing device 110 in the model compression system 100 can execute the model compression method.

[0147] As shown in FIG. 3, in some implementations, performing steps 210-230 once can be considered as performing a compression process on the model once. The model compression method can further include iteratively performing steps 210-230 to perform a plurality of compression processes (also referred to as a plurality of iterative compression processes) on the model, to obtain a desired compressed model.

[0148] The number of times of performing the plurality of compression processes on the model can be determined according to requirements, or whether the compressed model meets target requirements can be determined, and if yes, the compression process is ended.

[0149] Through the plurality of compression processes on the model, the quantization and sparsification of the model parameters can be more refined and accurate, and the model compression effect is better, for example, the parameter data amount of the model is reduced more, and the model performance of the compressed model is better guaranteed.

[0150] For example, the first execution of steps 210-230 can be referred to as a first compression process, and the obtained compressed model can be referred to as a first compressed model. The second execution of steps 210-230 can be referred to as a second compression process, and the obtained compressed model can be referred to as a second compressed model, and so on.

[0151] The second compression process can include: performing a quantization operation on the first compressed model according to a quantization operation parameter, then generating a sparsification operation parameter according to the quantized model parameter of the second compression process, and then performing a sparsification operation on the quantized model parameter of the second compression process according to the sparsification operation parameter to obtain the second compressed model.

[0152] In some implementations, the quantization operation parameter can also be updated during the multiple compression processes, and the quantization operation in the compression process is performed according to the updated quantization operation parameter.

[0153] For example, the quantization operation parameter can be updated in each compression process after the first compression process. Alternatively, the quantization operation parameter can also be updated in one or more compression processes after the first compression process.

[0154] For any compression process, the method of updating the quantization operation parameter can include: updating the quantization operation parameter according to the results of processing the sample data by the compressed model obtained in the previous compression process and the original model, respectively. The updated quantization operation parameter is used for the quantization operation in the current compression process.

[0155] In some embodiments, specifically, the first processing result of processing the sample data by the compressed model obtained in the previous compression process, and the second processing result of processing the sample data by the original model, can be used to update the quantization operation parameter.

[0156] In some embodiments, specifically, the first processing result of processing the sample data by the compressed model obtained in the previous compression process, and the second processing result of processing the sample data by the original model, can be used to update the quantization operation parameter.

[0157] By updating the quantization operation parameter according to the reconstruction error, it can be ensured that the loss of model performance caused by the quantization operation according to the updated quantization operation parameter is very small.

[0158] In some implementations, the sparsification operation parameter can also be updated during the multiple compression processes, and the sparsification operation in the compression process is performed according to the updated sparsification operation parameter.

[0159] For example, the sparsification operation parameter can be updated in each compression process after the first compression process. Alternatively, the sparsification operation parameter can also be updated in one or more compression processes after the first compression process.

[0160] For each compression processing, the method of updating the sparsification operation parameter can comprise: after the quantization operation is performed on the compressed model obtained from the previous compression processing according to the quantization operation parameter of the current compression processing, the sparsification operation parameter is generated according to the quantized model parameter of the current compression processing, that is, the sparsification operation parameter is updated (in the embodiments of the present application, the method can also be referred to as a reordering strategy). The updated sparsification operation parameter is used for the sparsification operation in the current compression processing.

[0161] The method of generating the sparsification operation parameter according to the quantized model parameter is similar to the method in step 220 in FIG. 2.

[0162] In some embodiments, the sparsification operation parameter is updated together with the quantization operation parameter. For example, the quantization operation parameter and the sparsification operation parameter are updated after each compression processing after the first compression processing.

[0163] Taking the second compression processing as an example, the second compression processing can comprise: updating the quantization operation parameter according to the results of processing the sample data by the original model and the first compressed model obtained from the first compression processing respectively. And performing the quantization operation on the first compressed model according to the updated quantization operation parameter, and then generating the sparsification operation parameter (that is, the updated sparsification operation parameter) according to the quantized model parameter of the second compression processing. And performing the sparsification operation on the quantized model parameter of the second compression processing according to the updated sparsification operation parameter, to obtain the second compressed model.

[0164] Taking the quantization operation according to the affine transformation parameter as an example, for each compression processing, the quantization operation parameter and the sparsification operation parameter can be updated according to the following formula:

[0165] Wherein, X represents the sample data, W represents the model parameter (such as the weight matrix) of the model, A represents the affine transformation parameter (such as the affine transformation matrix), M represents the sparsification operation parameter, and s represents the sparsification operation parameter;

[0166] represents that the quantization operation is performed on the model parameter of the model according to the affine transformation parameter A in the previous compression processing, and the sparsification operation is performed on the quantized model parameter s to obtain the compressed model;

[0167] represents the first processing result of processing the sample data X by the compressed model obtained from the previous compression processing, wherein A in the formula -1 ​is the inverse of the affine transformation parameter A (e.g. the inverse of the affine transformation matrix), which is used in the equation to restore the model parameters after linear transformation according to the affine transformation parameter A, so that the processing result of the compressed model processing sample data X will not be affected by the linear transformation;

[0168] XW represents the second processing result of the original model processing sample data X;

[0169] is the l2-norm norm, represents that the value of y is minimized as the optimization objective, the affine transformation parameter A (e.g. the value of each element in the affine transformation matrix) is solved, that is, the updated affine transformation parameter is solved. And after obtaining the updated affine transformation parameter, the updated sparsification operation parameter M can be obtained by the foregoing method s .

[0170] Wherein, the affine transformation parameter A in equation (3) can be solved by various feasible optimization algorithms such as gradient descent.

[0171] In some embodiments, the sample data used in FIG. 2 and FIG. 3 can be obtained by the computing device from the database, and the sample data can include the same type of data as the input data that the model can process, such as text, image and audio, etc.

[0172] In some embodiments, in the method of FIG. 3, the quantized operation parameter and the sparsification operation parameter can be updated according to the first processing result of the compressed model processing a batch of sample data, and the second processing result of the original model processing the batch of sample data.

[0173] By optimizing and updating the quantized operation parameter and the sparsification operation parameter according to the performance of the compressed model obtained by the previous compression processing in the process of multiple compression processing of the model, the influence of the quantized operation parameter and the sparsification operation parameter on the model performance can be adjusted and controlled. Quantization and sparsification according to the updated quantized operation parameter and the sparsification operation parameter can reduce the loss of model performance.

[0174] And the affine transformation parameter is used for linear transformation of the model parameters, which can adjust the size and distribution of the model parameters during model compression (e.g. adjust the model parameter range and parameter value during each compression processing). By updating the affine transformation parameter in multiple compression processes, the parameter change in the quantization and sparsification process can be effectively controlled, the information loss in the quantization and sparsification process is balanced, and the model performance of the compressed model is better guaranteed. For example, by adjusting the parameters of the affine transformation parameter, each quantization operation can only target part of the model parameters, so that more important information can be retained in the quantization operation, helping to more accurately identify and prune the parameters with low importance in the sparsification operation.

[0175] FIG. 4 is a flow diagram of a third embodiment of the model compression method according to the present application. The method shown in FIG. 4 mainly illustrates the detailed steps of the quantization operation on the model parameters according to the affine transformation parameters.

[0176] The model compression method shown in FIG. 4 can be executed by a computing device. For example, it can be executed by the compression processing device 110 in the model compression system 100.

[0177] The method of quantization operation on the model parameters according to the affine transformation parameters in FIG. 4 can be applied to the model compression method described in FIG. 2 or FIG. 3 to implement the quantization operation in the one-time compression processing in FIG. 2 or FIG. 3.

[0178] In some embodiments, the model parameters involved in the methods of FIG. 2, FIG. 3, and FIG. 4 can refer to the parameters of a single network layer of the model, such as the weight matrix of a single network layer such as a linear layer, a convolutional layer, etc. Each network layer can apply its own quantization operation parameters and sparsification operation parameters independently. That is, the parameters of each network layer in the model can be compressed (including quantization operation and sparsification operation) by the methods of FIG. 2, FIG. 3, and FIG. 4 respectively.

[0179] As an example, the model can be a Transformer, which can include a qkv layer, an fc1 layer, a qkt layer, an output layer, etc. The model compression method provided by the embodiments of the present application can apply its own quantization operation parameters and sparsification operation parameters to each of the aforementioned layers such as the qkv layer, the fc1 layer, the qkt layer, and the output layer, to compress each layer respectively and obtain a compressed model.

[0180] The parameters of different network layers (such as linear layers) of the model are different, and by finely compressing each network layer separately, each network layer applies its own quantization operation parameters and sparsification operation parameters, the model parameter compression of different network layers combines the characteristics of the parameters of different network layers, and the model parameter compression of each network layer is good.

[0181] As shown in FIG. 4, the method includes the following steps.

[0182] Step 410: performing linear transformation on the model parameters of the artificial intelligence model according to the affine transformation parameters.

[0183] In some embodiments, the affine transformation parameters can be an affine transformation matrix of the model parameters.

[0184] As an example, the model parameter is a weight matrix W, which includes n rows and m columns of elements, and the affine transformation parameter A of the model parameter is an affine transformation matrix of n rows and n columns. It can be known that the linear transformation of the model parameter according to the affine transformation parameter of the model parameter is to calculate AW, and an n-row and m-column matrix is obtained.

[0185] The value of the affine transformation parameter can be set according to requirements, and the value of the affine transformation parameter can be updated in the process of multiple compression processes (for example, the affine transformation parameter in the quantization operation parameter is updated according to the method in FIG. 3).

[0186] As an example, when the first compression process is performed, the value of the affine transformation parameter can be set according to requirements by various methods.

[0187] For example, when the first compression process is performed, the affine transformation parameter of the model parameter is set to a matrix in which the diagonal elements are 1 and the remaining elements except the diagonal elements are 0.

[0188] For another example, when the first compression process is performed, the smooth scale is calculated according to the input of the network layer (for example, the qkv layer, the fc1 layer, the output layer, etc.) of the model and the parameter of the network layer, and the affine transformation parameter of the model parameter of the network layer is determined according to the smooth scale. Wherein, the smooth scale can be determined according to the maximum value of each dimension of the input (such as the input matrix) of the network layer divided by the maximum value of each dimension of the parameter (such as the weight matrix) of the network layer.

[0189] In one compression process of multiple compression processes, the model parameter of the artificial intelligence model is linearly transformed according to the affine transformation parameter, including: calculating the Hadamard product of the affine transformation parameter (for example, the affine transformation matrix) and the progressive mask (for example, the progressive mask matrix) to obtain a target affine transformation matrix, and then linearly transforming the model parameter according to the target affine transformation matrix.

[0190] As an example, in one compression process, the model parameter is a weight matrix W, the affine transformation parameter is A, and the progressive mask matrix is G M , the target affine transformation matrix A` = A × G M , and the linear transformation of the model parameter of the artificial intelligence model according to the affine transformation parameter (which can be represented as AW) is A`W (that is, AW = A`W).

[0191] The value of the progressive mask can be updated in the process of multiple compression processes.

[0192] The value of the progressive mask can be determined by the following formula each time the compression process is performed:

[0193] GMij represents the element in the i-th row and j-th column of the progressive mask matrix, hidden size represents the dimension of the affine transformation matrix, t is the target number of times (t can be set according to requirements, for example, set to 5, 6, etc.), e ∈ [1, t] is the compression processing number of the current compression processing (for example, the compression processing number of the second compression processing is 2), and a is a stability factor (the value of a can be set according to requirements, for example, 0.1, 0.01, etc.).

[0194] As an example, FIG. 4 shows a schematic diagram of a progressive mask in multiple compression processes at 412. As shown in FIG. 4 at 412:

[0195] In the first compression process, only the elements on the diagonal line in the progressive mask take the value of 1, and the elements at other positions take the value of 0.

[0196] In the second compression process, based on the progressive mask in the first compression process, the values of the elements adjacent to the diagonal line in the progressive mask change from 0 to a, that is, the number of 0 elements in the progressive mask decreases.

[0197] The subsequent compression processes are sequentially updated according to the aforementioned progressive mask update mode until the progressive mask does not include 0 elements, that is, except for the diagonal line elements taking the value of 1, the values of the remaining elements are all a.

[0198] The aforementioned update process of the progressive mask can be understood as that the elements far from the diagonal line in the progressive mask are gradually unfrozen with the multiple compression processes.

[0199] For example, FIG. 4 also shows a schematic diagram of a target affine transformation matrix determined according to the progressive mask in multiple compression processes at 412.

[0200] As shown in FIG. 4 at 412:

[0201] The elements in the affine transformation matrix in the n-th compression process are represented by A n ij .

[0202] In the first compression process, according to the Hadamard product of the affine transformation matrix A and the progressive mask, the target affine transformation matrix A 1 is obtained, and only the elements A 1 11 , A 1 22 …A 1 nn on the diagonal line in the affine transformation matrix A 1 take non-0 values, and the elements at other positions take the value of 0. The linear transformation of the model parameters according to the target affine transformation matrix A 1 is represented as A 1 W.

[0203] In the second compression process, the affine transformation matrix A used in the previous compression process is updated according to the method in the aforementioned FIG. 3, to obtain an updated affine transformation matrix A, and the target affine transformation matrix A is obtained according to the Hadamard product of the updated affine transformation matrix A and the progressive mask. 2 In the target affine transformation matrix A of the first compression process 1 On the basis of the target affine transformation matrix A 2 The values of the elements adjacent to the diagonal line in the target affine transformation matrix A change from 0 to non-0 values, i.e., the target affine transformation matrix A 2 The number of 0 elements in the target affine transformation matrix A is reduced. According to the target affine transformation matrix A 2 The linear transformation of the model parameters can be expressed as A 2 W.

[0204] In the subsequent compression process, the affine transformation matrix is updated similarly to the second compression process, and the determination method of the aforementioned target affine transformation matrix is sequentially extended until the target affine transformation matrix does not include 0 elements.

[0205] The updating process of the aforementioned target affine transformation matrix can be understood as follows: during the multiple compression processes, the elements at positions other than the diagonal line in the affine transformation matrix are gradually unfrozen in order of increasing distance from the diagonal line, and the values of the elements at the positions in the affine transformation matrix are updated in the multiple compression processes.

[0206] Therefore, by the method in step 410, the diagonal line elements of the weight matrix can be quantized first in the multiple compression processes, and the elements at positions other than the diagonal line can be gradually quantized in order of increasing distance from the diagonal line. This method can reduce the negative impact of model parameter quantization on model performance.

[0207] Step 420: performing format conversion on the model parameters of the artificial intelligence model after the linear transformation.

[0208] In some embodiments, the format conversion of the model parameters after the linear transformation can be implemented in various ways to obtain the quantized model parameters.

[0209] As an example, the model parameters after the linear transformation can be scaled according to the scaling coefficient, then the scaled model parameters can be added with an offset, and the model parameters after the addition of the offset can be truncated according to the target quantization range to implement the format conversion and obtain the quantized model parameters.

[0210] The target quantization range can refer to the bit width of the model parameters after the format conversion, for example, if the model parameters are converted to INT8, the target quantization range can be 8 bits.

[0211] The model parameters after the linear transformation of step 410 can be denoted as AW. The format conversion of the linearly transformed model parameters AW can be denoted as .

[0212] For example, the format conversion of the linearly transformed model parameters AW can be performed according to the following formula:

[0213] wherein, Δ denotes a scaling factor; zp denotes an offset; n is the bit width of the target quantization range (for example, the target quantization range can be 8 bits, and n takes the value of 8); clamp(.) denotes that the input data AW is limited in the target quantization range.

[0214] For example, the format conversion of the linearly transformed model parameters AW can be performed according to the following formula:

[0215] In some embodiments, when the method in FIG. 4 is applied to the model compression method in FIG. 3, the quantization operation in FIG. 4 can be pseudo-quantization. Pseudo-quantization refers to a way of simulating quantization during multiple compression processes, and the model parameters are not actually quantized. After the multiple compression processes are completed, the model parameters are quantized according to the calculation results of the pseudo-quantization.

[0216] In the foregoing, the model compression method provided by the embodiments of the present application is described in detail with reference to FIGS. 2 to 4. In the following, the apparatus provided by the present application will be described with reference to FIG. 5. These apparatuses can be used to realize the functions of the processors in the above-mentioned method embodiments, and thus can also realize the beneficial effects possessed by the above-mentioned method embodiments.

[0217] FIG. 5 is a structural schematic diagram of a model compression apparatus provided by an embodiment of the present application. As shown in FIG. 5, the model compression apparatus 500 includes a quantization operation module 510 and a sparsification operation module 520. In some embodiments, the model compression apparatus 500 can further include a storage module 530.

[0218] The quantization operation module 510 can be used to perform a quantization operation on the artificial intelligence model according to quantization operation parameters, and the quantization operation is used to convert the format of the model parameters of the artificial intelligence model. For the specific content of the operation, please refer to step 210 in FIG. 2, FIG. 3, FIG. 4 and the related descriptions thereof.

[0219] The sparsification operation module 520 can be configured to generate the sparsification operation parameter according to the quantized model parameter of the artificial intelligence model. In addition, the sparsification operation module 520 can be further configured to perform the sparsification operation on the quantized model parameter of the artificial intelligence model according to the sparsification operation parameter, to obtain the compressed model. For details of the operation, please refer to steps 220 and 230 in FIG. 2, FIG. 3, and the related description.

[0220] The storage module 530 can be configured to save the compressed model in a target storage medium. For details of the operation, please refer to FIG. 1 and the related description.

[0221] In some implementations, the model parameter includes a plurality of parameters, and the sparsification operation module 520 can be further configured to generate the sparsification operation parameter according to the importance of the plurality of parameters in the quantized model parameter. For details of the operation, please refer to step 210 in FIG. 2, FIG. 3, FIG. 4, and the related description.

[0222] In some implementations, the quantization operation module 510 is further configured to update the quantization operation parameter according to the results of processing the sample data by the compressed model and the artificial intelligence model, respectively. For details of the operation, please refer to FIG. 2, FIG. 3, and the related description.

[0223] In some implementations, the quantization operation module 510 is further configured to process the sample data according to the compressed model to obtain a first processing result of the sample data, and process the sample data according to the artificial intelligence model to obtain a second processing result of the sample data, and update the quantization operation parameter according to the difference between the first processing result and the second processing result. For details of the operation, please refer to FIG. 3 and the related description.

[0224] In some implementations, the quantization operation module 510 is further configured to perform the quantization operation on the compressed model according to the updated quantization operation parameter. For details of the operation, please refer to FIG. 3, FIG. 4, and the related description.

[0225] In some implementations, the sparsification operation module 520 is further configured to update the sparsification operation parameter according to the quantized model parameter of the compressed model. For details of the operation, please refer to FIG. 3 and the related description.

[0226] In some implementations, the quantization operation parameter includes an affine transformation parameter of the model parameter of the artificial intelligence model. For details of the quantization operation parameter and the affine transformation parameter, please refer to step 210 in FIG. 2, FIG. 4, and the related description.

[0227] In some implementations, the quantization operation module 510 is further configured to perform a linear transformation on the model parameters of the artificial intelligence model according to the affine transformation parameters, and perform a format transformation on the model parameters of the artificial intelligence model after the linear transformation. For details of the operation, please refer to step 210 in FIG. 2, FIG. 4 and the related description.

[0228] In some implementations, the sparsification operation parameters are used to indicate the parameters to be pruned in the quantized model parameters. For details of the sparsification operation parameters, please refer to steps 220 and 230 in FIG. 2, FIG. 3 and the related description.

[0229] In some implementations, the sparsification operation module 520 is further configured to perform pruning on the parameters to be pruned in the quantized model parameters of the artificial intelligence model according to the sparsification operation parameters. For details of the operation, please refer to step 230 in FIG. 2 and the related description.

[0230] In some implementations, the model parameters include the parameters of the linear layer of the artificial intelligence model, the quantization operation parameters include the quantization operation parameters of the parameters of the linear layer, and the sparsification operation parameters include the sparsification operation parameters of the parameters of the linear layer. For details of the operation, please refer to FIG. 4 and the related description.

[0231] In some implementations, the compressed model is used to process at least one of text, image and audio to obtain a processing result. For details of the operation, please refer to FIG. 1 and the related description.

[0232] It should be noted that FIG. 5 is only a structural schematic diagram of the device, and the positional relationship of the quantization operation module 510, the sparsification operation module 520 and the storage module 530 shown in FIG. 5 does not constitute any limitation. For example, the quantization operation module 510 and the sparsification operation module 520 are arranged in one device in FIG. 5, and in other cases, the quantization operation module 510 and the sparsification operation module 520 can also be arranged in different devices respectively.

[0233] Optionally, the quantization operation module 510, the sparsification operation module 520 and the storage module 530 can each include a plurality of sub-modules, which can be respectively deployed to respectively implement part of the functions of the corresponding modules, for example, to implement one or more steps of the model compression method provided in the foregoing FIG. 2-FIG. 4.

[0234] The device can be implemented by software or by hardware. For example, the implementation of the model compression device 500 is introduced as follows.

[0235] As an example of a software functional unit, the model compression apparatus 500 can include code running on a compute instance. The compute instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, the model compression apparatus 500 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. The multiple hosts / virtual machines / containers running the code can be distributed in the same AZ, or in different AZs. Each AZ includes one data center or multiple data centers in close geographical proximity. Typically, one region can include multiple AZs.

[0236] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same VPC, or in multiple VPCs. Typically, one VPC is set up within one region. Communication between two VPCs in the same region, and cross-region communication between VPCs in different regions, requires a communication gateway to be set up in each VPC to achieve interconnection between VPCs.

[0237] As an example of a hardware functional unit, the model compression apparatus 500 can include at least one computing device, such as a server, and the like. Alternatively, the model compression apparatus 500 can also be a device implemented using an ASIC, or a PLD, and the like. The PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0238] The multiple computing devices included in the model compression apparatus 500 can be distributed in the same region, or in different regions. The multiple computing devices included in the model compression apparatus 500 can be distributed in the same AZ, or in different AZs. Similarly, the multiple computing devices included in the model compression apparatus 500 can be distributed in the same VPC, or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, and the like.

[0239] The application further provides a computing device 600. As shown in FIG. 6, the computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other through the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 600 is not limited. Optionally, the processor 604 of the computing device 600 can be connected with a display, an input device (not shown in FIG. 6) through the communication interface 608.

[0240] The bus 602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 6, but it does not mean that there is only one bus or only one type of bus. The bus 602 can include a path for transmitting information between various components (for example, the memory 606, the processor 604, the communication interface 608) of the computing device 600.

[0241] The processor 604 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc. In the embodiments of the application, the processor 604 is configured to perform the steps or operations in the embodiments of the application.

[0242] The memory 606 can include a volatile memory (for example, a random access memory (RAM)). The processor 604 can further include a non-volatile memory (for example, a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0243] The executable program code stored in the memory 606 is executed by the processor 604 to implement the functions of the aforementioned quantization operation module 510, the sparsification operation module 520, and the storage module 530 respectively, thereby implementing one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application. That is, the instructions for executing one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application are stored in the memory 606.

[0244] The communication interface 608 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, etc., to implement the communication between the computing device 600 and other devices or communication networks.

[0245] The display can include various display devices that can implement display functions, such as a plasma display, a liquid crystal display, etc. The computing device 600 can implement the display functions involved in one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application through the display.

[0246] The input device can include various input devices that can implement information / signal input, such as a keyboard, a mouse, a touch screen, etc.

[0247] The present application also provides a computing device cluster 700. As shown in FIG. 7, the computing device cluster includes at least one computing device 600. The memory 606 in one or more computing devices 600 in the computing device cluster can store the same instructions for executing one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application.

[0248] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster can also respectively store partial instructions for executing one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application. In other words, the combination of one or more computing devices 600 can collectively execute the instructions for executing one or more steps of the model compression method provided in FIGS. 2-4 of the embodiments of the present application.

[0249] It should be noted that the memories 606 in different computing devices 600 in the computing device cluster can store different instructions for respectively implementing part of the functions of the model compression apparatus 500. That is, the instructions stored in the memories 606 in different computing devices 600 can implement the functions of one or more of the aforementioned quantization operation module 510, the sparsification operation module 520, and the storage module 530.

[0250] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, etc.

[0251] In some possible implementation, the memory 606 of one or more computing devices 600 in the computing device cluster can also respectively store instructions for performing one or more steps of the model compression method provided in FIG. 2-FIG. 4 of the embodiments of the present application. In other words, the combination of one or more computing devices 600 can collectively perform the instructions of one or more steps of the model compression method provided in FIG. 2-FIG. 4 of the embodiments of the present application.

[0252] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions capable of running on a computing device or stored in any available medium. The instructions contained in the computer program product can include instructions for performing one or more steps of the model compression method provided in FIG. 2-FIG. 4 of the embodiments of the present application. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform the steps or operations of the model compression method provided in FIG. 2-FIG. 4 of the embodiments of the present application.

[0253] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device containing one or more available media that a computing device can store. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the steps or operations of the model compression method provided in FIG. 2-FIG. 4 of the embodiments of the present application.

[0254] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

[0255] The terms "first", "second", "third", and "fourth" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and not to limit a specific order.

[0256] In the present embodiments, the word "exemplary" or "for example" is used to mean "an example of" or "for the purpose of illustration." Any embodiment or design scheme described as "exemplary" or "for example" in the present embodiments should not be construed as preferred or advantageous over other embodiments or design schemes. Rather, the use of the word "exemplary" or "for example" is intended to present concepts in a concrete manner.

Claims

1. A model compression method, characterized in that, The method comprises: performing a quantization operation on the artificial intelligence model according to quantization operation parameters, the quantization operation being used for converting the format of model parameters of the artificial intelligence model; generating sparsification operation parameters according to the quantized model parameters of the artificial intelligence model; performing a sparsification operation on the quantized model parameters of the artificial intelligence model according to the sparsification operation parameters, to obtain a compressed model; storing the compressed model in a target storage medium.

2. The method of claim 1, wherein, The model parameters comprise a plurality of parameters, and the generating of the sparsification operation parameters according to the quantized model parameters of the artificial intelligence model comprises: generating the sparsification operation parameters according to the importance of the plurality of parameters in the quantized model parameters.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: updating the quantization operation parameters according to the results of processing sample data by the compressed model and the artificial intelligence model respectively.

4. The method of claim 3, wherein, The updating of the quantization operation parameters according to the results of processing sample data by the compressed model and the artificial intelligence model respectively comprises: processing the sample data according to the compressed model to obtain a first processing result of the sample data; processing the sample data according to the artificial intelligence model to obtain a second processing result of the sample data; updating the quantization operation parameters according to the difference between the first processing result and the second processing result.

5. The method according to any one of claims 1-4, characterized in that, The method further comprises: performing a quantization operation on the compressed model according to the updated quantization operation parameters; updating the sparsification operation parameters according to the quantized model parameters of the compressed model.

6. The method according to any one of claims 1-5, characterized in that, The quantization operation parameters comprise affine transformation parameters of the model parameters of the artificial intelligence model, and the performing of the quantization operation on the artificial intelligence model according to the quantization operation parameters comprises: performing a linear transformation on the model parameters of the artificial intelligence model according to the affine transformation parameters; performing a format conversion on the model parameters of the artificial intelligence model after the linear transformation.

7. The method according to any one of claims 1 to 6, characterized in that, The sparsification operation parameters are used for indicating to-be-pruned parameters in the quantized model parameters, and the performing of the sparsification operation on the quantized model parameters of the artificial intelligence model according to the sparsification operation parameters comprises: pruning the to-be-pruned parameters in the quantized model parameters of the artificial intelligence model according to the sparsification operation parameters.

8. The method according to any one of claims 1 to 7, characterized in that, The model parameters comprise parameters of a linear layer of the artificial intelligence model, the quantization operation parameters comprise quantization operation parameters of the parameters of the linear layer, and the sparsification operation parameters comprise sparsification operation parameters of the parameters of the linear layer.

9. The method according to any one of claims 1-8, characterized in that, The compressed model is used for processing at least one kind of data in text, image and audio to obtain a processing result.

10. A model compression apparatus, comprising: The device comprises: a quantization operation module, configured to perform a quantization operation on the artificial intelligence model according to quantization operation parameters, the quantization operation being used for converting the format of model parameters of the artificial intelligence model; a sparsification operation module, configured to generate sparsification operation parameters according to the quantized model parameters of the artificial intelligence model; a storage module, configured to store the compressed model in a target storage medium.

11. The apparatus of claim 10, wherein, The quantization operation module is further configured to: According to results of processing sample data respectively by the compressed model and the artificial intelligence model, the quantization operation parameter is updated.

12. The apparatus of any one of claims 10-11, wherein, The quantization operation module is further configured to perform a quantization operation on the compressed model according to the updated quantization operation parameter. The sparsification operation module is further configured to update the sparsification operation parameter according to quantized model parameters of the compressed model.

13. The apparatus of any one of claims 10-12, wherein, The quantization operation parameter comprises an affine transformation parameter of model parameters of the artificial intelligence model, and the quantization operation on the artificial intelligence model according to the quantization operation parameter comprises: performing a linear transformation on the model parameters of the artificial intelligence model according to the affine transformation parameter; and performing a format transformation on the model parameters of the artificial intelligence model after the linear transformation.

14. A computing device, comprising: The computing device comprises a processor and a memory, and the processor is configured to execute instructions stored in the memory to cause the computing device to perform the method of any one of claims 1-9.

15. A cluster of computing devices, characterized in that, The computing device cluster comprises at least one computing device, and each computing device comprises a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method of any one of claims 1-9.

16. A computer program product comprising instructions, characterized in that, The instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method of any one of claims 1-9.

Images