Inference model compression method and apparatus

Abstract

Disclosed are an inference model compression method and apparatus, which relate to the technical field of computers. The method comprises: inputting inference data into an inference model, and executing the inference model; then, for a first network layer in the inference model, acquiring a weight matrix of the first network layer and input data of the first network layer, and calculating the product of the weight matrix of the first network layer and the input data of the first network layer to obtain a result matrix that indicates the importance of weight values in the weight matrix; and next, in descending order of the importance, selecting the weight values that rank in the last a% in the weight matrix of the first network layer to perform a sparsification operation, and executing the sparsification operation on a plurality of network layers in the inference model to obtain a sparsified inference model. The first network layer is any one layer among the plurality of network layers, the result matrix and the weight matrix have the same size, and the magnitude of the value at the same position in the result matrix as in the weight matrix indicates the importance of the weight value at the same position in the weight matrix.

Description

Inference model compression method and device The present application claims priority to the Chinese Patent Application No. 202410525590.3, filed on April 28, 2024, entitled "Inference model compression method and device", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD The present application relates to the technical field of computer, and particularly relates to an inference model compression method and device. BACKGROUND With the continuous popularization of large language models (LLMs), enterprises or individuals are deploying and using LLMs in large quantities. Since LLMs often have a parameter quantity of hundreds of billions or trillions, the deployment of LLMs often occupies a large amount of hardware resources (such as storage resources or computing resources, etc.). In order to reduce the large occupation of hardware resources by LLMs, the parameters of LLMs are often reduced by means of cyclic pruning and quantization, or the weight matrix of LLMs is split into a dense matrix and a sparse matrix by principal component analysis, and then the dense matrix and the sparse matrix are quantized respectively. However, the structured pruning (such as pruning a certain network layer, channel or filter) is adopted in the process of cyclic pruning of LLMs, and the structured pruning has a large granularity, which can cause some features in the LLM to be directly deleted, and thus the processing precision of the LLM after cyclic pruning and quantization is greatly lost, and the processing precision is reduced. Since the sizes of the split dense matrix and sparse matrix are consistent with the size of the weight matrix of the LLM, in essence, two matrices of the same size are quantized, and thus the compression rate of the model is limited compared to quantizing only one matrix. Therefore, how to reduce the precision loss of the compressed model with a high compression rate has become a problem to be solved. SUMMARY The present application provides an inference model compression method and device to solve the problem of high precision loss of the compressed model with a high compression rate when the model is compressed. The present application adopts the following technical solutions. In a first aspect, the present application provides a method for compressing an inference model. The method can be applied to a computer system or a computing device of the computer system for implementing the method. The computing device can be a server or a terminal. In one possible example, the inference model includes a plurality of network layers. The method includes inputting inference data into the inference model and executing the inference model. Then, for a first network layer in the inference model, a weight matrix of the first network layer is obtained, which is the input data of the first network layer, and a product of the weight matrix and the input data of the first network layer is calculated to obtain a result matrix showing the importance of the weight values in the weight matrix. Thus, according to the order of importance from large to small, the weight values ranked at the last a% in the weight matrix of the first network layer are selected for sparse operation. After the sparse operation is performed on the plurality of network layers of the inference model, a sparse inference model is obtained. The first network layer is any one of the plurality of network layers. The result matrix has the same size as the weight matrix. The size of the value in the same position in the result matrix indicates the importance of the weight value in the same position in the weight matrix. The a% is the sparsity of the first network layer in the inference model. In the present application, the result matrix indicating the importance of the weight values in the weight matrix is used to sparsify the weight values with lower importance (ranked at the last a% in the order of importance from large to small) in the weight matrix. The weight matrix of the sparse inference model retains the weight values with higher importance. Therefore, the foregoing sparse operation has little effect on the inference accuracy of the inference model, and the inference accuracy loss of the obtained sparse inference model is small. In addition, the sparse operation is to set some weight values in the weight matrix to 0. Thus, the sparse inference model is compressed compared with the inference model. Therefore, the inference accuracy loss of the sparse inference model is reduced while achieving a high compression rate of the sparse inference model. In one possible case, the inference model is a pre-trained model. In one possible case, the product of the weight matrix of the first network layer and the input data of the first network layer in the inference model is calculated to obtain the result matrix, including calculating the product of the norm of the weight matrix of the first network layer and the input data of the first network layer to obtain the result matrix. In the present application, since the result matrix is obtained by multiplying the weight matrix of the first network layer in the inference model with the norm of the input data of the first network layer, and the norm is used to measure the size of the input data (vector), the value obtained by multiplying the weight matrix with the norm of the input data can indicate the influence degree of the weight value in the weight matrix on the output data of the network layer. The greater the value, the greater the influence degree, and the higher the importance of the weight value in the weight matrix. The value in the result matrix is used as an importance indicator of the weight in the weight matrix, and then the weight matrix is sparsified according to the importance indicator of the weight value in the weight matrix, which reduces the sparsity granularity and avoids sparsifying the weight value with high importance, reduces the influence of the sparsification process on the processing accuracy of the model, and realizes that the inference accuracy loss of the sparsified inference model is small. In a possible example, the norm can be Manhattan norm, Euclidean norm, infinite norm, etc. In a possible case, the sparsity rates of the network layers in the inference model can be different values. In a possible example, the sparsity rates of the network layers in the inference model are set by the user. That is, the computing device receives the sparsity rates of the network layers in the inference model set by the user. In another possible example, the sparsity rate is obtained by selecting from a plurality of set sparsity rates in a multi-objective optimization manner. The optimization objective of the multi-objective optimization is that the overall sparsity rate of the inference model is greater than or equal to the target sparsity rate, and the sum of the sparsity errors of all network layers in the inference model is less than or equal to a first threshold. In the present application, the optimal sparsity rate of each network layer in the inference model is determined in the case that the sum of the sparsity errors of all network layers is less than or equal to a second threshold, and the overall sparsity rate of the inference model is greater than or equal to a target sparsity rate, by a multi-objective optimization manner. The weight matrix of the network layer is sparsified according to the optimal sparsity rate of each network layer determined in the foregoing, to obtain a sparsified inference model, which realizes compression of the inference model, and the accuracy loss of the sparsified inference model is small. For example, the network layers in the inference model are sparsified by using a plurality of set sparsity rates, the sparsity errors of the network layers in the inference model before and after sparsification are determined, and then the sparsity rates of the network layers in the inference model are determined in the case that the overall sparsity rate of the inference model is greater than or equal to a target sparsity rate, and the sum of the sparsity errors of all network layers in the inference model is less than or equal to a first threshold, by a multi-objective optimization manner. The content of the first threshold can be referred to the threshold b in the specific embodiments described below. In a possible implementation, the weight values in the weight matrix of the first network layer in the inference model and the weight matrix of the sparse inference model are the same, and the same weight values include the weight values in the weight matrix of the first network layer of the inference model that are greater than or equal to the second threshold. The second threshold is any one of the M values that are last in the order of descending absolute values of the weight values in the weight matrix of the first network layer of the inference model, and M is an integer greater than or equal to 1. In this application, because the weight values in the weight matrix of the network layer that are arranged in the front have a greater impact on the processing of the input data to obtain the output data, when the weight matrix of the network layer is sparse, the weight values are protected, that is, not sparse, which can reduce the loss of the processing accuracy of the sparse inference model. In a possible example, according to the weight values in the weight matrix of the first network layer that are greater than or equal to the second threshold, a mask is generated (for example, the value of the first position in the mask is 1, the first position is the position of the weight value greater than or equal to the second threshold in the weight matrix, the value of the second position in the mask is 0, the second position is the position of the weight value less than the second threshold in the weight matrix, and the mask is consistent with the size of the weight matrix), and then when the weight matrix of the first network layer is sparse, the positions with the value 1 in the mask are not sparse, and the positions with the value 0 are allowed to be sparse. For the content of the second threshold, refer to the threshold a in the specific embodiments below. In a possible implementation, the inference model compression method further includes performing a quantization operation on the sparse inference model to obtain a quantized inference model. In this application, by performing a quantization operation on the sparse inference model, the bit width of the weight values in the weight matrix of the sparse inference model can be reduced, thereby improving the compression rate of the inference model and reducing the storage space occupied by the weight matrix and the like. Moreover, because the bit width of the weight values is reduced, the amount of calculation for inference using the weight values in the weight matrix is reduced, thereby improving the efficiency of inference using the quantized inference model. In a possible example, a uniform quantization operation is performed on the sparse inference model. In another possible example, a non-uniform quantization operation is performed on the sparse inference model. In a possible implementation, the inference model compression method further includes compressing the weight matrices of the network layers in the quantized inference model to obtain a compressed model, and the compressed model includes a compressed weight matrix. In a possible implementation, the inference model compression method further includes: receiving a compression request sent by a user, the compression request including a field for indicating a compression manner. The compression of the weight matrix of each network layer in the quantized inference model to obtain the compressed model includes: compressing the weight matrix of each network layer in the quantized inference model by using the compression manner indicated by the field to obtain the compressed model. In a possible example, the compression manner includes entropy encoding or run-length encoding. In this application, the quantized inference model is compressed by bytes, which further increases the compression rate of the inference model, and the compressed inference model occupies less memory, which is beneficial to deploying the compressed inference model to a device with limited hardware resources such as an embedded device, reduces the hardware requirements of an inference model with a large number of parameters (the number of weight values in the weight matrix), and does not affect the processing precision. In a possible implementation, the inference model further includes: if the number of weight values that are zero in the weight matrix of each network layer in the quantized model is greater than a third threshold, sending first information to a user interface. If the number of weight values that are zero in the weight matrix of each network layer in the quantized model is less than the third threshold, sending second information to the user interface. The first information is used to indicate that the compression manner with a compression rate less than a fourth threshold is used, and the second information is used to indicate that the compression manner with a compression rate greater than the fourth threshold is used. In a possible example, if the number of weight values that are zero in the weight matrix of each network layer in the quantized model is equal to the third threshold, the first information or the second information is sent to the user interface according to the configuration of the user. In this application, the first information or the second information is sent to the user interface, so as to give the user a suggestion on the compression manner through the first information or the second information, to improve the compression rate of the weight matrix of each network layer in the quantized inference model. In a possible implementation, the inference model further includes: if the inference accuracy loss of the sparse inference model is greater than or equal to a fifth threshold, retraining the sparse inference model, so that the accuracy loss of the retrained inference model is less than the fifth threshold. In this application, the sparse inference model with a large accuracy loss is retrained to reduce the accuracy loss, so as to reduce the processing accuracy loss of the compressed model (the sparse model) while ensuring that the compressed model has a high compression rate. In a second aspect, the present application provides a method for deploying an inference model. The method can be applied to a computer system or a computing device supporting the computer system to implement the method. The computing device can be a server or a terminal. The method comprises: obtaining a first model comprising a compressed weight matrix; decompressing the compressed weight matrix in the first model to obtain a second model; and deploying the second model. The second model is obtained by sparsifying and quantizing the weight matrix of a first network layer of a third model according to the importance of a result matrix obtained by multiplying the weight matrix of the first network layer of the third model with input data of the first network layer. The result matrix and the weight matrix have the same size, and the value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix. The first network layer is any one of the plurality of network layers included in the third model. In a possible case, the sparsifying operation is performed on the weight values ranked after a% in the weight matrix of the first network layer according to the order of the importance from large to small, and a% is the sparsity of the inference model. In a possible example, the first model is the compressed inference model in the first aspect, the second model is the quantized inference model in the first aspect, and the third model is the inference model in the first aspect. In a third aspect, the present application provides an apparatus for compressing an inference model. The apparatus can be applied to a computer system or a computing device supporting the computer system to implement the method for compressing an inference model. The apparatus comprises various modules for implementing the method for compressing an inference model in the first aspect or any possible implementation of the first aspect. For example, the apparatus comprises an input module, an obtaining module, a calculating module, and a sparsifying module. The input module is configured to input inference data into the inference model and execute the inference model. The obtaining module is configured to, for a first network layer in the inference model, obtain a weight matrix of the first network layer and input data of the first network layer, wherein the first network layer is any one of the plurality of network layers included in the inference model. The calculating module is configured to calculate the product of the weight matrix of the first network layer and the input data of the first network layer in the inference model to obtain a result matrix. The result matrix and the weight matrix have the same size, and the value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix. The sparsifying module is configured to perform a sparsifying operation on the weight values ranked after a% in the weight matrix of the first network layer according to the order of the importance from large to small, and a% is the sparsity of the inference model. After the sparsifying operation is performed on the plurality of network layers, a sparsified inference model is obtained. In a possible implementation, the sparsity is set by a user. In a possible implementation, the sparsity is selected from a plurality of set sparsities by using multi-objective optimization, and an optimization objective of the multi-objective optimization is that the sparsity of the entire inference model is greater than or equal to a target sparsity, and a sum of sparsity errors of all network layers in the inference model is less than or equal to a first threshold. In a possible implementation, the same weight values in the weight matrix of the first network layer of the inference model and the weight matrix of the sparse inference model include weight values greater than or equal to a second threshold in the weight matrix of the first network layer of the inference model, the second threshold is any one of M values arranged at the end in a descending order of absolute values of the weight values in the weight matrix of the first network layer of the inference model, and M is an integer greater than or equal to 1. In a possible implementation, the inference model compression apparatus further includes a quantization module configured to perform a quantization operation on the sparse inference model to obtain a quantized inference model. In a fourth aspect, the present application further provides a chip, which comprises an interface circuit and a power supply circuit; the interface circuit is configured to acquire resource occupation of a plurality of managed work nodes, and the control circuit is configured to execute the method in the first aspect or any possible implementation manner of the first aspect. In a fifth aspect, the present application provides a computing device cluster. The computing device cluster comprises at least one computing device; each computing device in the at least one computing device comprises a memory and a processor, the memory is configured to store computer instructions; and the processor is configured to execute the computer instructions to implement the method in the first aspect or any possible implementation manner of the first aspect. The computing device can refer to a server, a processor included in the server, etc. In a sixth aspect, the present application further provides a computer readable storage medium. The storage medium stores computer programs or instructions, and when the computer programs or instructions are executed by a computing device, the method in the first aspect or any possible implementation manner of the first aspect is implemented. In a seventh aspect, the present application further provides a computer program product. The computer program product comprises computer programs or instructions, and when the computer programs or instructions are executed by a computing device, the method in the first aspect or any possible implementation manner of the first aspect is implemented. The beneficial effects of the above second aspect to seventh aspect can refer to the description of the first aspect or any implementation manner of the first aspect, which will not be described here. On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a flowchart of a loop pruning and quantization model; FIG. 2 is an architecture diagram of a model compression system provided by the present application; FIG. 3 is a flow diagram of a reasoning model compression method provided by the present application; FIG. 4 is a quantization mapping diagram provided by the present application; FIG. 5 is a flow diagram of a reasoning model sparsification method provided by the present application; FIG. 6 is a flow diagram of a reasoning model sparsification method provided by the present application; FIG. 7 is a flow diagram of outlier identification and protection provided by the present application; FIG. 8 is a flow diagram of multi-objective optimization provided by the present application; FIG. 9 is a flow diagram of a weight matrix compression method in a model provided by the present application; FIG. 10 is a flow diagram of a reasoning model compression method provided by the present application; FIG. 11 is a flow diagram of a model deployment method provided by the present application; FIG. 12 is a flow diagram of a model inference method provided by the present application; FIG. 13 is a structural diagram of a reasoning model compression device provided by the present application; FIG. 14 is a structural diagram of a reasoning model compression device provided by the present application; FIG. 15 is a structural diagram of a computing device provided by the present application. DETAILED DESCRIPTION To facilitate understanding, first, the technical terms involved in the present application are introduced. LLM refers to a complex model built using deep learning technology that can process and understand natural language. These models are trained on large amounts of text data and can generate, understand and process various tasks of human language, such as text generation, translation, question answering, etc. The characteristics of LLM are large in size, containing tens of billions, hundreds of billions, or even thousands of billions of parameters, helping them learn complex patterns in language data. LLMs have shown strong performance in various NLP tasks, such as text generation, sentiment analysis, named entity recognition, machine translation, etc. They can learn the structure and rules of language from massive text data, and thus perform well in various language understanding tasks. Common network structures of LLMs are GPT (generative pre-trained transformer) or BERT (bidirectional encoder representations from transformers) and the like. Among them, GPT is a large language model based on the Transformer architecture, and BERT is a bidirectional encoding model based on the Transformer architecture. The Transformer mainly consists of the following parts: self-attention, positional encoding, encoder and decoder, multi-head attention, feed-forward neural network, residual connection, and layer normalization. Due to the large size of LLM, when deploying LLM to a computing device, the computing device needs to have more computing power and larger storage space, etc. When deploying LLM to an end-side host or embedded device, etc., the host or embedded device has limited computing power and storage space, and LLM cannot be completely deployed to the host or embedded device. Therefore, LLM needs to be compressed before being deployed to the host or embedded device. Model compression refers to reducing the size, parameter quantity or computational complexity of a model through a series of technical means, in order to reduce its resource consumption while maintaining its performance as much as possible. Model compression can help speed up the inference process, reduce memory usage, and reduce energy consumption, and is particularly suitable for resource-constrained environments such as mobile devices, embedded systems and edge computing. Common technical means include pruning, quantization, model distillation, transfer learning, etc. Among them, pruning (Pruning) means reducing the size and computational complexity of a model by removing redundant connections or parameters in a neural network to achieve model compression. Pruning can include structured pruning and unstructured pruning. Structured pruning is to directly delete a certain network layer, channel or filter in the model, and unstructured pruning is to delete or set parameters in the model to zero. For example, sparsity is a processing method in unstructured pruning, which sets some parameters (such as the values in the weights) in the model to zero or close to zero to reduce the parameter quantity of the model. Quantization is a technique to reduce the storage space and computational overhead of a model by reducing the number of bits of the parameters in the model. In traditional deep learning models, parameters are usually represented as floating-point numbers, such as 32-bit floating-point numbers (float32) or 64-bit floating-point numbers (float64). However, these floating-point number representations occupy a large storage space and require more computational resources for operations. The main idea of quantization is to convert the parameters in the model from floating-point number representations to lower-bit number representations. Lower-bit numbers are usually 8-bit integers (int8), 4-bit integers (int4), or even lower-bit representations. This can greatly reduce the storage space of the model and enable more efficient computation on hardware, accelerating the inference process of the model. Currently, as the size of artificial intelligence models continues to increase, the results of artificial intelligence models processing data are becoming more and more accurate. The increase in the size of artificial intelligence models will increase the difficulty of deployment, such as the increasing demand for hardware resources (computational resources or storage resources, etc.). However, in some application scenarios that require lightweight models or are sensitive to latency, such as intelligent robots or embedded devices, reducing the hardware resources required by artificial intelligence models can reduce the difficulty of deploying artificial intelligence models. In view of the above, the following provides two processing schemes for reducing the hardware resources required by artificial intelligence models to reduce the difficulty of deploying artificial intelligence models. Processing scheme 1 adopts cyclic pruning and quantized models until the compression rate of the model reaches the target compression rate, thereby reducing the number of parameters of the model and achieving reduced model running amount and required storage space. The artificial intelligence model can also be referred to as a model. As shown in FIG. 1, FIG. 1 is a flowchart of cyclic pruning and quantized models. The contents shown in FIG. 1 include the following steps ①-⑦. Step ①: Structurally prune 10% of the parameters in the model. Step ②: Train the pruned model until the accuracy of the pruned model processing data reaches a threshold. Step ③: Repeat the contents of step ① and step ② until about 80% of the parameters in the pruned model are obtained to obtain a pruned model. Step ④: Convert 50% of the parameters in the pruned model from floating-point number representation to 8-bit integer representation to obtain a quantized model. Step ⑤: Train the quantized model to adjust the parameters in the quantized model that have not been converted. Step ⑥: Convert 10% of the parameters in the model obtained in step ⑤ from floating-point number representation to 8-bit integer representation. Step ⑦: Repeat the contents of step ⑤ and step ⑥ until all parameters in the quantized model are converted to 8-bit integer representation, and the cyclic pruning and quantized model is obtained. The above parameters are the weights of the network layers of the model. However, in the above processing scheme 1, the model is compressed by using the structured pruning method. Since the structured pruning method usually prunes a certain network layer, channel or filter in the model, the pruning granularity is large. In the case of high model compression rate, part of the features in the model will be directly deleted, resulting in large loss of model accuracy and significant decrease of model processing accuracy. In processing scheme 2, the model is compressed by using the quantized sparse principal component analysis (QSPCA) technology. First, during the training of the model, the weight tensor of the model is decomposed by principal component analysis to obtain a dense matrix and a sparse matrix. Principal component analysis decomposition is a method of mathematically decomposing the weight tensor into two matrices, one of which represents the main components and the other represents the secondary components. The dense matrix is the main component, and the sparse matrix is the secondary component. Then, the dense matrix and the sparse matrix are quantized. Various quantization algorithms can be used, such as fixed-point quantization, floating-point quantization or binarization, to map the values in the matrix to a smaller representation. Finally, the quantized dense matrix and sparse matrix are stored to replace the original weight tensor, thereby achieving compression of the model. However, if the above weight tensor is an M x N matrix, the dense matrix and sparse matrix obtained by decomposing the weight tensor using principal component analysis are also M x N matrices. By independently quantizing two dense matrices and sparse matrices of the same size (number of rows and columns), compared with the cyclic pruning and quantized model shown in the above processing scheme 1, the compression multiple is limited, i.e., it cannot achieve a high compression rate. Based on this, the application provides a reasoning model compression method. The method can be applied to a computer system, which can be a server or a terminal, etc. The reasoning model includes multiple network layers. The model compression method includes: the computer system inputs reasoning data into the reasoning model and executes the reasoning model to compress the reasoning model. For a first network layer in the reasoning model, the computer system obtains a weight matrix of the first network layer and input data of the first network layer, calculates the product of the weight matrix of the first network layer and the input data of the first network layer in the reasoning model to obtain a result matrix. The computer system selects a weight value ranked after a% in the weight matrix according to the result matrix indicating the importance of the weight value in the weight matrix in descending order of importance, and performs a sparse operation on the weight value, and obtains a sparse reasoning model after performing the sparse operation on the multiple network layers. The first network layer is any one of the multiple network layers. The result matrix has the same size as the weight matrix, and the size of the value in the same position in the result matrix indicates the importance of the weight value in the same position in the weight matrix. The a% is the sparsity of the reasoning model. In the application, the result matrix indicating the importance of the weight value in the weight matrix is used to sparsify the weight value with lower importance (ranked after a% in descending order of importance) in the weight matrix. The weight matrix of the sparse reasoning model retains the weight value with higher importance, and therefore, the foregoing sparse operation has little effect on the reasoning accuracy of the reasoning model, and the reasoning accuracy loss of the obtained sparse reasoning model is small. In addition, the sparse operation is to set part of the weight values in the weight matrix to 0, and therefore, the sparse reasoning model is compressed compared with the reasoning model. Therefore, the reasoning accuracy loss of the sparse reasoning model is reduced while achieving a higher compression rate of the obtained sparse reasoning model. Next, the model compression method provided by the application will be described in detail in combination with the drawings. First, referring to FIG. 2, FIG. 2 is an architecture schematic diagram of a model compression system provided by the application. As shown in FIG. 2, the model compression system includes a reasoning device 210, a database 220, a terminal device 230 and a data collection device 240. The reasoning device 210 can be a server or a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera or a vehicle-mounted terminal, etc. It can also be an edge device, a smart robot (for example, a box with a processing-capable chip), etc. The intelligent robot includes a patrol robot and an interactive robot. The application scenario of the vehicle terminal is in automatic driving. As a possible embodiment, the inference device 210 can be different processors deployed on different physical devices (such as servers or servers in a cluster). For example, the inference device 210 can be a graphic processing unit (GPU), a central processing unit (CPU), a neural network processing unit (NPU), other general-purpose processors, a digital signal processing (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The data acquisition device 240 is configured to acquire training data or inference data and store the training data or inference data in the database 220. The data acquisition device 240 and the inference device 210 can be the same or different devices. The training data or inference data includes at least one form of data such as images, voices and texts. For example, the training data or inference data includes texts belonging to different scenarios. In some embodiments, the inference device 210 can configure the compressed neural network 201 to itself and perform inference using the compressed neural network 201. For example, the inference device 210 can perform various types of inference tasks such as processing images, voice processing, and conversing with a user in response to text / voice input from the user. In other embodiments, the inference device 210 can configure the compressed neural network 201 to other computing devices to run the compressed neural network 201 on the other devices. In combination with the model compression system, the inference model compression method provided in the embodiment can be applied when a large-parameter model is deployed on a device with limited hardware resources, to reduce the parameter quantity of the model and reduce the hardware resources required by the model. It should be noted that in actual applications, the training data or inference data (for example, text data in multiple scenarios) maintained in the database 220 can not all come from the data acquisition device 240, but can also be received from other devices. Further, according to the functions performed by the inference device 210, the inference device 210 can be further subdivided, as shown in FIG. 2, the inference device 210 is configured with a computing module 211, an I / O interface 212 and a preprocessing module 213. The I / O interface 212 is used for data interaction with external devices. The user can input data to the I / O interface 212 through the terminal device 230. The input data can include voice or text or image, etc. In addition, the input data can also come from the database 220. The preprocessing module 213 is used for preprocessing according to the input data received by the I / O interface 212. For example, the preprocessing module 213 can be used to improve the clarity of the voice or image received from the I / O interface 212, etc. During the preprocessing of the input data by the inference device 210, or during the processing of the computing module 211 of the inference device 210, etc., the inference device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data and instructions, etc. obtained by the corresponding processing into the database 220. For example, the neural network 201 stored by the inference device 210 can be applied to the inference device 210. After receiving the inference data, the computing module 211 compresses the neural network 201, such as sparsification, to obtain the compressed neural network 201. Then the computing module 211 can process the user input data using the compressed neural network 201 to obtain the processing result. Finally, the I / O interface 212 returns the processing result to the terminal device 230, so as to provide the user with the processing result for the user to view the processing result. It should be understood that the terminal device 230 and the inference device 210 can also be the same physical device. In the case shown in FIG. 2, the user can manually give the data to be processed, which can be operated through the interface provided by the I / O interface 212. In another case, the terminal device 230 can automatically send the data to be processed to the I / O interface 212, and if the terminal device 230 is required to automatically send the data to be processed, the authorization of the user is required, the user can set the corresponding permission in the terminal device 230. The user can view the processing result output by the inference device 210 in the terminal device 230, and the specific presentation form can be display, sound, action, etc. The terminal device 230 can also be used as a data acquisition end to collect the input data input into the I / O interface 212 and the processing result output from the I / O interface 212 as new sample data and store them into the database 220. Of course, the input data input into the I / O interface 212 and the processing result output from the I / O interface 212 can also be stored into the database 220 as new sample data without being collected through the terminal device 230. FIG. 2 is merely an architectural schematic diagram of a model compression system provided by an embodiment of the present application, and the positional relationship and quantity between the devices, elements, modules, etc. shown in FIG. 2 shall not constitute any limitation. For example, more computing modules 211 or more inference devices 210 can be included in FIG. 2. When the model compression system includes more inference devices 210, each of the plurality of inference devices 210 can compress at least one network layer in the inference model. For example, the inference model includes 5 network layers, and the model compression system includes 5 inference devices 210, each of the 5 inference devices 210 is configured to compress one network layer in the inference model. The present application shows a possible implementation of an inference model compression method, which can be applied to the model compression system shown in FIG. 2. As shown in FIG. 3, FIG. 3 is a flow schematic diagram of an inference model compression method provided by the present application, and the following takes the computer system 310 executing the inference model compression method as an example for illustration. The computer system 310 can include one or more inference devices 210 shown in FIG. 2, and the inference model can include a plurality of network layers. As shown in FIG. 3, the inference model compression method can include the following steps S310-S340. S310, the computer system 310 inputs inference data into the inference model and executes the inference model. The computer system 310 obtains the inference model and the inference data. In a possible case, the inference data is used to compress the inference model. In a possible example, the inference model can be a model with a parameter quantity greater than a threshold (e.g., 100 billion or 1,000 billion). For example, LLM or large image model, etc. The parameter quantity is the number of all weight values in the weight matrix of all network layers of the inference model. The LLM can include the GPT or BERT, etc. mentioned above, and the network structure of the large image model can include a series of convolutional neural networks (CNN), such as residual network (ResNet), VGGNet (visual geometry group network), etc. It is worth noting that the above threshold is merely an example and shall not be construed as a limitation of the present application. In other embodiments of the present application, the threshold can also be a larger or smaller value. In a possible case, the inference model is a pre-trained model. In a possible example, the inference data is used to determine the importance of each weight value of the weight matrix of each network layer in the inference model in the process of compressing the inference model, and then the weight values in the weight matrix are sparsified according to the importance. The sparsification is used to represent setting the weight values to 0. In a possible case, the type of the inference model matches the type of the inference data. In other words, after the inference data is input into the inference model, the inference model can process the inference data to obtain an accurate result. For example, if the inference model is used for text processing, the inference data is text data, such as text set 1, text set 2, and the like. If the inference model is used for image recognition, the inference data is image data. If the inference model is used for voice and video, the inference data is audio data, and the like. In a possible implementation, the following provides three possible examples for the computer system 310 to obtain the content of the inference model. In a first possible example, the computer system 310 obtains the inference model input by a user. In a second possible example, the computer system 310 obtains the inference model after training an initial model. The initial model is consistent with the network structure of the inference model, and only the parameters are different. In a third possible example, the computer system 310 obtains the inference model from the database 220 in FIG. 2. In a possible implementation, the following provides two possible examples for the computer system 310 to obtain the content of the inference data. In a first possible example, the computer system 310 obtains the inference data input by a user. In a second possible example, the computer system 310 obtains the inference data from the database 220 in FIG. 2. In a possible case, the inference data is vectorized data. The inference data is data conforming to a feature distribution, for example, a plurality of data included in the inference data conforms to a normal distribution, a uniform distribution, a skewed distribution, and the like. For example, when the inference data is text data and the inference data is uniformly distributed, 25% of the data included in the inference data has a label of "happy", 25% of the data has a label of "disappointed", 25% of the data has a label of "sad", and 25% of the data has a label of "excited". The computer system 310 inputs the inference data conforming to the same feature distribution into the inference model, and the similarity of the result matrix of the network layer in the obtained inference model is greater than or equal to a numerical value 1. The numerical value 1 can be 95% or 98%, and the like, which is not limited in the present application. For example, the computer system 310 inputs the inference data 1 and the inference data 2 of the same feature distribution into the inference model respectively, obtains a result matrix A of a network layer 1 in the inference model, and a result matrix B of a network layer 2 in the inference model, and the similarity of the result matrix A and the result matrix B is greater than or equal to a numerical value 1. The similarity can be a mean-square error (MSE) or a kullback-leibler divergence (KL divergence) of numerical values in the two result matrices, and the network layer 1 and the network layer 2 are the same layer in the inference model. In a possible implementation, the computer system 310 inputs the inference data into the inference model and executes the inference model, including: The computer system 310 inputs the inference data into a first network layer of the inference model, and then the inference data is processed by the multiple network layers included in the inference model in sequence, so that each network layer in the multiple network layers has corresponding input data. In a possible embodiment, if the computer system 310 includes only one inference device 210, the centralized model compression is implemented. In another possible embodiment, if the computer system 310 includes multiple inference devices 210, each inference device 210 in the multiple inference devices 210 is configured to compress at least one network layer in the inference model, so that the distributed model compression is implemented. S320, for a first network layer in the inference model, the computer system 310 obtains a weight matrix of the first network layer and input data of the first network layer. The first network layer is any one of the multiple network layers. Since the inference model is a pre-trained model, each network layer included in the inference model has a corresponding trained weight matrix. The computer system 310 obtains the weight matrix of each network layer included in the inference model. After the inference data is input into the inference model for execution, the inference data is processed by the multiple network layers included in the inference model in sequence, and each layer has corresponding input data. For example, the input data of the first network layer is the inference data, the input data of the second network layer is the output data of the first network layer after processing the inference data, and so on, to obtain the input data of each network layer in the multiple network layers. S330, the computer system 310 calculates the product of the weight matrix of the first network layer in the inference model and the input data of the first network layer, and obtains a result matrix. The result matrix and the weight matrix are of the same size, and the size of the numerical value in the same position of the result matrix indicates the importance of the weight value in the same position of the weight matrix. The computer system 310 determines a result matrix obtained by multiplying the weight matrix of the first network layer in the inference model with the input data of the first network layer. The input data of the first network layer in the inference model is the data input to the first network layer after the inference data is input to the inference model. The inference model can include multiple network layers, such as network layer 1, network layer 2, …, and network layer n. Each network layer in the multiple network layers has a corresponding weight matrix. Since the inference model has multiple network layers, the input data received by each network layer in the multiple network layers is different. In a possible example, after the inference data is input to the inference model, the inference data is sequentially processed by each network layer in the inference model. After each network layer processes the data, intermediate data is obtained, which is passed to the next network layer for processing. For example, the input data received by the first network layer is the data input by a user, such as the inference data described above. Except for the first network layer in the inference model, the input data received by other network layers is the output data of the previous network layer. In a possible case, in the same network layer, the value in the same position in the result matrix is used to indicate the importance of the weight value in the same position in the weight matrix. For example, the higher the value in the first position in the result matrix of network layer A, the higher the importance of the weight value in the first position in the weight matrix of network layer A. The lower the value in the first position in the result matrix of network layer A, the lower the importance of the weight value in the first position in the weight matrix of network layer A. For example, the result matrix of network layer A is: The weight matrix of network layer A is: The first position in the result matrix of network layer A is the first row and the first column, and the corresponding value is 4. Similarly, the first position in the weight matrix of network layer A is the first row and the first column, and the corresponding weight value is 4. For the content that the higher the value in the first position in the result matrix, the higher the importance of the weight value in the first position in the weight matrix, refer to the description in S340 below, which is not repeated here. In a possible implementation, the computer system 310 calculates the product of the weight matrix of the first network layer in the inference model and the input data of the first network layer to obtain a result matrix, including: The computer system 310 multiplies the weight matrix of the first network layer in the inference model with the norm of the input data of the first network layer to obtain a result matrix. The norm of the input data refers to a preset manner measurement operation on the amplitude of all elements in the input feature vector corresponding to the input data. The norm is a way to measure the size of a vector, usually represented as ||x||, where x is the vector. The preset manner measurement can be Manhattan norm (L1 norm): the sum of the absolute values of each element in the vector, i.e. ||x||1 = |x1| + |x2| +... + |xn|. Euclidean norm (L2 norm): the square root of the sum of the squares of the absolute values of each element in the vector, i.e. ||x||2 = √(x1 2 +x2 2 +...+xn 2 ). Infinity norm: the maximum value of the absolute values of the elements in the vector, i.e. ||x||∞ = max(|x1|, |x2|,..., |xn|). It is worth noting that if the input data is represented as a vector matrix, the norm of the input data can be represented as a vector. Taking the first layer network layer in the inference model as an example, the computer system 310 multiplies the weight matrix of the first layer network layer with the norm of the inference data to obtain the result matrix of the first layer network layer. The weight matrix of the first layer network layer can be The inference data can be

[1283] . The computer system 310 multiplies the weight matrix with the L2 norm of the inference data to obtain the result matrix of the first layer network layer Taking the second layer network layer in the inference model as an example, the computer system 310 multiplies the weight matrix of the second layer network layer with the norm of the input data of the second layer network layer to obtain the result matrix of the second layer network layer. The input data of the second layer network layer is the output data of the first layer network layer. For example, the output data of the first layer network layer can be the result of multiplying the weight matrix of the first layer network layer with the inference data. The description of the result matrix of the network layer after the second layer network layer in the inference model can refer to the description of the result matrix of the second layer network layer, which is not repeated here. It is worth noting that when multiplying the weight matrix of the first layer network layer with the norm of the inference data, the inference data is vectorized data, i.e. the inference data is a vector or a vector matrix. By the implementation manner, the result matrix of each network layer in the inference model is determined, and then the importance of each weight value in the weight matrix of each network layer in the inference model can be obtained according to the numerical value in the result matrix. The higher the numerical value in the result matrix is, the higher the importance of the weight value at the corresponding position is, and the lower the numerical value is, the lower the importance of the weight value at the corresponding position is. In the present application, since the result matrix is obtained by multiplying the weight matrix of the first network layer in the inference model and the norm of the input data of the first network layer, the norm is used to measure the size of the vector, and therefore the numerical value obtained by multiplying the weight matrix and the norm of the input data can indicate the influence degree of the weight value in the weight matrix on the output data of the network layer. The greater the numerical value is, the greater the influence degree is, and therefore the higher the importance of the weight value in the weight matrix is. The computer system 310 takes the numerical value in the result matrix as an index for judging the importance of the weight value in the weight matrix, and then performs sparsification on the weight matrix according to the importance index of the weight value in the weight matrix, thereby reducing the granularity of sparsification and avoiding sparsifying the weight with high importance, and reducing the influence of the sparsification process on the inference accuracy of the inference model. S340, the computer system selects the weight values ranked after a% in the weight matrix of the first network layer in the order of importance from large to small to perform the sparsification operation, and obtains the sparsified inference model after performing the sparsification operation on the plurality of network layers. Wherein, a% is the sparsity of the inference model. The computer system 310 sparsifies the weight matrix of the network layer in the inference model according to the result matrix indicating the importance of the weight value in the weight matrix, and obtains the sparsified inference model. In the weight matrix of the first network layer of the sparsified inference model, the value of the non-zero weight value at the same position in the first result matrix corresponding to the first network layer is greater than or equal to the value of the zero weight value at the same position in the first result matrix. In a possible example, since the numerical value in the result matrix of the network layer A indicates the importance of the weight value in the weight matrix of the network layer A. The computer system 310 can select the weight values ranked after a% in the weight matrix of the network layer A in the order of importance from large to small according to the importance indicated by the numerical value in the result matrix to perform the sparsification operation (i.e. set to 0). Except for the aforementioned weight values ranked after a% in the weight matrix of the network layer A, other weight values are not changed, thereby obtaining the sparsified network layer A. The computer system 310 obtains the sparsified inference model by performing the sparsification operation on the weight matrix of the network layer A as described above on the plurality of network layers included in the inference model. It is worth noting that the sparsity rates corresponding to the plurality of network layers included in the inference model are not the same. For example, the weight matrix of the network layer A in the inference model is: The weight matrix of the network layer A in the sparse inference model is The first result matrix of the network layer A in the inference model is In the above example, the value in the same position of the first result matrix of the network layer A indicates the importance of the weight value in the same position of the weight matrix of the network layer A in the inference model, and then the computer system 310 retains the weight of position 1 in the weight matrix of the network layer A in the inference model and sets the weight of position 2 to 0. The position 1 is the position of the value arranged in the front (1-a) % in the order from large to small in the result matrix, and the position 2 is the position of the value arranged in the rear a % in the order from large to small in the result matrix. Therefore, in the weight matrix of the network layer A in the sparse inference model, the non-zero weight values (such as 4, 1, -2, -1, -3, 2) are greater than or equal to the value of the zero weight value (such as 0) in the same position of the first result matrix of the network layer A. The network layer A can be any one of the plurality of network layers included in the inference model, and each network layer A corresponds to a first result matrix. For example, in the weight matrix of the network layer A in the sparse inference model, the non-zero weight value (such as 4) in the first row and the first column is greater than or equal to the value of the zero weight value (such as 0) in the first row and the second column in the first result matrix. For another example, in the weight matrix of the network layer A in the sparse inference model, the non-zero weight value (such as 1) in the first row and the third column is greater than or equal to the value of the zero weight value (such as 0) in the first row and the fourth column in the first result matrix. For the content of S340, the following provides a possible embodiment. The computer system 310 sparsifies the weight matrix of the network layer in the inference model according to the result matrix and the sparsity, to obtain a sparse inference model. The ratio of the number of zero weight values in the weight matrix of the network layer in the sparse inference model to the number of all weight values in the weight matrix is less than or equal to the sparsity. In a possible case, the sparsity of each network layer of the inference model can be different or the same. In a possible implementation, the computer system 310 can determine the number of weight values that need to be sparsified in the weight matrix of the network layer according to the product of the sparsity and the number of all values in the weight matrix of the network layer of the inference model, and then determine the number of weight values that need to be sparsified from the weight matrix and sparsify to obtain a sparse inference model. In the present application, the computer system 310 sparsifies the weight matrix of each network layer in the inference model according to the sparsity of each network layer in the inference model and the importance indicated by each value in the result matrix, reduces the granularity of sparsification, avoids sparsifying weights with high importance, reduces the influence of the sparsification process on the processing accuracy of the model, and reduces the inference accuracy loss of the sparsified inference model. For specific contents of the above implementation manners, refer to the contents shown in FIG. 5 or FIG. 6 below, which will not be repeated here. In a possible embodiment, the inference model compression method shown in FIG. 3 can further include the following content: the computer system 310 quantizes the sparsified inference model to obtain a quantized inference model. For example, the weight values in the weight matrix of the quantized inference model can be represented as int8 or int4 integer values. It is worth noting that the weight values in the weight matrix of the quantized inference model represented as int8 or int4 integer values are only examples and should not be construed as a limitation of the present application. In other embodiments of the present application, the number of bits of the weight values in the weight matrix of the quantized inference model is lower than the number of bits of the weight values in the weight matrix of the sparsified inference model, and the number of bits of the weight values in the weight matrix of the quantized inference model can be more or less. The lower the number of bits of the weight values in the weight matrix of the quantized inference model, the higher the compression rate of the model, thereby reducing the consumption of the obtained quantized inference model in memory and computing resources. In a possible implementation manner, the computer system 310 uniformly quantizes the sparsified inference model to obtain a quantized inference model. As shown in FIG. 4a, FIG. 4 is a quantization mapping provided by the present application. As shown in FIG. 4a, uniform quantization refers to dividing the weights in the weight matrix of the network layer of the model into discrete values according to the same interval, thereby reducing the number of bits of the weight values in the weight matrix, thereby reducing the consumption of the obtained quantized inference model in memory and computing resources. For example, the computer system 310 can quantize the weight values from floating-point values to 8-bit integer values, and all the weight values use the same quantization range and interval. In another possible implementation manner, the computer system 310 non-uniformly quantizes the sparsified inference model to obtain a quantized inference model. As shown in FIG. 4b, non-uniform quantization refers to quantizing the weight values according to the distribution of the weight values in the weight matrix of the network layer of the model using different quantization ranges and intervals. Non-uniform quantization can better adapt to the dynamic range of the weight values in the weight matrix, thereby reducing the inference accuracy loss of the model. For example, the computer system 310 can perform different quantization processing on the weight values in the weight matrix according to the statistical characteristics or importance of the weight values, so that important weight values have higher precision and unimportant weight values have lower precision. For the computer system 310 to sparsify the weight matrix of the network layer in the inference model according to the result matrix and the sparsity, the content of the sparsified inference model is obtained, and two possible examples are provided below. In the first possible example, as shown in FIG. 5, FIG. 5 is a flowchart of an inference model sparsification method provided by the present application. The content shown in FIG. 5 includes the following steps S510 to S520. S510, the computer system 310 determines all values in the result matrix of the network layer of the inference model, and the last N values in the second preset order. Wherein, N is the product of the sparsity and the number of all values in the weight matrix of the network layer of the inference model. In a possible case, the second preset order is descending order. In a possible case, since the sparsity of each network layer of the inference model can be different, the N values corresponding to each network in the inference model can also be different. Taking the network layer B in the inference model as an example, the computer system 310 determines all values in the result matrix of the network layer B, and the last N values in descending order. For example, the computer system 310 reads all values in the result matrix of the network layer B, obtains (4, 0, 8, 3, 3, 4, 8, 9, 3, 2, 0, 6), and determines the last 3 values in descending order of all values, such as 0, 0, and 2. The 3 in the aforementioned “last 3 values” is determined according to the product of the sparsity and the number of all values in the weight matrix (such as 12). For example, the computer system 310 can arrange all values in descending order, and then determine the values arranged in the last 3 positions. S520, the computer system 310 sets the weights at the same positions in the weight matrix of the network layer to 0, and obtains the sparsified inference model. Since the size of the result matrix of the same network layer is consistent with the size of the weight matrix, and the positions of the result matrix of the same network layer and the weight matrix have a one-to-one correspondence, the computer system 310 can determine that the last N values correspond to the same positions in the weight matrix of the network layer based on the last N values, and then set the weight values at the same positions in the weight matrix to 0, while the weight values at other positions in the weight matrix remain unchanged, thereby obtaining the sparsified inference model. It is worth noting that the computer system 310 sparsifies each network layer of the inference model according to the sparsity requirement of each network layer of the inference model, and then obtains the sparsified inference model. In a possible example, the computer system 310 sets the weight values of the same positions in the weight matrix of the network layer to 0 according to the last N values, to obtain the sparse inference model, including: the computer system 310 determines the second positions of the last N values in the result matrix, and then sets the weight values of the second positions in the weight matrix of the network layer to 0, to obtain the sparse inference model. For example, the weight matrix of the network layer B is The result matrix of the network layer B is The computer system 310 determines that the last three values in descending order are 0, 0, and 2, and the corresponding positions in the result matrix are the first row and the second column, the third row and the second column, and the third row and the third column. The computer system 310 sets the weight values of the first row and the second column, the third row and the second column, and the third row and the third column in the weight matrix of the network layer B to 0, to obtain the weight matrix of the network layer B in the sparse inference model The computer system 310 performs the same sparse processing on each network layer in the inference model according to the content of the above example, to obtain the sparse inference model. In this application, the computer system 310 determines the weights (such as the weight values of the same positions in the weight matrix of the network layer according to the last N values) in the weight matrix of each network layer that need to be sparse according to the sparsity of each network layer in the inference model. Since the N values are all values in the result matrix, the last N values in descending order, that is, the last N values with smaller values, the importance of the N values is lower, and therefore, sparse processing on the N values with lower importance in the weight matrix can reduce the precision loss of the sparse inference model. In other examples of this application, the computer system 310 determines the last N values in a second preset order from all values in the result matrix, and then sets the weight values of the same positions in the weight matrix of the network layer to 0 according to the last N values, to obtain the inference model to be trained, so as to train the inference model to be trained, to obtain the sparse inference model. For the content of the computer system 310 obtaining the sparse model, reference can be made to the content shown in FIG. 5, which will not be described here. For example, the computer system 310 trains the inference model to be trained to obtain the sparse inference model, including: the computer system 310 inputs the training data into the inference model to be trained for training until the loss function of the model converges, or the loss function training is completed, or the set training round is reached, etc., to obtain the sparse inference model. In a second possible example, as shown in FIG. 6, FIG. 6 is a flowchart of a second inference model sparse method provided by this application. The content shown in FIG. 6 includes the following steps S610 to S620. S610, the computer system 310 arranges all the values in the result matrix of the network layer C in descending order, and determines the values located at the last a%. The a% is the sparsity of the network layer C. In a possible scenario, since the sparsity of each network layer in the inference model can be different, the sparsity of each network layer in the inference model can be different. Taking the network layer C in the inference model as an example, the computer system 310 reads all the values in the result matrix of the network layer C, and determines the values located at the last a% in descending order of all the values. For example, the computer system 310 reads all the values in the result matrix of the network layer C, and obtains (4, 0, 8, 3, 3, 4, 8, 9, 3, 2, 0, 6), and determines the values located at the last a% in descending order of all the values. If the a% is 25%, the values located at the last 25% are 0, 0, and 2. S620, the computer system 310 determines the position 1 of the values located at the last a% in the result matrix, and then sparsifies the weight value at the position 1 in the weight matrix of the network layer C, to obtain the sparse inference model. In a possible implementation, the computer system 310 determines the position 1 of the values located at the last a% in the result matrix, and then sets the weight at the position 1 in the weight matrix of the network layer to 0, and the weights at other positions remain unchanged, to obtain the sparse inference model. It is worth noting that the computer system 310 sparsifies each network layer in the inference model according to the sparsity requirement of each network layer in the inference model, to obtain the sparse inference model. The sparsity in the above two possible implementations can be set by a user, or determined by the computer system 310 in a multi-objective optimization manner. For the content of the computer system 310 determining the sparsity of each network layer in the inference model in a multi-objective optimization manner, reference can be made to the description of FIG. 8 below, which is not described herein again. In this application, the computer system 310 determines the weight values (i.e., the weight values corresponding to the values located at the last a% after arranging all the values in the result matrix in descending order) in the weight matrix of each network layer that need to be sparsified according to the sparsity of each network layer in the inference model. The weight values that need to be sparsified are the weight values with low importance, and thus sparsifying the weight values with low importance can reduce the loss of accuracy of the obtained sparse inference model. In a possible embodiment, there are some abnormal values in the weight matrix of the inference model, which have a greater impact on the accuracy of the model. Therefore, in order to ensure the accuracy of the model after sparsification, the abnormal values need to be identified before sparsification and protected when the model is sparsified, that is, the importance of the abnormal values is the highest. The following takes the computer system 310 identifying and protecting the abnormal values in the weight matrix of the second network layer of the inference model as an example. As shown in FIG. 7, FIG. 7 is a flowchart of the identification and protection of abnormal values provided by the present application. The content shown in FIG. 7 can be executed by the computer system 310. In this embodiment, the second network layer can also be referred to as network layer D. The network layer D is any one of the plurality of models included in the inference model. The content shown in FIG. 7 includes the following steps S710 to S730. S710, the computer system 310 obtains a target ratio. In a possible example, the computer system 310 obtains the target ratio input by the user. The target ratio is specified by the user, such as 1% or 0.5%, which is not limited in the present application. S720, the computer system 310 determines the weights in the weight matrix of the network layer D in the inference model that are greater than or equal to a threshold value a. Wherein, the threshold value a is any one of the last M values in the target ratio in the order from large to small in the absolute value of the weight value in the weight matrix of the network layer D of the inference model, and M is an integer greater than or equal to 1. In a possible case, the above-mentioned M can be specified by the user, or M is set to 1 by default. For example, the computer system 310 obtains M input by the user. In a possible implementation, the computer system 310 determines the weights in the weight matrix of the network layer D in the inference model that are greater than or equal to the threshold value a, including: the computer system 310 determines the absolute value of the weight value in the weight matrix of the network layer D, and then arranges any one of the last M values in the target ratio in the order from large to small in the absolute value of the weight value as the threshold value a, and determines the weights in the weight matrix of the network layer D that are greater than or equal to the threshold value a. For example, the computer system 310 determines the absolute value of the weight value in the weight matrix of the network layer D, and then arranges the absolute values of the weight values in the weight matrix in descending order, and determines any one of the last M values in the target ratio in the arrangement as the threshold value a, thereby determining the weights in the weight matrix of the network layer D that are greater than or equal to the threshold value a. For example, the weight matrix of the network layer D is The computer system 310 takes the absolute values of the weight values in the weight matrix to obtain (4 0 1 1 3 2 1 3 3 1 0 2), and then arranges the absolute values of the weight values in descending order to obtain (4 3 3 3 2 2 1 1 1 1 0 0), and determines that the smallest one of the values arranged in the front 8% in the foregoing arrangement is 4. The computer system 310 determines that the weight greater than or equal to 4 in the weight matrix of the network layer D is 4 in the first row and the first column. In a possible case, the computer system 310 determines that the two smallest values arranged in the front 16.7% are 4 and 3, and the computer system 310 can select any one of 4 or 3 as the threshold a, such as determining that 3 is the threshold a. The computer system 310 determines that the weights greater than or equal to 3 in the weight matrix of the network layer D are 4 in the first row and the first column and 3 in the second row and the first column. S730, the computer system 310 determines a first mask of the weights greater than or equal to the threshold a in the weight matrix. The first mask is used to hide the weights greater than or equal to the threshold a in the weight matrix, that is, the weights under the first mask are fixed and unchanged when the model is sparse. The weights greater than or equal to the threshold a in the weight matrix are also outliers. The following gives a possible example of a first mask. In this example, the weight matrix is 4x3, and the matrix shown by the same first mask is also 4x3. For example, the first mask can be represented as: Wherein, 1 indicates that the weight in the corresponding position of the weight matrix is hidden, and 0 indicates that the weight in the corresponding position of the weight matrix is not hidden. Thus, there are the same weight values in the weight matrix of the network layer D in the inference model and the weight matrix of the network layer D in the sparse inference model, and the same weight values include the weight values greater than or equal to the threshold a in the weight matrix of the network layer D in the inference model. For example, according to the foregoing first mask, the computer system 310 hides the weights in the first row and the first column and the second row and the first column of the weight matrix when sparsifying the inference model, and does not perform sparsification. For another example, according to the foregoing contents shown in FIG. 5 or FIG. 6, the computer system 310 modifies the weight value at position 1 in the weight matrix of the network layer D in the sparse inference model to the weight value at position 1 in the weight matrix of the network layer D in the inference model according to the position 1 (the first row and the first column and the second row and the first column) where the “1” in the foregoing first mask is located, to realize that the weight at position 1 is hidden when sparsifying the inference model. The computer system 310 can protect the outliers, that is, the values with higher importance in the weight matrix, by the first mask, and the loss of inference accuracy of the sparse inference model obtained according to the foregoing sparsification process is small. The computer system 310 identifies the outliers in the weights of other network layers and the content to be protected. For details, refer to the description of the computer system 310 identifying and protecting the outliers in the weight matrix of the network layer D in the inference model. In this application, due to the outliers in the weight matrix of the network layer D, the influence on the processing of the input data by the network layer D to obtain the output data is relatively large. Therefore, when the computer system 310 sparsifies the weight matrix of the network layer D, the outliers are protected, that is, the outliers are not sparsified, which can reduce the accuracy loss of the obtained sparse inference model. For the content that the computer system 310 determines the sparsity of each network layer in the inference model by using the multi-objective optimization method, the following provides a possible embodiment. As shown in FIG. 8, FIG. 8 is a flowchart of the multi-objective optimization provided by the present application, and the content shown in FIG. 8 can be executed by the computer system 310. The content shown in FIG. 8 can include the following steps S810-S830. S810, the computer system 310 sparsifies each network layer in the inference model by using a plurality of set sparsities. In a possible implementation manner, the computer system 310 receives the set plurality of sparsities of each network layer in the inference model input by the user, and then sparsifies each network layer according to the set plurality of sparsities. In another possible implementation manner, the computer system 310 generates a plurality of sparsities of each network layer in the inference model. In a possible case, the random sparsity of each network layer is determined according to the target sparsity. In a possible example, the computer system 310 determines a plurality of sparsities within a preset range or a preset value from the target sparsity. For example, the target sparsity is 50%, and the preset value is 10%. The computer system 310 can determine a plurality of values such as 40%, 50%, and 60% in the range of 40%-60% as the random sparsity. The number of values can be set according to the user's demand. S820, the computer system 310 determines the sparsity error of each network layer in the inference model before and after sparsification. In a possible implementation manner, the computer system 310 respectively inputs the inference data into the inference model and the sparse inference model, obtains the corresponding output data a of each network layer in the inference model, and obtains the corresponding output data b of each network layer in the sparse inference model, and determines the sparsity error of each network layer in the inference model before and after sparsification according to the output data a and the output data b. For example, the computer system 310 determines the sparsity error according to the similarity between the output data a and the output data b. For example, the computer system 310 subtracts the similarity between the output data a and the output data b from 1 to obtain the sparsity error. The computer system 310 can calculate the MSE or the mean absolute error (MAE) between the output data a and the output data b to obtain the sparsity error described above. S830, the computer system 310 determines the sparsity rate of each network layer in the inference model in a multi-objective optimization manner when the sparsity rate of the inference model as a whole is greater than or equal to the target sparsity rate and the sum of the sparsity errors of all network layers in the inference model is less than or equal to the threshold b. In a possible case, the target sparsity rate and the threshold b are specified by a user. For example, the computer system 310 receives the target sparsity rate and the threshold b input by the user. In a possible case, the sparsity rate of the inference model as a whole is the sum of the sparsity rate of each network layer in the inference model and the product of the number of weight values included in each weight matrix of the network layer and the ratio of the number of weight values included in the weight matrix of each network layer to the number of weight values included in the weight matrix of all network layers in the inference model. In a possible implementation, the computer system 310 determines the sparsity rate of each network layer in the inference model in a multi-objective optimization manner when the sparsity rate of the inference model as a whole is greater than or equal to the target sparsity rate and the sum of the sparsity errors of all network layers in the inference model is less than or equal to the threshold b. The sparsity rate of each network layer determined by the computer system 310 is one of the plurality of sparsity rates set for the network layer in S810. For example, the computer system 310 uses a multi-objective optimization algorithm to continuously optimize the sparsity rate of each network layer in the inference model according to the sparsity error and the target sparsity rate, so as to achieve the sparsity rate of each network layer in the inference model when the sparsity rate of the inference model as a whole is greater than or equal to the target sparsity rate and the sum of the sparsity errors of all network layers in the inference model is less than or equal to the threshold b. The multi-objective optimization algorithm described above can be a multi-objective genetic algorithm (MOGA), a multi-objective particle swarm optimization (MOPSO), a multi-objective ant colony optimization (MOACO), a multi-objective differential evolution (MODE), or the like. The multi-objective optimization algorithm is taken as the multi-objective differential evolution algorithm as an example. The computer system 310 performs optimization search through a mutation operation based on a differential form and a crossover operation based on a probability selection to find an optimal solution or a frontier of the sparsity of each network layer in the inference model. The frontier represents a set of all optimal solutions under a given constraint condition (e.g., the sparsity of the entire inference model is greater than or equal to a target sparsity, and the sum of the sparsity errors of all network layers in the inference model is less than or equal to a threshold b). In a possible example, the computer system 310 solves the following formula according to a multi-objective optimization algorithm to determine the sparsity of each network layer in the inference model when the sparsity of the entire inference model is greater than or equal to the target sparsity and the sum of the sparsity errors of all network layers in the inference model is the minimum. where Error represents the sparsity error of the network layer, W represents the weight matrix of each network layer in the inference model, M is the sparse Mask corresponding to each network layer in the inference model, data is inference data input into the inference model, and ∑ layers sparse(W) represents the sparsity of the entire model, and L represents the target sparsity. The sparse Mask described above is used to indicate the sparsity of the network layer. It is worth noting that the data described above can also represent other data input into the inference model, which is not limited in the present application. In the present application, the computer system 310 determines the optimal sparsity of each network layer in the inference model in a manner of multi-objective optimization, under the condition that the sum of the sparsity errors of all network layers is less than or equal to the threshold b and the sparsity of the entire inference model is greater than or equal to the target sparsity. The computer system 310 sparsifies the weight matrix of the network layer according to the optimal sparsity of each network layer determined in the foregoing manner, to obtain a sparse inference model, and the inference accuracy loss of the sparse inference model is small. To further compress the model to reduce the memory required by the model, the present application further provides a possible embodiment. The embodiment shows that the computer system 310 compresses the weight matrix in the quantized inference model to obtain a compressed model, and the compressed model includes the content of the compressed weight matrix. As shown in FIG. 9, FIG. 9 is a flowchart of a compression method of the weight matrix in the model provided by the present application. The content shown in FIG. 9 can be executed by the computer system 310 described above. The content shown in FIG. 9 can include the following steps S910 to S920. S910, the computer system 310 acquires a compression request sent by a user. The compression request includes a field for indicating a compression mode. In a possible example, the compression mode includes entropy encoding or run-length encoding. It is worth noting that the above is only an example provided by the present application and should not be construed as a limitation of the present application. In other examples of the present application, the compression mode can also include dictionary encoding, differential encoding, transform encoding, or arithmetic encoding, etc. In one possible case, the field in the compression request is 0, indicating entropy encoding; the field in the compression request is 1, indicating run-length encoding. In one possible case, the compression request described above can be generated by the user's triggering operation on the control component indicating the compression mode on the user interface. The triggering operation can be a click operation, a sliding operation, etc. S920, the computer system 310 compresses the weight matrix in the quantized inference model using the compression mode indicated by the field to obtain a compressed inference model. The compressed inference model includes a compressed weight matrix. In one possible implementation, the computer system 310 compresses the weight matrix in the third model using entropy encoding or run-length encoding to obtain a fourth model. Take the computer system 310 compressing the weight matrix of network layer A in the quantized inference model as an example. The weight matrix of network layer A is The computer system 310 compresses the weight matrix of network layer A to obtain "604122", and stores "604122" in byte format. The computer system 310 can obtain the compressed inference model by compressing the weight matrix of all network layers in the quantized inference model. In one possible example, the computer system 310 calls the underlying compression interface to compress the weight matrix in the quantized inference model, thereby obtaining a compressed inference model. The compressed inference model includes a compressed weight matrix, which is stored in byte format. In the present application, the computer system 310 performs byte compression on the weight matrix of the quantized inference model, so that the weight matrix of the compressed inference model occupies less memory, which is conducive to deploying the compressed inference model to hardware resource-limited devices such as embedded devices, and reduces the hardware requirements for deploying models with large parameter quantities (number of weights in the weight matrix). In a possible embodiment, after obtaining the quantized inference model, the computer system 310 can identify the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model. If the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model is greater than a third threshold, the computer system 310 sends first information to the user interface; if the number of weight values that are zero in the weight matrix of each network layer in the third model is less than the third threshold, the computer system 310 sends second information to the user interface. The first information is used to indicate a compression mode with a compression rate less than a fourth threshold, and the second information is used to indicate a compression mode with a compression rate greater than the fourth threshold. In a possible case, the third threshold and the fourth threshold are set by the user as needed. For example, the computer system 310 receives the third threshold and the fourth threshold input by the user. The third threshold can be 10,000 or 5,000, and the fourth threshold can be 50%. The compression mode with a compression rate greater than 50% can be entropy coding, and the compression mode with a compression rate less than 50% can be run-length coding. The values of the third threshold and the fourth threshold are only examples and should not be construed as limiting the present application. In other embodiments of the present application, the third threshold can be greater than 10,000 or less than 10,000, and the fourth threshold can be greater than 50% or less than 50%. When the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model is equal to the third threshold, the computer system 310 can send the first information or the second information to the user interface according to the user's needs or configuration. The first information or the second information can also indicate a compression mode with a compression rate equal to the fourth threshold, such as differential coding, transform coding, and the like. In a possible case, the user interface can be a front-end display interface, and the front end can be a display of the computer system 310 or a display connected to the computer system 310. In a possible case, the present application does not limit the display form of the first information or the second information on the user interface. For example, the first information or the second information is displayed in the form of a pop-up window on the user interface. In the present application, the computer system 310 sends the first information or the second information to the user interface to realize the suggestion of the compression mode to the user through the first information or the second information, so as to improve the compression rate of the weight matrix of the quantized inference model. In a possible embodiment, the computer system 310 can identify the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model. If the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model is greater than a third threshold, the computer system 310 can compress the weight matrix of each network layer in the quantized inference model by using a compression manner with a compression rate less than a fourth threshold, to obtain a compressed inference model. If the number of weight values that are zero in the weight matrix of each network layer in the quantized inference model is greater than the third threshold, the computer system 310 can compress the weight matrix of each network layer in the quantized inference model by using a compression manner with a compression rate greater than the fourth threshold, to obtain a compressed inference model. The compressed inference model includes the compressed weight matrix. For the expression that the computer system 310 compresses the weight matrix of each network layer in the quantized inference model by using a compression manner to obtain a compressed inference model, refer to the content shown in FIG. 9, which will not be described herein again. In view of the content shown in FIGS. 3 to 9, a complete embodiment is provided as follows. As shown in FIG. 10, FIG. 10 is a flowchart of another method for compressing an inference model. The content shown in FIG. 10 includes the following steps ① to ⑦. Step ①: The computer system 310 obtains an inference model and inference data. For the specific content of step ①, refer to the expression of S310 in FIG. 3, which will not be described herein again. Step ②: The computer system 310 identifies the outliers in the weight matrix of each network layer in the inference model. For the specific content of step ②, refer to the content shown in FIG. 7 or FIG. 8, which will not be described herein again. Step ③: The computer system 310 determines the result matrix obtained by multiplying the weight matrix of each network layer in the inference model by the input data of the network layer. For example, the computer system 310 inputs the inference data into the inference model and executes the inference model, so as to obtain the weight matrix of a first network layer and the input data of the first network layer for the first network layer in the inference model. The first network layer is any one of the plurality of network layers. The result matrix has the same size as the weight matrix, and the size of the value in the same position in the result matrix indicates the importance of the weight value in the same position in the weight matrix. For the specific content of step ③, refer to the expression of S310-S330 in FIG. 3, which will not be described herein again. Step ④: The computer system 310 determines the sparsity of each network layer in the inference model by using a multi-objective optimization manner. For the specific expression of step ④, refer to the content shown in FIG. 8, which will not be described herein again. Step V: The computer system 310 spars the weight matrix of the network layer in the inference model according to the result matrix of the network layer and the sparsity, to obtain the sparse inference model. For example, the sparsification operation is performed on the weight values ranked at the last a% in the weight matrix of the first network layer according to the order of importance from large to small indicated by each value in the result matrix. The a% is the sparsity of the inference model. During the sparsification, the computer system 310 protects the outliers in the weight matrix of the network layer shown in step II, that is, does not sparsify the outliers in the weight matrix of the network layer. For the specific description of step V, refer to the content shown in S340 of FIG. 3, FIG. 5 or FIG. 6, which will not be repeated here. Step VI: The computer system 310 quantizes the sparse inference model to obtain the quantized inference model. Step VII: The computer system 310 compresses the weight matrix of the quantized inference model to obtain the compressed inference model. The compressed inference model includes the compressed weight matrix. For the specific description of step VII, refer to the content shown in FIG. 9, which will not be repeated here. It is worth noting that the execution order between the above steps is not limited in the present application. In other embodiments of the present application, the computer system 310 can first execute the content shown in step III, and then execute the content shown in step II. In one possible embodiment, the content of deploying the compressed inference model obtained through the content shown in FIG. 9 or FIG. 10 is shown. As shown in FIG. 11, FIG. 11 is a flowchart of the model deployment method provided by the present application. The method shown in FIG. 11 can be executed by the computer system 310, and the content shown in FIG. 11 includes the following steps S1110-S1130. S1110, the computer system 310 obtains a first model. The first model includes the compressed weight matrix. In one possible example, the compressed weight included in the first model is stored in byte format. In one possible case, the computer system 310 obtains the first model input by a user. In one possible case, the computer system 310 obtains the first model from a storage. The storage can be included in the plurality of hardware included in the computer system 310, or the storage is externally connected to the computer system 310. Exemplarily, the memory can be a high bandwidth memory (HBM), a double data rate synchronous dynamic random access memory (DDR), a Level 2 buffer (L2), etc. It is worth noting that the above content of the memory is only an example and should not be understood as a limitation of the present application. In other embodiments of the present application, the memory can also be a Level 3 buffer (L3). In a possible example, the computer system 310 obtains the compressed weight matrix of the first model from the L2. For more details of the first model, refer to the description of the compressed inference model in the above FIG. 9 or FIG. 10, which will not be repeated here. S1120, the computer system 310 decompresses the compressed weight matrix in the first model to obtain a second model. Wherein, the second model is obtained by sparsifying and quantizing the weight matrix of the first network layer in the third model according to the importance indicated by the result matrix obtained by multiplying the weight matrix of the first network layer of the third model with the input data of the first network layer. The result matrix and the weight matrix of the same network layer are of the same size, and the size of the value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix. The first network layer is any one of the multiple network layers included in the third model. For the content of the second model, refer to the description of the quantized inference model above, which will not be repeated here. For the content of the third model, refer to the description of the inference model shown in the above FIG. 3-FIG. 9, which will not be repeated here. In a possible implementation, the computer system 310 decompresses the compressed weight matrix in the first model to obtain the second model, including: The computer system 310 decompresses the compressed weight matrix in the first model to obtain the second model by using a decompression unit. The decompression unit is hardware in the computer system 310 for decompressing the weight matrix. Exemplarily, the computer system 310 converts the compressed weight matrix from byte format storage to uncompressed weight matrix stored in binary format. In a possible case, the weight in the weight matrix of the second model is an 8-bit integer type value or a 4-bit integer type value, and the number of bits of the weight in the weight matrix of the third model is greater than that of the weight in the weight matrix of the second model. It is worth noting that the specific content of the bit number of the weight in the weight matrix described above is only an example and should not be understood as a limitation of the present application. In other cases of the present application, the weight in the weight matrix of the second model can also be a 16-bit floating-point numerical value. S1130, the computer system 310 deploys the second model. In one possible implementation, the computer system 310 deploys the second model, comprising: the computer system 310 deploying the second model to a computing unit. The computing unit can be a processor core in the computer system 310 that performs inference according to input data, or a processor core connected to the computer system 310 for performing inference according to input data. In the case where the computer system 310 includes the above-mentioned processor core that performs inference according to input data, the following provides a possible embodiment. The computer system 310 stores the decompressed weight matrix to a Level 1 buffer (L1) in the computer system 310, and then the processor core obtains the weight matrix from the L1 to process the input data during the inference process. The L1 is the cache closest to the processor core. In relation to the content shown in FIG. 11 described above, the following provides a possible embodiment. As shown in FIG. 12, FIG. 12 is a flowchart of the model inference method provided by the present application. The content shown in FIG. 12 can include the following steps ①-⑤. Step ①: The computer system 310 obtains the compressed weight matrix of the first model from the L2. Step ②: The decompression unit in the computer system 310 decompresses the compressed weight matrix to obtain the weight matrix of the second model. Step ③: The computer system 310 saves the weight matrix of the second model to the L1. Step ④: The computing unit in the computer system 310 obtains input data. It is worth noting that the bit number of the input data and the bit number of the weight in the weight matrix need to be consistent. If the bit number of the input data and the bit number of the weight in the weight matrix are inconsistent, the bit number of the input data needs to be converted to make the bit number of the converted input data consistent with the bit number of the weight in the weight matrix. For example, if the bit number of the weight in the weight matrix is an 8-bit integer numerical value, and the bit number of the input data is a 16-bit floating-point numerical value, the computing unit in the computer system 310 will quantize the input data, and the bit number of the quantized input data is an 8-bit integer numerical value. Step V: The computing unit in the computer system 310 obtains the weight matrix of the second model from the L1, and processes the input data according to the weight matrix of the second model to obtain the final result. Taking the network layer B in the second model as an example, the network layer B uses the matmul operator to calculate the product of two matrices. After the computing unit inputs the input data into the second model, the input data is processed by each network layer in the second model in turn. If the input data is processed by the network layers before the network layer B in the second model, a tensor a is obtained, and at this time, the network layer B is used to process the tensor a, and then the computing unit obtains the weight matrix of the network layer B from the L1, multiplies the weight matrix of the network layer B with the tensor a to obtain the output data of the network layer B. The output data of the network layer B is transmitted to the next network layer of the network layer B in the second model, until the input data is processed by all the network layers in the second model to obtain the final result. In the foregoing, the inference model compression method provided in the present application is described in detail in combination with FIGS. 2 to 10. In the following, the inference model compression device provided in the present application will be described in combination with FIG. 13, which is a structural schematic diagram of an inference model compression device provided in the present application. The inference model compression device 1300 can be used to realize the functions of the computer system in the method embodiments described above, and thus can also realize the beneficial effects possessed by the method embodiments described above. As shown in FIG. 13, the inference model compression device 1300 includes an input module 1310, an obtaining module 1320, a computing module 1330 and a sparsing module 1340. The inference model compression device 1300 is used to realize the functions of the computer system in the method embodiments corresponding to FIGS. 2 to 10 described above. In one possible example, the inference model compression device 1300 is used to realize the specific process of the inference model compression method described above, which includes the following processes: The input module 1310 is configured to input inference data into an inference model and execute the inference model. The obtaining module 1320 is configured to, for a first network layer in the inference model, obtain a weight matrix of the first network layer and input data of the first network layer, where the first network layer is any one of a plurality of network layers included in the inference model. The computing module 1330 is configured to calculate the product of the weight matrix of the first network layer and the input data of the first network layer in the inference model to obtain a result matrix. The result matrix has the same size as the weight matrix, and the value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix. The sparse module 1340 is configured to perform a sparse operation on the weight values ranked after a percentage a in the weight matrix of the first network layer according to the order from large to small in importance, the percentage a being a sparsity of the inference model; and obtain the sparse inference model after performing the sparse operation on the plurality of network layers. To further implement the functions in the method embodiments shown in FIGS. 2-10, the present application also provides an inference model compression apparatus, as shown in FIG. 14, which is a structural schematic diagram of an inference model compression apparatus provided by the present application. The inference model compression apparatus 1300 further includes a quantization module 1350. The quantization module 1350 is configured to perform a quantization operation on the sparse inference model to obtain a quantized inference model. It should be understood that the inference model compression apparatus 1300 of the embodiments of the present application can be implemented by a GPU, an NPU, an ASIC, or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), an FPGA, a generic array logic (GAL), or any combination thereof. When the methods shown in FIGS. 2-11 are implemented by software, the inference model compression apparatus 1300 and each module thereof can also be a software module. The inference model compression apparatus 1300 according to the embodiments of the present application can correspond to performing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each unit in the inference model compression apparatus 1300 are respectively for implementing the corresponding flow of the method shown in any one of FIGS. 2-10, and for brevity, will not be repeated here. In addition, the inference model compression apparatus shown in FIGS. 13 and 14 can also be implemented by a communication device. The communication device can be the computer system in the foregoing embodiments, or when the communication device is a chip or a chip system applied to a computing device, the data pre-fetching apparatus can also be implemented by the chip or the chip system. The embodiments of the present application also provide a chip including a processor and a power supply circuit configured to supply power to the processor, and the processor is configured to implement the functions of the computer device in the method shown in any one of FIGS. 3-10. In a possible design, the chip further includes a memory configured to store program instructions and / or data. The chip can further include other discrete devices. The application further provides a computing device. As shown in FIG. 15, FIG. 15 is a structural schematic diagram of a computing device provided by the application. The computing device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate through the bus 1502. The computing device 1500 can be a server, a terminal device, and can be the inference device described above. It is worth noting that the number of processors and memories in the computing device 1500 is not limited by the application. The bus 1502 can be, but is not limited to, a PCIe bus, a universal serial bus (USB), or an inter-integrated circuit (I2C) bus, an EISA bus, a UB, a CXL, a CCIX, etc. The bus 1502 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 15, but it does not mean that there is only one bus or only one type of bus. The bus 1502 can include a path for transmitting information between various components of the computing device 1500 (for example, the memory 1506, the processor 1504, the communication interface 1508). The processor 1504 can include any one or more of a CPU, a GPU, a microprocessor (MP), or a DSP, etc. The memory 1506 can include a volatile memory, such as a random access memory (RAM). The memory 1506 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard disk drive (HDD), or a solid state drive (SSD). The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to respectively implement the functions of the input module 1310, the acquisition module 1320, the calculation module 1330, and the sparsification module 1340 described above, so as to implement the inference model compression method described above. That is, the memory 1506 has instructions for executing the inference model compression method. The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card and a transceiver, to realize the communication between the computing device 1500 and other devices or communication networks. The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device 1500. The same instructions for performing the inference model compression method can be stored in the memory 1506 of one or more computing devices 1500 in the computing device cluster. The computing device 1500 can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 1500 can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone. In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the inference model compression method. The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the inference model compression method. In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; or an optical medium, for example, a digital video disc (digital video disc, DVD); or a semiconductor medium, for example, a solid state disk (solid state drive, SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for compressing inference models, characterized in that, The inference model comprises a plurality of network layers, and the method comprises: inputting inference data into the inference model and executing the inference model; for a first network layer in the inference model, obtaining a weight matrix of the first network layer and input data of the first network layer, wherein the first network layer is any one of the plurality of network layers; calculating the product of the weight matrix of the first network layer in the inference model and the input data of the first network layer to obtain a result matrix; the result matrix and the weight matrix are of the same size, and the size of the numerical value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix; performing a sparse operation on the weight values ranked after a% in the weight matrix of the first network layer according to the order of importance from large to small, wherein the a% is a sparsity of the inference model; after performing the sparse operation on the plurality of network layers, a sparse inference model is obtained.

2. The method of claim 1, wherein, The sparsity is set by a user.

3. The method of claim 1, wherein, The sparsity is selected from a plurality of set sparsities in a multi-objective optimization manner, and optimization objectives of the multi-objective optimization are that the sparsity of the entire inference model is greater than or equal to a target sparsity, and the sum of sparsity errors of all network layers in the inference model is less than or equal to a first threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The weight matrix of the first network layer in the inference model and the weight matrix of the sparse inference model have the same weight value, and the same weight value includes a weight value greater than or equal to a second threshold in the weight matrix of the first network layer of the inference model, the second threshold is any one of the last M values in the order of absolute values of weight values from large to small in the weight matrix of the first network layer of the inference model, and M is an integer greater than or equal to 1.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: performing a quantization operation on the sparse inference model to obtain a quantized inference model.

6. An inference model compression apparatus, comprising: The inference model comprises a plurality of network layers, and the device comprises: an input module configured to input inference data into the inference model and execute the inference model; an obtaining module configured to, for a first network layer in the inference model, obtain a weight matrix of the first network layer and input data of the first network layer, wherein the first network layer is any one of the plurality of network layers; a calculation module configured to calculate the product of the weight matrix of the first network layer in the inference model and the input data of the first network layer to obtain a result matrix; the result matrix and the weight matrix are of the same size, and the size of the numerical value at the same position in the result matrix indicates the importance of the weight value at the same position in the weight matrix; a sparse module configured to perform a sparse operation on the weight values ranked after a% in the weight matrix of the first network layer according to the order of importance from large to small, wherein the a% is a sparsity of the inference model; after performing the sparse operation on the plurality of network layers, a sparse inference model is obtained.

7. The apparatus of claim 6, wherein, The sparsity is set by a user.

8. The apparatus of claim 6, wherein, The sparsity is obtained by selecting from a plurality of sparsities set in a multi-objective optimization manner, and an optimization objective of the multi-objective optimization is that a sparsity of the entire inference model is greater than or equal to a target sparsity, and a sum of sparsity errors of all network layers in the inference model is less than or equal to a first threshold.

9. The apparatus of any one of claims 6-8, wherein, The weight matrix of the first network layer in the inference model and the weight matrix of the sparse inference model have the same weight value, and the same weight value includes a weight value greater than or equal to a second threshold in the weight matrix of the first network layer of the inference model, the second threshold is any one of M values arranged at the end in a sequence from large to small in an absolute value of the weight value in the weight matrix of the first network layer of the inference model, and M is an integer greater than or equal to 1.

10. The apparatus of any one of claims 6 to 9, wherein, The apparatus further includes: The quantization module is configured to perform a quantization operation on the sparse inference model to obtain a quantized inference model.

11. A cluster of computing devices, characterized in that, Each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a processing device, the method of any one of claims 1 to 5 is implemented.

13. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or instructions are executed by a processing device, the method of any one of claims 1 to 5 is implemented.

Images