Model compression method, device, electronic device and storage medium

Abstract

The present disclosure provides a model compression method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology, and in particular to the field of large language models. The specific implementation scheme is: in response to receiving the initial model parameters of the model to be compressed and the target compression accuracy set for the initial model parameters, the model weight matrix of the model to be compressed is read from the initial model parameters; the model weight matrix is ​​sparsely processed to obtain a sparse weight matrix; the target weights in the sparse weight matrix are fused to obtain fused weights, and the target weights represent the model weights of the sub-networks with coupling relationships in the model to be compressed; according to the target compression accuracy, the target model parameters including the fused weights are quantized to obtain quantized parameters, and the target model parameters include the model weights represented by the sparse weight matrix and the activation values ​​determined based on the sparse weight matrix; and according to the quantized parameters, the compressed model corresponding to the model to be compressed is determined.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of large language models, and specifically to a model compression method, device, electronic device and storage medium. Background Art

[0002] As an important technology in the field of natural language processing, large language models are characterized by huge model size and wide versatility, and perform well in general-purpose tasks. They have broad application scenarios in many fields such as machine translation, text generation, and question-answering systems. Summary of the invention

[0003] The present disclosure provides a model compression method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, a model compression method is provided, comprising: in response to receiving initial model parameters of a model to be compressed and a target compression accuracy set for the initial model parameters, reading a model weight matrix of the model to be compressed from the initial model parameters; performing sparse processing on the model weight matrix to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold; fusing the target weights in the sparse weight matrix to obtain fused weights, the target weights representing the model weights of sub-networks with a coupling relationship in the model to be compressed, and the model to be compressed includes at least one sub-network; according to the target compression accuracy, quantizing the target model parameters including the fused weights to obtain quantization parameters, the target model parameters including the model weights represented by the sparse weight matrix and activation values ​​determined based on the sparse weight matrix; and determining a compressed model corresponding to the model to be compressed according to the quantization parameters.

[0005] According to another aspect of the present disclosure, a model compression device is provided, including: a reading module, configured to read a model weight matrix of a model to be compressed from the initial model parameters in response to receiving the initial model parameters of the model to be compressed and the target compression accuracy set for the initial model parameters; a sparse processing module, configured to perform sparse processing on the model weight matrix to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold; a fusion module, configured to fuse the target weights in the sparse weight matrix to obtain fused weights, the target weights representing the model weights of sub-networks having a coupling relationship in the model to be compressed, and the model to be compressed includes at least one sub-network; a quantization module, configured to quantize the target model parameters including the fused weights according to the target compression accuracy to obtain quantization parameters, the target model parameters including the model weights represented by the sparse weight matrix and the activation values ​​determined based on the sparse weight matrix; and a compressed model determination module, configured to determine a compressed model corresponding to the model to be compressed according to the quantization parameters.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model compression method of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the model compression method of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the model compression method of the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 An exemplary system architecture to which the model compression method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 2 A flowchart schematically shows a content processing method according to an embodiment of the present disclosure;

[0013] Figure 3 A schematic diagram schematically shows a sparse processing of a model weight matrix according to an embodiment of the present disclosure;

[0014] Figure 4 The structure diagram of the attention head according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 5 A schematic diagram of model compression in a system based on a compression acceleration mechanism according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 6 A block diagram schematically shows a model compression device according to an embodiment of the present disclosure; and

[0017] Figure 7 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0020] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0021] Due to the massive data processing scale and the large number of parameters of the large language model itself, it requires high computing resources to support model training and reasoning. For ordinary users, the deployment cost of large language models is unaffordable, which limits the development and promotion of this technology. Therefore, compressing and accelerating large language models to lower the threshold for model deployment has broad application scenarios and significant product optimization effects.

[0022] Model quantization and pruning are the two mainstream compression methods in the field of model compression.

[0023] Model quantization can be roughly divided into two categories. One is called quantization-aware training (QAT) in the industry. This type of quantization algorithm is performed in the forward propagation of the model training process, and the weights and activation values ​​are quantized, but the back propagation and loss calculation are still completed in the form of floating-point numbers. The advantage of this type of method is that the model can adapt to the loss caused by quantization during the training process, and adjust the parameters to achieve the effect of lossless model accuracy after quantization. The other type is static quantization (Post-training quantization, PTQ) that does not require training. This type of quantization method calculates the data distribution of model weights and activation values ​​through the statistical reasoning process of a certain amount of data, and performs numerical mapping from high-precision to low-precision data to complete quantization. The advantage of this type of method is that it is fast and does not require training.

[0024] However, these existing quantization compression acceleration tools have obvious shortcomings in the application scenarios of large language models: QAT requires the participation of the training process. Due to the huge number of parameters and the huge amount of training data, the training process of large language models often requires a lot of machine time on expensive machines, and the quantization cost is too high. It is not widely available. PTQ is also subject to the huge number of parameters of large language models. The distribution range of its parameter data will be extremely wide, and many outliers and abnormal values ​​will inevitably appear. These factors will greatly affect the quantization results, resulting in a cliff-like drop in the accuracy of the model after quantization. Although model compression inference acceleration has been achieved, the model performance has been severely damaged.

[0025] Model pruning technology solutions can be roughly divided into two categories. One is structured pruning. This method reduces model parameters by adjusting the structure, reduces the model size in terms of the number of parameters, and intuitively achieves model compression. The other is unstructured pruning. This method gathers the model feature expression parameters on a small number of key parameters through sparse means, and achieves model reasoning acceleration with a specific acceleration library.

[0026] Since structured pruning changes the model structure, it often requires retraining or fine-tuning the model to restore model accuracy. Unstructured pruning cannot effectively reduce the model size, and requires additional acceleration libraries to adapt to hardware to achieve model acceleration.

[0027] In the process of realizing the concept of the present disclosure, the inventors discovered that the model compression process requires complex numerical calculations and huge data distribution statistics, involving multiple subtasks such as model structure analysis, model parameter distribution statistics, model structure adjustment, and compression accuracy loss repair. The accuracy loss caused by the compression task also gradually increases with the complexity of the model structure, the degree of coupling between model parameters, and the scale of the model parameters. The more complex and large the model, the more difficult it is to compress it, and the more difficult it is to maintain the performance of the model after compression. When dealing with emerging models such as large language models with huge parameters and extremely complex and diverse structures, ordinary developers will face difficult compression algorithm adaptation problems and unpredictable model effect losses after compression.

[0028] Figure 1 An exemplary system architecture to which the model compression method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0029] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the model compression method and apparatus can be applied may include a terminal device, but the terminal device may implement the model compression method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0034] It should be noted that the model compression method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the model compression device provided in the embodiment of the present disclosure can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0035] Alternatively, the model compression method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the model compression device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The model compression method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the model compression device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0036] For example, when model compression is required, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can obtain the target content in the electronic book pointed by the user's sight, and then send the obtained target content to the server 105, and the server 105 analyzes the target content to determine the characteristic information of the target content; predicts the content that the user is interested in based on the characteristic information of the target content; and extracts the content that the user is interested in. Alternatively, a server or server cluster that can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105 analyzes the target content and finally extracts the content that the user is interested in.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Figure 2 The flowchart of the content processing method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method includes operations S210 to S250.

[0040] In operation S210, in response to receiving initial model parameters of a model to be compressed and a target compression accuracy set for the initial model parameters, a model weight matrix of the model to be compressed is read from the initial model parameters.

[0041] According to an embodiment of the present disclosure, the model to be compressed may include a large language model, for example, it may include any at least one series of models such as LLaMa (an open source large model), LLaMa2 (an open source large model), Bloom (a new large language model), OPT (optimum performance training, optimal performance training model), etc., and may not be limited thereto. The initial model parameters may include a model weight matrix of the model to be compressed, and may also include an activation value determined based on the model to be compressed, input data and output data of each sub-network of the model to be compressed, etc., and may not be limited thereto.

[0042] It should be noted that the model to be compressed is not limited to the above, for example, it can also include various deep learning models, neural network models, etc.

[0043] According to an embodiment of the present disclosure, the target model accuracy can be used to define the accuracy of the model parameters of the compression model. For example, the target compression accuracy can include any one of the following: int8, int4, etc., and may not be limited thereto. int8 can be used to control the initial model parameters of the model to be compressed to be compressed into 8-bit integer parameters. Int4 can be used to control the initial model parameters of the model to be compressed to be compressed into 4-bit integer parameters.

[0044] In operation S220, the model weight matrix is ​​sparsely processed to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold.

[0045] According to an embodiment of the present disclosure, the process of sparse processing of the model weight matrix can be implemented in an unstructured pruning manner, and the process of unstructured pruning can be implemented based on a model sparse module. This method does not require changing the model structure, and the sparse process will not affect the accuracy and performance of the model. As a pre-module for model quantization, the model sparse module is mainly responsible for parameter aggregation. By adjusting the model weight matrix to a large number of zeros and dense elements represented by non-zero elements, the model parameters are gathered on a small number of key parameters by sparse means. In a large language model with a huge number of parameters, the difficulty of quantization can be reduced for subsequent quantization operations and the parameter distribution can be simplified. Another is to enable the model to adapt to a specific hardware acceleration library. The hardware acceleration library can extract the non-zero elements in the sparse matrix and combine them into a new matrix. The sparse module can optimize the computational efficiency of the GEMM (GEneral Matrix to Matrix Multiplication) operator with the help of the CUTLASS (a high-performance computing library) acceleration library, which can improve the inference speed of the model on the GPU (Graphics Processing Unit). For example, if the sparseness ratio of each column of a sparse matrix is ​​set to 50%, the column vector of the new matrix will be only half of the original matrix, and the overall matrix parameter volume will be reduced by half. Using the Sparse Tensor Cores of GPU hardware for accelerated calculations can increase the calculation speed of matrix multiplication and realize the function of inference acceleration.

[0046] According to an embodiment of the present disclosure, the preset threshold can be customized according to actual business needs. In the process of performing sparsification processing on the model weight matrix based on the model sparsification module to obtain a sparse weight matrix, the sparsification processing can be implemented with the condition that the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than the preset threshold as a constraint. For example, after each sparsification processing of the model weight matrix, the target accuracy of the model determined by the model weight matrix obtained by the sparsification processing can be determined, and the target accuracy is compared with the initial accuracy of the model to be compressed. When it is determined that the difference between the target accuracy and the initial accuracy is less than the preset threshold, the model weight matrix obtained by the sparsification processing that can achieve the target accuracy is determined as the sparse parameter matrix.

[0047] It should be noted that the above-mentioned sparse processing process is not limited to random zeroing, random adjustment and other processing processes, as long as the sparse processing can be achieved and the difference between the target accuracy of the model determined by the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold.

[0048] In operation S230, target weights in the sparse weight matrix are fused to obtain fused weights, where the target weights represent model weights of sub-networks having a coupling relationship in the model to be compressed, and the model to be compressed includes at least one sub-network.

[0049] According to an embodiment of the present disclosure, a sub-network with a coupling relationship can, for example, characterize a layer structure network in a model to be compressed. The layer structure network can be specifically characterized as a linear layer structure network. The fusion method may include at least one of the following: layer fusion, splicing fusion, etc., and may not be limited thereto. For example, for target weights that have a top-down relationship in matrix calculation and can perform serial calculations, layer fusion can be implemented. For target weights that have a parallel relationship, splicing fusion can be performed on the target weight or the output features determined based on the target weight.

[0050] In operation S240, target model parameters including fusion weights are quantized according to target compression accuracy to obtain quantized parameters, where the target model parameters include model weights represented by a sparse weight matrix and activation values ​​determined based on the sparse weight matrix.

[0051] According to an embodiment of the present disclosure, the target model parameters may have the same or similar technical features as the initial model parameters, which are not limited here. The quantization process can be implemented based on the model quantization module. The model quantization module can take the output of the model sparsification module as input and then perform a quantization operation. The activation value can represent the vector features obtained after various types of information such as text, images, audio signals, etc. are processed by the sparse weight matrix of the model to be compressed. The vector features can be represented by floating-point values. Model quantization can convert high-precision data types into low-precision data types by processing weights and activation values ​​stored in floating-point values ​​in deep neural networks, thereby reducing the space occupied by data storage.

[0052] For example, in the process of performing a quantization operation, the floating-point model parameters can be mapped to the corresponding integer data precision according to the target compression precision using the PTQ principle.

[0053] In operation S250, a compressed model corresponding to the to-be-compressed model is determined according to the quantization parameter.

[0054] According to an embodiment of the present disclosure, the model weights of each sub-network of the model to be compressed can be updated according to the model weights represented by the quantization parameters to obtain the compressed model.

[0055] Taking a large language model with hundreds of billions of parameters as an example, for example, when it performs reasoning and operation for text information, it requires at least 5 NVIDIA A100-80GB GPUs to support it. Based on the above method, the initial large language model parameters of the large language model can be first screened from the large language model. The initial large language model parameters may include the large language model weight matrix of the large language model, and the text vector features output by the large language model based on the large language model weight matrix. By first sparsely processing the large language model weight matrix, a large language model sparse weight matrix is ​​obtained. Then, the large language model target weights in the large language model sparse weight matrix are coupled to obtain the large language model fusion weight. And according to the target compression accuracy, the large language model sparse weight matrix including the large language model fusion weight and the text vector features are processed by converting floating point values ​​to integer values, and the large language model quantization parameters can be obtained. According to the large language model quantization parameters, the compressed large language model can be determined. In this embodiment, when the compressed large language model is used to reason and run text information, for example, only one GPU is needed for reasoning, which can effectively optimize the computing resource occupancy.

[0056] Taking the LLaMa2-70b model as an example, for example, using the original model for image reasoning requires 132GB of video memory, and it is expected to use 4 A100-40GB graphics card resources. Based on a compression method similar to the aforementioned large language model, for example, the initial LLaMa2-70b model parameters of the LLaMa2-70b model can be first screened from the LLaMa2-70b model. The initial LLaMa2-70b model parameters may include the LLaMa2-70b model weight matrix of the LLaMa2-70b model, and the image vector features output by the LLaMa2-70b model based on the LLaMa2-70b model weight matrix. By first sparsely processing the LLaMa2-70b model weight matrix, the LLaMa2-70b model sparse weight matrix is ​​obtained. Then, the LLaMa2-70b model target weights in the LLaMa2-70b model sparse weight matrix are coupled to obtain the LLaMa2-70b model fusion weights. And according to the target compression accuracy, the LLaMa2-70b model sparse weight matrix including the LLaMa2-70b model fusion weights and the image vector features are processed from floating point values ​​to integer values, and the LLaMa2-70b model quantization parameters can be obtained. The lightweight LLaMa2-70b model can be obtained according to the LLaMa2-70b model quantization parameters. In this embodiment, when image information is inferred and run based on the lightweight LLaMa2-70b model, only 35GB of video memory will be required. It is expected that one graphics card of the same type can be used to complete the inference and achieve inference acceleration.

[0057] It should be noted that the above-mentioned use of the large language model and the LLaMa2-70b model as the models to be compressed and processing text and image data is only an exemplary embodiment and is not limited to this. Corresponding to actual business needs, the above-mentioned model compression operation can be performed on various types of models to be compressed that process various types of data, which is not limited here.

[0058] Through the above-mentioned embodiments of the present disclosure, since the computing acceleration library will extract the non-zero elements in the sparse matrix and combine them into a new matrix, in the process of using GPU hardware to process the sparse weight matrix, the amount of parameters to be processed can be reduced, accelerated calculation can be achieved, and the calculation speed of matrix multiplication can be effectively improved. Combined with the layer fusion method, the amount of calculation during quantization can be further reduced, and models of various structures can be adapted. The compressed model obtained can effectively reduce the occupation of computer resources. In addition, based on this method, the compression acceleration processing of various large language models can be completed with one click. The user only needs to upload or select the original model to be compressed and specify the target compression accuracy they want, and the structural analysis and quantization compression of the target model can be automatically completed, and finally a compressed model with a smaller model size and faster inference speed is provided.

[0059] In conjunction with specific embodiments, Figure 2 The method shown is further explained.

[0060] According to an embodiment of the present disclosure, the model to be compressed includes multiple channels. The above operation S220 may include: taking the model weights belonging to the same channel in the model weight matrix as a unit, and performing sparse processing on the model weight matrix.

[0061] According to an embodiment of the present disclosure, the model weights of the same channel may be recorded in the model weight matrix in the form of columns, rows, or other random records.

[0062] According to an embodiment of the present disclosure, when the model weights of the same channel are randomly recorded, or when the model weight matrix is ​​not obtained, when it is necessary to sparse the model weight matrix information, the model weights belonging to the same channel can be sparsely processed on a channel-by-channel basis. After the model weights of some or all channels are sparsely processed, a sparse weight matrix is ​​obtained.

[0063] According to one embodiment of the present disclosure, each column of the model weight matrix can represent the model weight corresponding to the same input channel, and each row can represent the model weight corresponding to the same input token.

[0064] According to an embodiment of the present disclosure, when the model weights corresponding to the same input channel are recorded in the form of columns, the model weights belonging to the same channel in the model weight matrix are taken as a unit, and sparse processing of the model weight matrix may include: taking the columns in the model weight matrix as units, and sparse processing of the column model weights in the columns.

[0065] According to an embodiment of the present disclosure, performing sparse processing on the column model weights in the column may include: resetting the first part of the weights in the column model weights to zero, and adjusting the first target weights in the same row as the first part of the weights.

[0066] According to an embodiment of the present disclosure, the first part of weights randomly determined in the column model weights can be reset to zero, or the number of first part of weights in the column model weights that need to be reset to zero can be determined based on a predefined sparsity, and the corresponding number of first part of weights randomly determined in the column model weights can be reset to zero.

[0067] According to an embodiment of the present disclosure, after resetting the first part of the weights in the column model weights to zero, all first target weights in the same row as the first part of the weights in the model weight matrix can be adjusted. The non-zero first target weights in the same row as the first part of the weights in the model weight matrix can also be adjusted to ensure the sparsity of the model weight matrix.

[0068] Through the above-mentioned embodiments of the present disclosure, the accuracy of the model determined according to the sparse weight matrix can be effectively guaranteed by combining the two operations of zeroing and adjusting.

[0069] According to an embodiment of the present disclosure, the above-mentioned resetting the first part of the weights in the column model weights may include: resetting the first part of the weights in the mth column in the model weight matrix, where m is an integer from 1 to M in sequence, and M represents the number of columns in the model weight matrix. On this basis, the above-mentioned adjustment of the first target weights in the same row as the first part of the weights may include: adjusting the first target weights in the m+1th column to the Mth column in the model weight matrix.

[0070] According to an embodiment of the present disclosure, when a certain column of the model weight matrix is ​​subjected to sparse processing by taking the column of the model weight matrix as the operation unit, the performance loss caused by the sparse processing can be offset by adjusting the other weights on the right side of the column, that is, the first target weight in the same row as the first part of the weight. Similarly, by operating the model parameters column by column, the sparse processing of the model weight matrix can be achieved, and a sparse weight matrix can be obtained.

[0071] Figure 3 The figure schematically shows a schematic diagram of performing sparse processing on a model weight matrix according to an embodiment of the present disclosure.

[0072] like Figure 3 As shown, the model weight matrix 300 is, for example, a matrix of 6 rows × 3 columns, including a first column of model weights 301, a second column of model weights 302, a third column of model weights 303, and a fourth column of model weights 304. Each column in the matrix represents the model weights corresponding to the same input channel, and each row represents the model weights corresponding to the same input token.

[0073] For example, the unit is a column, and the predefined sparsity is set to 50%.

[0074] First, for the model weight matrix 300, the three random first partial weights in the first column of model weights 301 including six weight elements can be reset to zero, and the first target weights in the second column of model weights 302, the third column of model weights 303, and the fourth column of model weights 304, which are in the same row as the first partial weights in the first column of model weights 301, can be adjusted to obtain a first weight matrix 310. For example, see Figure 3The first weight matrix 310 in may include a first column of model weights 311, a second column of model weights 312, a third column of model weights 313, and a fourth column of model weights 314. The black area in the first column of model weights 311 may represent weights that have been set to zero, and the gray area in each of the second column of model weights 312, the third column of model weights 313, and the fourth column of model weights 314 may represent weights to be adjusted.

[0075] Then, for the first weight matrix 310, the three random first partial weights in the second column model weight 312 including six weight elements are reset to zero, and the first target weights in the third column model weight 313 and the fourth column model weight 314 in the same row as the first partial weights in the second column model weight 312 are adjusted to obtain the second weight matrix 320. For example, see Figure 3 The second weight matrix 320 in may include a first column model weight 321, a second column model weight 322, a third column model weight 323, and a fourth column model weight 324. The first column model weight 321 is determined according to the first column model weight 311, the black area in the second column model weight 322 may represent the weights that have been set to zero, and the gray area in each of the third column model weight 323 and the fourth column model weight 324 may represent the weights to be adjusted.

[0076] Afterwards, the aforementioned random zeroing and peer adjustment operations can be performed on the third column model weight 323 and the fourth column model weight 324 of the second weight matrix 320, respectively, to obtain a sparse weight matrix 330. The proportion of 0 values ​​in the sparse weight matrix 330 is 50%.

[0077] It should be noted that after the first part of the weights in the Mth column in the model weight matrix is ​​reset to zero, the same-row adjustment operation may no longer be performed. Corresponding to the above embodiment, after the aforementioned random zeroing operation is performed on the fourth column model weight 334, the same-row adjustment operation may no longer be performed.

[0078] Through the above-mentioned embodiments of the present disclosure, by automatically adjusting the remaining weights after weight sparsification, retraining after sparsification can be reduced, and the impact of sparsification on model performance can be reduced. When this method is applied to a large language model, the training cost can be effectively reduced.

[0079] According to an embodiment of the present disclosure, the above-mentioned adjustment of the first target weight in the same row as the first part of the weight may include: inputting the first input data into the model to be compressed to obtain a first initial output result. Inputting the first input data into the first adjusted model to obtain a first target output result, and the model weight matrix of the first adjusted model is characterized as the model weight matrix after the first part of the parameters are set to zero. Determine the first sensitivity loss based on the first initial output result and the first target output result. Adjust the first target weight based on the first sensitivity loss.

[0080] According to an embodiment of the present disclosure, the first input data is used as the input data of the model to be compressed and the first adjusted model. It can be determined based on the model input data of the model to be compressed, or it can be determined by randomly selecting relevant data, which is not limited here. The model input data may include at least one of the following: text, image, audio signal, etc., and may not be limited thereto. For example, the model input data is an audio signal, the first initial output result can represent the initial audio vector features output by the model to be compressed based on the initial model weight matrix, and the first target output result can represent the target audio vector features output by the first adjusted model based on the model weight matrix after setting the first part of the parameters to zero. A first sensitivity loss can be determined based on the initial audio vector features and the target audio vector features.

[0081] According to an embodiment of the present disclosure, when a certain column of the model weight matrix is ​​sparsely processed with the column of the model weight matrix as the operation unit, the influence of resetting the weight to zero on the model inference result can be quantified and statistically analyzed by layer-wise sensitivity loss. Figure 3 In the embodiment shown, the first input data can be input into the model to be compressed determined based on the model weight matrix 300 to obtain a first initial output result. The first input data is input into the model determined based on the first weight matrix 310 to obtain a first target output result. Then, the first sensitivity loss is determined based on the difference between the first initial output result and the first target output result. Thereafter, the weights represented by the gray areas in the same row as the black areas in the first column of model weights 311 in the second column of model weights 312, the third column of model weights 313, and the fourth column of model weights 314 can be adjusted based on the first sensitivity loss.

[0082] It should be noted that in the process of performing peer adjustments on the third column model weights 323 and the fourth column model weights 324 in the second weight matrix 320, as well as resetting each subsequent column model weight and performing peer adjustments on the relevant parameters on the right side of the column, adjustments can be implemented in combination with the aforementioned first sensitivity loss, which will not be repeated here.

[0083] Through the above-mentioned embodiments of the present disclosure, by using sensitivity loss, the feature differences of the layer outputs before and after sparseness are statistically analyzed to measure the performance impact before and after the model parameter changes, and to guide the subsequent weight adjustment strategy, the process of retraining after sparsification can be reduced, the impact of sparsification on model performance can be reduced, and the training cost can be reduced.

[0084] According to another embodiment of the present disclosure, each column of the model weight matrix can represent the model weight corresponding to the same input token, and each row can represent the model weight corresponding to the same input channel. For example, after transposing the input matrix of the model weight, the model weight matrix represented by this case can be obtained.

[0085] According to an embodiment of the present disclosure, when the model weights corresponding to the same input channel are recorded in the form of rows, the model weights belonging to the same channel in the model weight matrix are taken as a unit, and sparse processing of the model weight matrix may include: taking the rows in the model weight matrix as units, and sparse processing of the row model weights in the rows.

[0086] According to an embodiment of the present disclosure, the above-mentioned sparse processing of the row model weights in the row may include: resetting the second part of the weights in the row model weights to zero, and adjusting the second target weights in the same column as the second part of the weights.

[0087] According to an embodiment of the present disclosure, the second part of the weights may have the same or similar technical features as the aforementioned first part of the weights. The second target weights may have the same or similar technical features as the aforementioned first target weights. The process of sparse processing of the row model weights has the same or similar implementation method as the aforementioned process of sparse processing of the column model weights. The process of sparse processing of the row model weights can refer to the aforementioned process of sparse processing of the column model weights. It is only necessary to transform the columns in the aforementioned implementation process into rows, which will not be repeated here.

[0088] Through the above-mentioned embodiments of the present disclosure, the accuracy of the model determined according to the sparse weight matrix can be effectively guaranteed by combining the two operations of zeroing and adjusting.

[0089] According to an embodiment of the present disclosure, the above-mentioned resetting the second part of the weights in the row model weights may include: resetting the second part of the weights in the nth row in the model weight matrix, where n is an integer from 1 to N in sequence, and N represents the number of rows in the model weight matrix. On this basis, the above-mentioned adjustment of the second target weights in the same column as the second part of the weights may include: adjusting the second target weights in the n+1th to Nth rows in the model weight matrix.

[0090] According to an embodiment of the present disclosure, when a row of the model weight matrix is ​​subjected to sparse processing using the behavior operation unit of the model weight matrix, the performance loss caused by the sparse processing can be offset by adjusting the other weights on the lower side of the row, that is, the second target weights in the same row as the second part of the weights. Similarly, by operating the model weights row by row, the sparse processing of the model weight matrix can be achieved, and a sparse weight matrix can be obtained.

[0091] According to an embodiment of the present disclosure, the process of operating the model weights row by row is similar to Figure 3 The process of operating on the model weights column by column is the same as shown above, just replace Figure 3 The columns in are transformed into rows, which will not be repeated here.

[0092] Through the above-mentioned embodiments of the present disclosure, by automatically adjusting the remaining weights after weight sparsification, retraining after sparsification can be reduced, and the impact of sparsification on model performance can be reduced. When this method is applied to a large language model, the training cost can be effectively reduced.

[0093] According to an embodiment of the present disclosure, the above-mentioned adjustment of the second target weight in the same column as the second part weight may include: inputting the second input data into the model to be compressed to obtain a second initial output result. Inputting the second input data into the second adjusted model to obtain a second target output result, and the model weight matrix of the second adjusted model is characterized as the model weight matrix after the second part parameters are set to zero. Determine the second sensitivity loss based on the second initial output result and the second target output result. Adjust the second target weight based on the second sensitivity loss.

[0094] According to an embodiment of the present disclosure, the second input data may have the same or similar technical features as the first input data. The second sensitivity loss may have the same or similar technical features as the aforementioned first sensitivity loss. The process of obtaining the second initial output result and the second target output result may refer to the aforementioned process of obtaining the first initial output result and the first target output result. The process of determining the second sensitivity loss may refer to the aforementioned process of determining the first sensitivity loss. The process of adjusting the second target weight according to the second sensitivity loss may refer to the aforementioned process of adjusting the first target weight according to the first sensitivity loss, which will not be repeated here.

[0095] Through the above-mentioned embodiments of the present disclosure, by using sensitivity loss, the feature differences of the layer outputs before and after sparseness are statistically analyzed to measure the performance impact before and after the model parameter changes, and to guide the subsequent weight adjustment strategy, the process of retraining after sparsification can be reduced, the impact of sparsification on model performance can be reduced, and the training cost can be reduced.

[0096] According to an embodiment of the present disclosure, sub-networks with a coupling relationship may constitute a layer structure network. The model to be compressed may include at least one layer structure network.

[0097] According to an embodiment of the present disclosure, the above operation S230 may include: obtaining a layer target weight related to the layer structure network from a sparse weight matrix. Performing layer fusion on the layer target weight to obtain a layer fusion weight corresponding to the layer structure network. Determining a fusion weight according to at least one layer fusion weight corresponding to at least one layer structure network.

[0098] According to the embodiments of the present disclosure, the model structure of the model to be compressed can be analyzed first, and the target weights that can be integrated can be determined by screening the linear layer structure. For example, the model structure of the large language model can be analyzed to obtain the model weights of the Attention Head (attention head), MLP (multi-layer perceptron), etc. in the large language model structure, and by analyzing the layer structure network composed of the sub-networks with coupling relationships in the Attention Head and MLP, the model weights in the Attention Head, MLP, etc. are integrated layer by layer network structure.

[0099] It should be noted that the layer fusion process can be implemented according to a predefined layer fusion formula, or it can be implemented by screening the layer target weights and performing simple splicing. There is no limitation here, as long as the layer target weights can be fused.

[0100] According to an embodiment of the present disclosure, the above operation S240 may include: inputting the third input data into the layer structure network to obtain a layer activation value related to the layer structure network, and the layer model weight of the layer structure network is the model weight related to the layer structure network in the sparse weight matrix. According to the weight values ​​of the layer activation value and the layer model weight, the data distribution range is determined. According to the target compression accuracy and the data distribution range, the layer activation value and the layer model weight are quantized to obtain a layer quantization parameter corresponding to the layer structure network. According to at least one layer quantization parameter corresponding to at least one layer structure network, a quantization parameter is determined.

[0101] According to an embodiment of the present disclosure, the third input data can represent the input data of the model, and can be determined based on the model input data of the model to be compressed, or can be determined by randomly selecting relevant data. In the case where the model to be compressed includes multiple layer structure networks, the third input data can also represent the input data of each layer structure network. The input data of each sub-network can be determined based on the model input data of the model to be compressed, or can be determined based on the output characteristics of the layer structure network of the model to be compressed after the model input data is input into the model to be compressed. For example, the model to be compressed includes a first layer structure network and a second layer structure network, the output of the first layer structure network is the input of the second layer structure network, the third input data corresponding to the first layer structure network can be the model input data, and the third input data corresponding to the second layer structure network can be the output characteristics obtained after processing the model input data through the first layer structure network.

[0102] According to an embodiment of the present disclosure, the quantization process may include a layer-by-layer parameter distribution statistics and a layer-by-layer quantization stage. In the layer-by-layer parameter distribution statistics stage, a set of data may be input for each layer structure network in a layer structure network unit, and the layer model weights of the layer structure network and the layer activation values ​​obtained by processing the set of data based on the layer structure network are counted. In the layer-by-layer quantization stage, the layer model parameters of the corresponding layer structure network may be quantized layer by layer according to the data distribution of the layer model weights and the layer activation values, including quantizing the layer model weights and the layer activation values ​​in the corresponding layer structure network. After the layer model weights and layer activation values ​​in all layer structure networks are quantized, the quantization parameters may be determined based on the quantization results.

[0103] According to an embodiment of the present disclosure, when an activation function is included in the model to be compressed, the activation function will hinder the quantization process of the model. In this embodiment, after layer fusion, in order to cooperate with layer fusion, an additional quantization node can be set corresponding to the activation value output by the activation function, and the activation value can be quantized during the quantization process. Through the process of layer-by-layer parameter distribution statistics and layer-by-layer quantization, for activation functions such as GeLU (gaussian error linear units) that are gradually popularized in large language models, their characteristics that cannot be fused with FC layers (Full Connection) can be adapted to alleviate the problem of activation functions hindering model quantization.

[0104] Through the above-mentioned embodiments of the present disclosure, the above-mentioned quantization method can be made more universal, and the functional adaptation surface of the above-mentioned compression method can be effectively expanded, so as to be able to process models including various structures with activation functions.

[0105] According to an embodiment of the present disclosure, the layer model weight may include layer initial weights and layer fusion weights related to the layer structure network. The layer fusion weight may represent the fusion weight obtained by fusing the layer target weights related to the layer structure network in the sparse parameter matrix. The layer initial weight may represent other weights in the layer model weight except the layer target weight. The above-mentioned quantization of the layer activation value and the layer model weight according to the target compression accuracy and the data distribution range to obtain the layer quantization parameters corresponding to the layer structure network may include: determining the scaling factor according to the data distribution range. According to the scaling factor and the target compression accuracy, the layer activation value, the layer initial weight and the layer fusion weight are quantized into integer parameters that match the target compression accuracy. According to the integer parameters, the layer quantization parameters are determined.

[0106] According to an embodiment of the present disclosure, the data distribution range determined according to the layer model weights and layer activation values ​​of each layer structure network can be the same or different. According to the data distribution range corresponding to different layer structure networks, the scaling factors corresponding to different layer structure networks can be the same or different, which is not limited here.

[0107] According to an embodiment of the present disclosure, the integer parameters may include the quantization structure corresponding to the layer model weight and the quantization result corresponding to the layer activation value. After obtaining the integer parameters of the model parameters in each layer structure network, the integer parameters may be determined as quantization parameters.

[0108] For example, the layer structure network is an attention head.

[0109] Figure 4 The structure diagram of the attention head according to an embodiment of the present disclosure is schematically shown.

[0110] like Figure 4 As shown, the attention head 410 has, for example, an embedding layer 411 connected thereto, and may also include a first activation layer 412, a Q module 413, a K module 414, a V module 415, a first KV cache 416, a second KV cache 417, a second activation layer 418, and a fully connected layer 419.

[0111] According to an embodiment of the present disclosure, the Q module 413, the K module 414, the V module 415, the first KV cache 416, the second KV cache 417, and the fully connected layer 419 can each have a corresponding model weight. Corresponding activation values ​​can be obtained based on the first activation layer 412 and the second activation layer 418. In the above quantization process, when determining a layer structure network such as an attention head, the model parameters of each module in the attention head can be quantized to obtain quantized parameters. When applied, the model parameters of each module in the attention head can be replaced with quantized parameters, such as the quantized attention head 420.

[0112] According to an embodiment of the present disclosure, for the fused QKV linear layer structure, for example, Figure 4 When quantizing V in , the scaling factor of the fused output operator in the attention can be calculated independently so that it is not affected by the calculation dimension of the fusion structure.

[0113] Through the above-mentioned embodiments of the present disclosure, respective scaling factors can be calculated in units of layer activation values, layer initial weights or layer fusion weights, and a quantization process can be implemented, thereby reducing the number of calculation steps and effectively improving the quantization efficiency.

[0114] According to an embodiment of the present disclosure, the above-mentioned determination of the layer quantization parameter according to the integer parameter may include: inputting the third input data into the quantized layer structure network corresponding to the layer structure network to obtain the first activation value of the layer, and the model weight of the quantized layer structure network is the model weight represented by the integer parameter. Determine the first quantization loss according to the first activation value of the layer and the layer activation value. Determine the layer quantization parameter according to the integer parameter and the first quantization loss.

[0115] According to an embodiment of the present disclosure, a first quantization loss can be determined by determining the difference between a quantized model and an original model. For example, the first quantization loss can be calculated based on the difference between the output features of the current layer before and after quantization. Specifically, for example, the third input data is image data, the layer activation value can represent the layer image vector features output by the layer structure network based on the model weights related to the layer structure network in the sparse weight matrix, and the first activation value can represent the layer first image feature vector output by the layer structure network after quantization based on the model weights represented by integer parameters. A first quantization loss can be determined based on the layer image vector features and the layer first image feature vector.

[0116] According to an embodiment of the present disclosure, an integer parameter may be determined as a layer quantization parameter. A loss threshold may also be preset, and when it is determined that the first quantization loss is less than or equal to the loss threshold, the integer parameter may be determined as the layer quantization parameter.

[0117] According to an embodiment of the present disclosure, the above-mentioned determination of the layer quantization parameter based on the integer parameter and the first quantization loss may include: adjusting the integer parameter for the i-th time according to the i-th quantization loss to obtain the i-th adjusted integer parameter, where i takes integers between 1 and I in turn, and I is a predefined positive integer. The third input data is input into the i-th adjusted layer structure network corresponding to the layer structure network to obtain the i+1 activation value of the layer, and the model weight of the i-th adjusted layer structure network is the model weight represented by the i-th adjusted integer parameter. Determine the i+1 quantization loss according to the i+1 activation value of the layer and the layer activation value. Determine the layer quantization parameter according to the integer parameter corresponding to the smallest quantization loss from the first quantization loss to the 1+1 quantization loss.

[0118] According to an embodiment of the present disclosure, when it is determined that the first quantization loss is greater than the loss threshold, the quantization parameter adjustment stage can be entered. In the quantization parameter adjustment stage, the integer parameter can be adjusted for the first time according to the first quantization loss to obtain the first adjusted integer parameter, and the first adjustment layer structure network determined based on the first adjusted integer parameter processes the third input data to obtain the second activation value. Then, the second quantization loss is determined according to the second activation value and the layer value. Afterwards, it can be determined whether it is necessary to continue to adjust the integer parameter according to the second quantization loss. For example, when it is determined that the second quantization loss is greater than the aforementioned loss threshold, the aforementioned adjustment process can be continued according to the second quantization loss, and the third quantization loss can be determined. And so on, until the determined quantization loss is less than or equal to the loss threshold, or the number of executions of the adjustment process reaches the preset rounds, the aforementioned adjustment process is terminated.

[0119] According to the embodiments of the present disclosure, the quantization parameter adjustment stage can find the quantization parameter with the smallest change in layer output before and after quantization based on the first quantization loss in the quantization stage, or the i-th quantization loss in the quantization stage and the adjustment stage. For example, the quantization parameter can be determined by the method shown in formula (1):

[0120]

[0121] In formula (1), can represent the quantization loss, W can represent the parameters before quantization, It can represent an integer parameter, and X can represent the third input data.

[0122] It should be noted that the i+1th activation value of the layer may have the same or similar technical features as the first activation value of the layer, and the process of determining the i+1th quantization loss may be the same or similar to the process of determining the first quantization loss in the aforementioned embodiment, which will not be repeated here.

[0123] Through the above-mentioned embodiments of the present disclosure, by using layer-by-layer quantization and layer-by-layer loss calculation, and by adjusting the parameters after quantization, the accuracy of the model before and after quantization can be maintained, the training process of the model after quantization can be reduced, and the training cost can be reduced.

[0124] According to the embodiments of the present disclosure, based on the above-mentioned model compression method, by adopting sparse unstructured pruning technology means, combined with the PTQ-like quantization means after the model accuracy is optimized, while avoiding large-scale model training, the performance accuracy of the model after compression is guaranteed, and an efficient and stable reasoning compression acceleration mechanism for large language models is provided. The system built based on this mechanism can complete the compression acceleration processing of various large language models commonly used in the industry with one click. The user only needs to upload or select the original model to the system and specify the desired target compression accuracy, and the structural analysis and quantization compression of the target model can be automatically completed, and a compressed model with a smaller model size and faster reasoning speed can be provided.

[0125] Figure 5 A schematic diagram of model compression in a system constructed based on a compression acceleration mechanism according to an embodiment of the present disclosure is schematically shown.

[0126] like Figure 5 As shown, the system 500 of the compression acceleration mechanism includes a model sparsification module 510, a model quantization module 520, and an inference evaluation module 530. The user can input a dense large model 501 into the system 500. In some embodiments, the desired target compression accuracy 502 can also be input into the system 500. After the sparsification, layer fusion and quantization processing of the sparsification module 510 and the model quantization module 520 in the system 500, a compressed model 503 is obtained. The inference evaluation module 530 can evaluate the dense large model 501 and the compressed model 503.

[0127] According to the embodiments of the present disclosure, the specific implementations of the sparsification module 510 and the model quantization module 520 have been described in the aforementioned embodiments, and reference may be made to the aforementioned embodiments, and will not be repeated here.

[0128] According to an embodiment of the present disclosure, the reasoning evaluation module 530 can be used to evaluate the model before and after compression, and the evaluation indicators can include the model-generated text PPL (perplexity) and the model reasoning throughput speed. PPL can be used in NLP (Natural Language Processing) as an indicator to measure the quality of a language model. The probability of a sentence appearing can be estimated based on each word, and the sentence length can be used for normalization. The throughput speed can be measured by the number of tokens generated by the model per second.

[0129] Through the above-mentioned embodiments of the present disclosure, users only need to provide the original dense large model to be compressed, and the system can automatically complete the analysis and adaptation of the model through the combined compression acceleration tool processing of model sparsification and model quantization, and quickly generate a lightweight large language model with smaller volume and faster reasoning speed. This allows users to easily obtain the acceleration optimization capabilities for large language models, lowering the threshold for users to use large language models. In addition, the above-mentioned compression acceleration mechanism can control the optimization cost of large language models within the effective profit range, providing users with low-cost large language model optimization acceleration services.

[0130] Based on the combination of the above strategies, model compression can optimize the model in many aspects, such as storage usage, reasoning acceleration, and accuracy recovery. It reduces the threshold for users to use excellent large language models and achieves considerable computing resource savings. It solves the problem of high technical difficulty and high cost of large language model application development.

[0131] Figure 6 A block diagram of a model compression device according to an embodiment of the present disclosure is schematically shown.

[0132] like Figure 6 As shown, the model compression device 600 includes a reading module 610, a sparse processing module 620, a fusion module 630, a quantization module 640 and a compressed model determination module 650.

[0133] The reading module 610 is used to read the model weight matrix of the model to be compressed from the initial model parameters in response to receiving the initial model parameters of the model to be compressed and the target compression accuracy set for the initial model parameters.

[0134] The sparse processing module 620 is used to perform sparse processing on the model weight matrix to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold.

[0135] The fusion module 630 is used to fuse the target weights in the sparse weight matrix to obtain fusion weights, where the target weights represent the model weights of the sub-networks with coupling relationships in the model to be compressed, and the model to be compressed includes at least one sub-network.

[0136] The quantization module 640 is used to quantize the target model parameters including the fusion weights according to the target compression accuracy to obtain quantization parameters, where the target model parameters include the model weights represented by the sparse weight matrix and the activation values ​​determined based on the sparse weight matrix.

[0137] The compressed model determination module 650 is used to determine the compressed model corresponding to the model to be compressed according to the quantization parameter.

[0138] According to an embodiment of the present disclosure, the model to be compressed includes multiple channels. The sparse processing module includes a sparse processing submodule.

[0139] The sparse processing submodule is used to treat the model weights belonging to the same channel in the model weight matrix as a unit and perform sparse processing on the model weight matrix.

[0140] According to an embodiment of the present disclosure, the thinning processing submodule includes a first thinning processing unit or a second thinning processing unit.

[0141] The first sparse processing unit is used to perform sparse processing on the column model weights in the columns based on the columns in the model weight matrix.

[0142] The second sparse processing unit is used to perform sparse processing on the row model weights in the rows based on the rows in the model weight matrix.

[0143] According to an embodiment of the present disclosure, the first thinning processing unit includes a first zeroing subunit and a first adjusting subunit.

[0144] The first zeroing subunit is used to reset the first part of the weights in the column model weights to zero.

[0145] The first adjustment subunit is used to adjust the first target weight in the same row as the first partial weight.

[0146] According to an embodiment of the present disclosure, the first zeroing subunit includes a column-by-column zeroing subunit, which is used to reset the first part of the weights of the mth column in the model weight matrix to zero, where m is an integer from 1 to M, and M represents the number of columns of the model weight matrix. The first adjustment subunit includes a column-by-column adjustment subunit, which is used to adjust the first target weights of the m+1th column to the Mth column in the model weight matrix.

[0147] According to an embodiment of the present disclosure, the first adjustment subunit includes a first initial output result output subunit, a first target output result output subunit, a first sensitivity loss determination subunit and a first target weight adjustment subunit.

[0148] The first initial output result output subunit is used to input the first input data into the model to be compressed to obtain the first initial output result.

[0149] The first target output result output subunit is used to input the first input data into the first adjusted model to obtain the first target output result. The model weight matrix of the first adjusted model is represented by the model weight matrix after the first part of the parameters is set to zero.

[0150] The first sensitivity loss determining subunit is used to determine a first sensitivity loss according to the first initial output result and the first target output result.

[0151] The first target weight adjustment subunit is used to adjust the first target weight according to the first sensitivity loss.

[0152] According to an embodiment of the present disclosure, the second thinning processing unit includes a second zeroing subunit and a second adjusting subunit.

[0153] The second zeroing subunit is used to reset the second part of the weights in the row model weights to zero.

[0154] The second adjustment subunit is used to adjust the second target weight in the same column as the second partial weight.

[0155] According to an embodiment of the present disclosure, the second zeroing subunit includes a row-by-row zeroing subunit, which is used to reset the second partial weights of the nth row in the model weight matrix to zero, where n is an integer from 1 to N, and N represents the number of rows of the model weight matrix. The second adjustment subunit includes a row-by-row adjustment subunit, which is used to adjust the second target weights of the n+1th row to the Nth row in the model weight matrix.

[0156] According to an embodiment of the present disclosure, the second adjustment subunit includes a second initial output result output subunit, a second target output result output subunit, a second sensitivity loss determination subunit and a second target weight adjustment subunit.

[0157] The second initial output result output subunit is used to input the second input data into the model to be compressed to obtain the second initial output result.

[0158] A second target output result output subunit is used to input the second input data into the second adjusted model to obtain the second target output result, wherein the model weight matrix of the second adjusted model is characterized as the model weight matrix after the second part of parameters are set to zero;

[0159] a second sensitivity loss determining subunit, configured to determine a second sensitivity loss according to the second initial output result and the second target output result; and

[0160] The second target weight adjustment subunit is used to adjust the second target weight according to the second sensitivity loss.

[0161] According to an embodiment of the present disclosure, the subnetworks with coupling relationships constitute a layer structure network, and the model to be compressed includes at least one layer structure network. The fusion module includes a layer target weight acquisition submodule, a layer fusion submodule and a fusion weight determination submodule.

[0162] The layer target weight acquisition submodule is used to obtain the layer target weights related to the layer structure network from the sparse weight matrix.

[0163] The layer fusion submodule is used to perform layer fusion on the layer target weights to obtain the layer fusion weights corresponding to the layer structure network.

[0164] The fusion weight determination submodule is used to determine the fusion weight according to at least one layer fusion weight corresponding to at least one layer structure network.

[0165] According to an embodiment of the present disclosure, the sub-networks with coupling relationships constitute a layer structure, and the model to be compressed includes at least one layer structure network. The quantization module includes a layer activation value acquisition sub-module, a data distribution determination sub-module, a quantization sub-module and a quantization parameter determination sub-module.

[0166] The layer activation value obtaining submodule is used to input the third input data into the layer structure network to obtain the layer activation value related to the layer structure network. The layer model weight of the layer structure network is the model weight related to the layer structure network in the sparse weight matrix.

[0167] The data distribution determination submodule is used to determine the data distribution range according to the layer activation value and the weight value of the layer model weight.

[0168] The quantization submodule is used to quantize the layer activation values ​​and layer model weights according to the target compression accuracy and data distribution range, and obtain the layer quantization parameters corresponding to the layer structure network.

[0169] The quantization parameter determination submodule is used to determine the quantization parameter according to at least one layer quantization parameter corresponding to at least one layer structure network.

[0170] According to an embodiment of the present disclosure, the layer model weight includes the layer initial weight and the layer fusion weight related to the layer structure network, the layer fusion weight represents the fusion weight obtained by fusing the layer target weight related to the layer structure network in the sparse parameter matrix, and the layer initial weight represents the other weights in the layer model weight except the layer target weight. The quantization submodule includes a scaling factor determination unit, a quantization unit and a quantization parameter determination unit.

[0171] The scaling factor determination unit is used to determine the scaling factor according to the data distribution range.

[0172] The quantization unit is used to quantize the layer activation value, layer initial weight and layer fusion weight into integer parameters that match the target compression accuracy according to the scaling factor and the target compression accuracy.

[0173] The quantization parameter determination unit is used to determine the layer quantization parameter according to the integer parameter.

[0174] According to an embodiment of the present disclosure, the quantization parameter determination unit includes a layer first activation value obtaining subunit, a first quantization loss determination subunit and a first layer quantization parameter determination subunit.

[0175] The layer first activation value obtaining subunit is used to input the third input data into the quantized layer structure network corresponding to the layer structure network to obtain the layer first activation value, and the model weight of the quantized layer structure network is the model weight represented by the integer parameter.

[0176] The first quantization loss determination subunit is used to determine the first quantization loss according to the layer first activation value and the layer activation value.

[0177] The first layer quantization parameter determination subunit is used to determine the layer quantization parameter according to the integer parameter and the first quantization loss.

[0178] According to an embodiment of the present disclosure, the first layer quantization parameter determination subunit includes an i-th adjustment subunit, an i+1-th activation value acquisition subunit, an i+1-th quantization loss determination subunit and a second layer quantization parameter determination subunit.

[0179] The i-th adjustment subunit is used to perform an i-th adjustment on the integer parameter according to the i-th quantization loss to obtain an i-th adjusted integer parameter, where i is an integer between 1 and I, and I is a predefined positive integer;

[0180] The layer i+1 activation value obtaining subunit is used to input the third input data into the i-th adjustment layer structure network corresponding to the layer structure network to obtain the layer i+1 activation value, and the model weight of the i-th adjustment layer structure network is the model weight represented by the i-th adjustment integer parameter;

[0181] An i+1th quantization loss determining subunit, configured to determine an i+1th quantization loss according to an i+1th activation value of a layer and an activation value of a layer; and

[0182] The second layer quantization parameter determination subunit is used to determine the layer quantization parameter according to the integer parameter corresponding to the smallest quantization loss among the first quantization loss to the (I+1)th quantization loss.

[0183] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0184] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the present disclosure.

[0185] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0186] According to an embodiment of the present disclosure, a computer program product includes a computer program, wherein the computer program is stored in at least one of a readable storage medium and an electronic device, and the computer program implements the method of the present disclosure when executed by a processor.

[0187] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0188] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0189] A number of components in the device 700 are connected to an input / output (I / O) interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0190] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the model compression method. For example, in some embodiments, the model compression method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the model compression method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the model compression method in any other appropriate manner (e.g., by means of firmware).

[0191] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0192] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0193] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0194] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0195] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0196] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0197] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0198] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model compression method, comprising: In response to receiving initial model parameters of a model to be compressed and a target compression accuracy set for the initial model parameters, reading a model weight matrix of the model to be compressed from the initial model parameters; Performing sparse processing on the model weight matrix to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold; Fusing the target weights in the sparse weight matrix to obtain fused weights, wherein the target weights represent model weights of subnetworks having a coupling relationship in the model to be compressed, wherein the model to be compressed includes at least one subnetwork; According to the target compression accuracy, quantizing the target model parameters including the fusion weights to obtain quantized parameters, the target model parameters including the model weights represented by the sparse weight matrix and the activation values ​​determined based on the sparse weight matrix, the model input data of the model to be compressed includes at least one of the following: text, image, audio signal, the activation value is used to represent the vector features obtained after at least one of the text, the image, and the audio signal is processed by the sparse weight matrix; and A compressed model corresponding to the model to be compressed is determined according to the quantization parameter.

2. The method according to claim 1, wherein: The model to be compressed includes a plurality of channels; the sparse processing of the model weight matrix to obtain a sparse weight matrix includes: The model weights belonging to the same channel in the model weight matrix are taken as a unit, and the model weight matrix is ​​sparsely processed.

3. The method according to claim 2, wherein: The method of taking the model weights belonging to the same channel in the model weight matrix as a unit and performing sparse processing on the model weight matrix comprises: Taking the columns in the model weight matrix as units, performing sparse processing on the column model weights in the columns; or Taking the rows in the model weight matrix as units, the row model weights in the rows are sparsely processed.

4. The method according to claim 3, wherein: The sparse processing of the column model weights in the column includes: Resetting the first portion of the column model weights to zero; and The first target weight in the same row as the first partial weight is adjusted.

5. The method according to claim 4, wherein: The resetting the first part of the weights in the column model weights to zero comprises: Reset the first part of the weights of the mth column in the model weight matrix to zero, where m is an integer from 1 to M, and M represents the number of columns of the model weight matrix; The adjusting the first target weight in the same row as the first partial weight includes: The first target weights from the m+1th column to the Mth column in the model weight matrix are adjusted.

6. The method according to claim 4 or 5, wherein: The adjusting the first target weight in the same row as the first partial weight includes: Inputting first input data into the model to be compressed to obtain a first initial output result; Inputting the first input data into a first adjusted model to obtain a first target output result, wherein a model weight matrix of the first adjusted model is characterized as a model weight matrix after setting the first part of parameters to zero; determining a first sensitivity loss according to the first initial output result and the first target output result; and The first target weight is adjusted according to the first sensitivity loss.

7. The method according to claim 3, wherein: The sparse processing of the row model weights in the row includes: Resetting a second portion of the row model weights to zero; and The second target weight in the same column as the second partial weight is adjusted.

8. The method according to claim 7, wherein: The resetting the second part of the row model weights to zero comprises: Resetting the second part of the weights of the nth row in the model weight matrix to zero, where n is an integer from 1 to N, and N represents the number of rows in the model weight matrix; The adjusting the second target weight in the same column as the second partial weight includes: The second target weights from the n+1th row to the Nth row in the model weight matrix are adjusted.

9. The method according to claim 7 or 8, wherein: The adjusting the second target weight in the same column as the second partial weight includes: Inputting the second input data into the model to be compressed to obtain a second initial output result; Inputting the second input data into a second adjusted model to obtain a second target output result, wherein the model weight matrix of the second adjusted model is characterized by setting the second part of parameters to zero; determining a second sensitivity loss according to the second initial output result and the second target output result; and The second target weight is adjusted according to the second sensitivity loss.

10. The method according to any one of claims 1 to 5, 7 and 8, wherein: The sub-networks with coupling relationships constitute a layer structure network, and the model to be compressed includes at least one layer structure network; The fusing the target weights in the sparse weight matrix to obtain the fused weights comprises: Obtaining layer target weights associated with the layer structure network from the sparse weight matrix; Performing layer fusion on the layer target weights to obtain layer fusion weights corresponding to the layer structure network; and The fusion weight is determined according to at least one layer fusion weight corresponding to the at least one layer structure network.

11. The method according to any one of claims 1 to 5, 7 and 8, wherein: The sub-networks with coupling relationships constitute a layer structure network, and the model to be compressed includes at least one layer structure network; According to the target compression accuracy, the target model parameters including the fusion weights are quantized to obtain quantized parameters including: Inputting the third input data into the layer structure network to obtain a layer activation value related to the layer structure network, wherein the layer model weight of the layer structure network is the model weight related to the layer structure network in the sparse weight matrix; Determining a data distribution range according to the layer activation value and a weight value of the layer model weight; quantizing the layer activation value and the layer model weight according to the target compression accuracy and the data distribution range to obtain a layer quantization parameter corresponding to the layer structure network; and The quantization parameter is determined according to at least one layer quantization parameter corresponding to the at least one layer structure network.

12. The method according to claim 11, wherein: The layer model weights include layer initial weights and layer fusion weights related to the layer structure network, the layer fusion weights represent fusion weights obtained by fusing layer target weights related to the layer structure network in the sparse weight matrix, and the layer initial weights represent other weights in the layer model weights except the layer target weights; The step of quantizing the layer activation value and the layer model weight according to the target compression accuracy and the data distribution range to obtain the layer quantization parameter corresponding to the layer structure network includes: Determining a scaling factor according to the data distribution range; According to the scaling factor and the target compression accuracy, quantizing the layer activation value, the layer initial weight, and the layer fusion weight into integer parameters matching the target compression accuracy; and The layer quantization parameter is determined according to the integer parameter.

13. The method according to claim 12, wherein: The determining, according to the integer parameter, the layer quantization parameter comprises: Input the third input data into a quantized layer structure network corresponding to the layer structure network to obtain a first activation value of the layer, wherein the model weight of the quantized layer structure network is the model weight represented by the integer parameter; Determining a first quantization loss according to the layer first activation value and the layer activation value; and The layer quantization parameter is determined according to the integer parameter and the first quantization loss.

14. The method according to claim 13, wherein: The determining, according to the integer parameter and the first quantization loss, the layer quantization parameter comprises: According to the i-th quantization loss, the integer parameter is adjusted for the i-th time to obtain the i-th adjusted integer parameter, where i is an integer between 1 and I, and I is a predefined positive integer; Input the third input data into the i-th adjustment layer structure network corresponding to the layer structure network to obtain the i+1-th activation value of the layer, wherein the model weight of the i-th adjustment layer structure network is the model weight represented by the i-th adjustment integer parameter; Determining an i+1th quantization loss according to the i+1th activation value of the layer and the layer activation value; and The layer quantization parameter is determined according to an integer parameter corresponding to the smallest quantization loss among the first quantization loss to the (I+1)th quantization loss.

15. A model compression device, comprising: A reading module, configured to read a model weight matrix of the model to be compressed from the initial model parameters in response to receiving the initial model parameters of the model to be compressed and the target compression accuracy set for the initial model parameters; A sparse processing module, used for performing sparse processing on the model weight matrix to obtain a sparse weight matrix, wherein the initial model parameters include the model weight matrix, and the difference between the target accuracy of the model determined according to the sparse weight matrix and the initial accuracy of the model to be compressed is less than a preset threshold; A fusion module, used to fuse the target weights in the sparse weight matrix to obtain fusion weights, wherein the target weights represent model weights of sub-networks having a coupling relationship in the model to be compressed, and the model to be compressed includes at least one sub-network; a quantization module, configured to quantize target model parameters including the fusion weights according to the target compression accuracy to obtain quantization parameters, wherein the target model parameters include model weights represented by the sparse weight matrix and activation values ​​determined based on the sparse weight matrix, the model input data of the model to be compressed includes at least one of the following: text, image, and audio signal, and the activation value is used to represent a vector feature obtained after at least one of the text, the image, and the audio signal is processed by the sparse weight matrix; and The compressed model determination module is used to determine the compressed model corresponding to the model to be compressed according to the quantization parameter.

16. The device according to claim 15, wherein: The model to be compressed includes a plurality of channels; The sparse processing module includes: The sparse processing submodule is used to treat the model weights belonging to the same channel in the model weight matrix as a unit and perform sparse processing on the model weight matrix.

17. The device according to claim 16, wherein: The sparse processing submodule includes: A first sparse processing unit is used to perform sparse processing on the column model weights in the column in the model weight matrix, taking the column in the column as a unit; or The second sparse processing unit is used to perform sparse processing on the row model weights in the rows based on the rows in the model weight matrix.

18. The device according to claim 17, wherein: The first sparse processing unit includes: A first zeroing subunit is used to reset a first part of the weights in the column model weights to zero; and The first adjustment subunit is configured to adjust a first target weight in the same row as the first partial weight.

19. The device according to claim 18, wherein: The first zeroing subunit comprises: A column-by-column zeroing subunit is used to reset the first part of the weights of the m-th column in the model weight matrix to zero, where m is an integer from 1 to M, and M represents the number of columns of the model weight matrix; The first adjustment subunit comprises: The column-by-column adjustment subunit is used to adjust the first target weights from the m+1th column to the Mth column in the model weight matrix.

20. The device according to claim 18 or 19, wherein: The first adjustment subunit comprises: A first initial output result output subunit, used for inputting first input data into the model to be compressed to obtain a first initial output result; A first target output result output subunit is used to input the first input data into a first adjusted model to obtain a first target output result, wherein the model weight matrix of the first adjusted model is characterized by setting the first part of parameters to zero; a first sensitivity loss determining subunit, configured to determine a first sensitivity loss according to the first initial output result and the first target output result; and The first target weight adjustment subunit is used to adjust the first target weight according to the first sensitivity loss.

21. The device according to claim 17, wherein: The second sparse processing unit includes: A second zeroing subunit is used to reset a second part of the row model weights to zero; and The second adjustment subunit is used to adjust the second target weight in the same column as the second part weight.

22. The device according to claim 21, wherein The second zeroing subunit comprises: A row-by-row zeroing subunit is used to reset the second part of the weights of the nth row in the model weight matrix to zero, where n is an integer from 1 to N, and N represents the number of rows of the model weight matrix; The second adjustment subunit includes: The row adjustment subunit is used to adjust the second target weights from the n+1th row to the Nth row in the model weight matrix.

23. The device according to claim 21 or 22, wherein: The second adjustment subunit includes: A second initial output result output subunit, used for inputting second input data into the model to be compressed to obtain a second initial output result; A second target output result output subunit is used to input the second input data into a second adjusted model to obtain a second target output result, wherein the model weight matrix of the second adjusted model is characterized by setting the second part of parameters to zero; a second sensitivity loss determining subunit, configured to determine a second sensitivity loss according to the second initial output result and the second target output result; and The second target weight adjustment subunit is used to adjust the second target weight according to the second sensitivity loss.

24. The device according to any one of claims 15 to 19, 21 and 22, wherein: The sub-networks with coupling relationships constitute a layer structure network, and the model to be compressed includes at least one layer structure network; The fusion module includes: A layer target weight acquisition submodule, used to acquire the layer target weights associated with the layer structure network from the sparse weight matrix; A layer fusion submodule, used for performing layer fusion on the layer target weights to obtain layer fusion weights corresponding to the layer structure network; and The fusion weight determination submodule is used to determine the fusion weight according to at least one layer fusion weight corresponding to the at least one layer structure network.

25. The device according to any one of claims 15-19, 21 and 22, wherein: The sub-networks with coupling relationships constitute a layer structure network, and the model to be compressed includes at least one layer structure network; The quantization module comprises: A layer activation value obtaining submodule, used for inputting the third input data into the layer structure network to obtain a layer activation value related to the layer structure network, wherein the layer model weight of the layer structure network is the model weight related to the layer structure network in the sparse weight matrix; A data distribution determination submodule, used to determine the data distribution range according to the layer activation value and the weight value of the layer model weight; a quantization submodule, configured to quantize the layer activation value and the layer model weight according to the target compression accuracy and the data distribution range, so as to obtain a layer quantization parameter corresponding to the layer structure network; and The quantization parameter determination submodule is used to determine the quantization parameter according to at least one layer quantization parameter corresponding to the at least one layer structure network.

26. The device according to claim 25, wherein The layer model weights include layer initial weights and layer fusion weights related to the layer structure network, the layer fusion weights represent fusion weights obtained by fusing layer target weights related to the layer structure network in the sparse weight matrix, and the layer initial weights represent other weights in the layer model weights except the layer target weights; The quantization submodule comprises: A scaling factor determining unit, configured to determine a scaling factor according to the data distribution range; a quantization unit, configured to quantize the layer activation value, the layer initial weight, and the layer fusion weight into integer parameters matching the target compression accuracy according to the scaling factor and the target compression accuracy; and A quantization parameter determination unit is used to determine the layer quantization parameter according to the integer parameter.

27. The device according to claim 26, wherein: The quantization parameter determination unit comprises: A layer first activation value obtaining subunit is used to input the third input data into a quantized layer structure network corresponding to the layer structure network to obtain a layer first activation value, wherein the model weight of the quantized layer structure network is the model weight represented by the integer parameter; a first quantization loss determining subunit, configured to determine a first quantization loss according to the layer first activation value and the layer activation value; and The first layer quantization parameter determination subunit is used to determine the layer quantization parameter according to the integer parameter and the first quantization loss.

28. The device according to claim 27, wherein The first-layer quantization parameter determination subunit includes: The i-th adjustment subunit is used to perform an i-th adjustment on the integer parameter according to the i-th quantization loss to obtain an i-th adjusted integer parameter, where i is an integer between 1 and I, and I is a predefined positive integer; The layer i+1 activation value obtaining subunit is used to input the third input data into the i-th adjustment layer structure network corresponding to the layer structure network to obtain the layer i+1 activation value, and the model weight of the i-th adjustment layer structure network is the model weight represented by the i-th adjustment integer parameter; an i+1th quantization loss determining subunit, configured to determine an i+1th quantization loss according to the i+1th activation value of the layer and the layer activation value; and The second layer quantization parameter determination subunit is used to determine the layer quantization parameter according to an integer parameter corresponding to the smallest quantization loss among the first quantization loss to the (I+1)th quantization loss.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

31. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images