Neural network model compression method, device, computing system and storage medium

Abstract

The present application discloses a compression method for a neural network model, including: obtaining a weight matrix of the neural network model; dividing the weight matrix into multiple levels of weight groups based on any one of rows, columns, and blocks, each level including multiple levels of weight groups, the level weight group including multiple weight elements, and the weight element is the next level of weight group or weight value; performing sparse processing on each level of weight group in sequence according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix. The present application can improve the sparsity of the weight matrix by dividing the weight matrix into multiple levels of weight groups based on rows or columns, and sparsely processing each level of weight group in sequence, and taking into account the model accuracy and compression ratio that can be achieved by fine-grained sparsification, thereby reducing the storage and computing requirements of the neural network model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a neural network model compression method, device, computing system and storage medium. Background Art

[0002] At present, neural networks have achieved great success in computer vision, natural language processing and other fields. However, as the scale of neural networks becomes larger and larger, the number of parameters and the amount of calculations are increasing, which restricts their use on resource-constrained hardware platforms, such as edge devices and mobile devices. Therefore, it is necessary to compress the neural network model to reduce the computing and storage requirements of the neural network and accelerate the training and reasoning of the neural network model. Sparsification is one of the methods to lightweight the model.

[0003] Network parameter sparsification is to reduce the redundant components in the network through appropriate methods to reduce the network's demand for computing and storage space. Existing network sparsification is divided into fine-grained sparsification and coarse-grained sparsification in terms of sparsity granularity. Among them, fine-grained sparsification is to sparsify the weights at different positions of the neural network. Generally, the accuracy loss of the model after sparsification is very small, or even no accuracy loss occurs. However, the sparsification method of fine-grained sparsification is unstructured, and the sparsified model parameters are not hardware-friendly. There are some problems during deployment, which will affect the actual performance. Coarse-grained sparsification is to sparsify the entire convolution kernel of the neural network weights. It is a structured sparsification method, and the performance improvement is more obvious after actual hardware deployment. However, the sparsification method of coarse-grained sparsification will lose important weights in the convolution kernel when sparsifying the entire convolution kernel, resulting in a large loss of model accuracy. Summary of the invention

[0004] In view of the above problems, the object of the present invention is to provide a compression method, device, electronic device and computer-readable storage medium for a neural network model to solve or at least alleviate the above problems.

[0005] According to a first aspect of the present invention, there is provided a compression method for a neural network model, comprising: obtaining a weight matrix of the neural network model; dividing the weight matrix into a plurality of levels of weight groups based on any one of rows, columns, and blocks, wherein each level comprises a plurality of level weight groups, and the level weight groups comprise a plurality of weight elements, and the weight elements are the next level weight groups or weight values; performing sparse processing on each level weight group in turn according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0006] Preferably, dividing the weight matrix into multiple hierarchical weight groups based on one of rows, columns and blocks includes: dividing the weight matrix into multiple zeroth-level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values; dividing multiple zeroth-level weight groups into multiple first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zeroth-level weight groups; and so on, dividing the K-1th level weight group into multiple Kth level weight groups according to any one of rows, columns and blocks, and the weight elements are K-1th level weight groups, wherein K is a positive integer.

[0007] Preferably, weight groups at different levels are divided in the same or different ways.

[0008] Preferably, the data length of the zeroth-level weight group is determined based on the bit width of an acceleration unit, and the acceleration unit is used to perform operations related to the neural network model.

[0009] Preferably, the process of sparsifying the weight groups of each level according to the hierarchical order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target includes: sparsifying the weight groups of each level in turn according to the hierarchical order to obtain a sparse weight matrix; obtaining the accuracy and / or compression ratio of the neural network model after the sparsification of the weight groups of each level; judging whether the accuracy of the neural network model after the sparsification of the weight groups of the current level is less than a first threshold; if the accuracy of the neural network model after the sparsification of the weight groups of the current level is less than the first threshold, stopping the sparsification of the weight groups of the next level and sparsifying the weight groups of the previous level to obtain the sparse weight matrix; The sparse weight matrix obtained is used as the target weight matrix; if the accuracy of the neural network model after the sparse processing of the weight group of the current level is greater than the first threshold, then determine whether the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than the second threshold; if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than or equal to the second threshold, then stop continuing to sparse the weight group of the next level, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold, then continue to sparse the weight group of the next level.

[0010] Preferably, the hierarchical order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

[0011] Preferably, the compression method also includes: dividing the weight matrix into multiple weight block matrices; determining the number of levels of division according to the density of valid data in each weight block matrix; based on one of rows, columns and blocks, dividing each weight block matrix into a corresponding number of hierarchical weight groups according to the number of levels of each weight block matrix; wherein the number of levels is inversely correlated with the density of valid data in the weight block matrix.

[0012] Preferably, the compression method also includes: determining the sparsity of each weight block matrix according to the accuracy of the weight values ​​in the matrix; and performing sparse processing on the multiple hierarchical weight groups divided by each weight block matrix in turn according to the sparsity of each weight block matrix until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain the target weight matrix.

[0013] Preferably, the data formats of different weight block matrices are the same or different, and the data formats include data type and data precision; wherein, the weight block matrices that are sensitive to model precision can choose high-precision data representation, and the weight block matrices that are insensitive to model precision can choose low-precision data representation.

[0014] Preferably, in each weight block matrix, the weight groups of each level are divided into one of rows, columns and blocks, and the weight groups of different levels adopt the same or different division methods; the weight groups of the same level of different weight block matrices adopt the same or different division methods.

[0015] Preferably, the sparse processing includes structured sparse processing or unstructured sparse processing.

[0016] According to a second aspect of the present invention, there is provided a compression device for a neural network model, comprising: an acquisition module for acquiring a weight matrix of the neural network model; a hierarchical division module for dividing the weight matrix into a plurality of hierarchical weight groups based on any one of rows, columns, and blocks, wherein each hierarchical weight group comprises a plurality of hierarchical weight groups, and the hierarchical weight groups comprise a plurality of weight elements, wherein the weight elements are weight groups or weight values ​​of the next hierarchical weight group; and a sparse module for performing sparse processing on the weight groups of each hierarchical level in turn according to the hierarchical order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0017] Preferably, the hierarchical division module is also used to divide the weight matrix into multiple zero-level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values; divide the multiple zero-level weight groups into multiple first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zero-level weight groups; and so on, divide the K-1-th level weight group into multiple K-th level weight groups according to any one of rows, columns and blocks, and the weight elements are K-1-th level weight groups, wherein K is a positive integer.

[0018] Preferably, weight groups at different levels are divided in the same or different ways.

[0019] Preferably, the data length of the zeroth-level weight group is determined based on the bit width of an acceleration unit, and the acceleration unit is used to perform operations related to the neural network model.

[0020] Preferably, the sparse module includes: a sparse unit, which is used to perform sparse processing on the weight group of each level in turn according to the hierarchical order to obtain a sparse weight matrix; an acquisition unit, which is used to obtain the accuracy and / or compression ratio of the neural network model after the weight group of each level is sparsely processed; a judgment unit, which is used to judge whether the accuracy of the neural network model after the weight group of the current level is sparsely processed is less than a first threshold; the sparse unit is also used to stop continuing to perform sparse processing on the weight group of the next level if the accuracy of the neural network model after the weight group of the current level is sparsely processed is less than the first threshold, and use the sparse weight matrix obtained after the weight group of the previous level is sparsely processed as the target weight matrix ; The judgment unit is also used to judge whether the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than the second threshold if the accuracy of the neural network model after the sparse processing of the weight group of the current level is greater than the first threshold; The sparse unit is also used to stop the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than or equal to the second threshold, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; and continue the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold.

[0021] Preferably, the hierarchical order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

[0022] Preferably, the compression device also includes: a block division module, used to divide the weight matrix into multiple weight block matrices; a level determination module, used to determine the number of levels of division according to the density of valid data in each weight block matrix; the level division module is also used to divide each weight block matrix into a corresponding number of level weight groups based on one of rows, columns and blocks according to the number of levels of each weight block; wherein the number of levels is inversely correlated with the density of valid data in the weight block matrix.

[0023] Preferably, the sparse module is also used to determine the sparsity of each weight block matrix according to the accuracy of the weight values ​​in the matrix; and to perform sparse processing on the multiple levels of weight groups divided by each weight block matrix in turn according to the sparsity of each weight block matrix until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain the target weight matrix.

[0024] Preferably, the data formats of different weight block matrices are the same or different, and the data formats include data type and data precision; wherein, the weight block matrices that are sensitive to model precision can choose high-precision data representation, and the weight block matrices that are insensitive to model precision can choose low-precision data representation.

[0025] Preferably, in each weight block matrix, the weight groups of each level are divided into one of rows, columns and blocks, and the weight groups of different levels adopt the same or different division methods; the weight groups of the same level of different weight block matrices adopt the same or different division methods.

[0026] According to a third aspect of the present invention, there is provided a computing system comprising a system memory, a processor and an acceleration unit which are mutually coupled, wherein the system memory comprises a program and a neural network model, the program comprises instructions of the above-mentioned neural network model compression method; the processor is used to execute the instructions to compress the neural network model, and write the calculation data of the compressed neural network model into the acceleration unit; and the acceleration unit is used to perform operations related to the neural network model.

[0027] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program implements the above-described method when executed by a processor.

[0028] The compression method of the neural network model provided by the present invention divides the weight matrix into multiple hierarchical weight groups based on one of rows, columns and blocks, and sparsifies each hierarchical weight group in turn, which can improve the sparsity of the weight matrix and take into account the model accuracy and compression ratio that can be achieved by fine-grained sparsification, thereby reducing the storage and computing requirements of the neural network model.

[0029] Furthermore, the sparse model is also structured, which is more friendly to hardware deployment and can take into account both accuracy and performance.

[0030] Furthermore, the weight matrix is ​​divided into multiple weight block matrices. Different weight block matrices can achieve different levels of sparsity, which can retain more effective weight values ​​and improve calculation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0032] Figure 1 A schematic diagram showing a computing system according to one embodiment of the present invention;

[0033] Figure 2 A flowchart showing a method for compressing a neural network model according to an embodiment of the present invention;

[0034] Figure 3 A flowchart showing step S220 of one embodiment of the present invention;

[0035] Figure 4 A schematic diagram showing the division of hierarchical weight groups based on rows according to an embodiment of the present invention;

[0036] Figure 5 A schematic diagram showing the division of hierarchical weight groups based on columns according to an embodiment of the present invention;

[0037] Figure 6 A schematic diagram showing a block-based division of hierarchical weight groups according to an embodiment of the present invention;

[0038] Figure 7 A flowchart of step S230 according to an embodiment of the present invention is shown;

[0039] Figure 8 A flowchart showing a method for compressing a neural network model according to an embodiment of the present invention;

[0040] Fig. 9 A schematic diagram showing a compression device for a neural network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Various embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by the same or similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale.

[0042] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0043] Figure 1 FIG. 1 is a schematic diagram of a computing system 100 according to an embodiment of the present invention. Figure 1As shown, the computing system 100 includes one or more processors 110, a system memory 120 communicatively connected to at least one processor 110, and one or more acceleration units 200 communicatively connected to the processor 110 and the system memory 120, and can be applied to laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, cellular phones, portable media players, handheld devices, and various other electronic devices.

[0044] Depending on the desired configuration, the processor 110 can be any type of processor, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), or any combination thereof. The system memory 120 can be any type of memory, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The acceleration unit 200 can perform operations related to the neural network model and is a special processing unit designed to increase the processing speed of operations related to the neural network model. The acceleration unit 200 can be implemented as a processing unit (KPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any combination thereof, which is specially designed for neural network model operation processing.

[0045] The processor 110 is connected to other components of the computing system 100 via the system bus 130, which is used to transmit communication signals, such as address information or control information, between the processor 110 and other components in the computing system 100. One or more processors 110 may also be connected to the acceleration unit 200 via the interconnection bus 140, which is used to transmit communication signals, such as calculation data of a neural network model, between the processor 110 and the acceleration unit 200 at high speed.

[0046] System memory 120 may include an operating system, one or more application programs, and program data. Processor 110 may use the program data to execute instructions of the application programs stored in system memory 120, thereby performing various functions and data processing.

[0047] The system memory 120 may also include various neural network models, such as parameters of the model. These neural network models are deployed to the acceleration unit 200 by the processor 110 when needed, and the acceleration unit 200 performs the operation of the neural network model. For example, the processor 110 may send the address information of the parameters of the neural network model in the system memory 120 to the acceleration unit 200 in the form of instructions. When the acceleration unit 200 actually uses the neural network model for calculation, it will directly address these parameters in the system memory 210 according to the address information of these parameters (such as weight values) in the system memory 210, and temporarily store them in the on-chip memory of the acceleration unit 200. When the acceleration unit 200 actually uses the neural network model for calculation, the processor 110 will also send the input data of the model to the acceleration unit 200 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 200. In this way, the acceleration unit 200 can perform reasoning calculations based on these input data and the parameters in the model (such as weight values).

[0048] In the architecture design of traditional processors, such as central processing units, the control unit and storage unit occupy a large part of the space, and the computing unit occupies insufficient space, which makes it very effective in logic control but not efficient in large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed of different functions and different fields.

[0049] The acceleration unit 200 disclosed in the embodiment of the present invention adopts a data-driven parallel computing architecture, and is a processing unit for processing a large number of operations (such as convolution, pooling, etc.) of each neural network model node. Since the calculation data and intermediate results in the large number of operations of each neural network model node are closely related in the entire calculation process, they will be used frequently. If the existing central processing unit architecture is adopted, since the memory capacity in the core of the central processing unit is very small, a large number of frequent accesses to the external core memory are required, resulting in inefficient processing. However, the use of this acceleration unit dedicated to accelerating the calculation processing speed of the neural network model, because each core has an on-chip memory with a storage capacity suitable for the calculation of the neural network model, can avoid frequent accesses to the memory outside the core, which can greatly improve the processing efficiency and computing performance.

[0050] It should be noted that the computing system 100 shown is exemplary and not restrictive, and in other implementations, the computing system 100 may include different and / or additional modules.

[0051] At present, neural networks have achieved great success in computer vision, natural language processing and other fields. However, as the scale of neural networks becomes larger and larger, the number of parameters and the amount of calculations are increasing, which restricts their use on resource-constrained hardware platforms, such as edge devices and mobile devices. Therefore, it is necessary to compress the neural network model, reduce the computing and storage requirements of the neural network, and accelerate the training and reasoning of the neural network model.

[0052] In an embodiment according to the present invention, the application 122 of the computing system 100 may be implemented as a compression device 400 for a neural network model and include a plurality of instructions for executing a compression method for a neural network model.

[0053] Figure 2 FIG. 1 is a flowchart of a neural network model compression method according to an embodiment of the present invention. Figure 2 As shown, the neural network model compression method includes the following steps.

[0054] In step S210, a weight matrix of the neural network model is obtained.

[0055] In this embodiment, the neural network model includes but is not limited to a deep neural network model (DNN), a convolutional neural network model (CNN), a recursive neural network model (RNN), etc. The neural network model includes multiple neural network layers, and the weight data of each neural network layer can be represented in the form of a matrix, that is, a weight matrix.

[0056] The compression method of the neural network model according to the embodiment of the present invention can obtain the weight matrix of any neural network layer and compress it independently. Preferably, in some embodiments, the weight matrix of a neural network layer with severe parameter redundancy can be obtained, such as a fully connected layer in a convolutional neural network model.

[0057] In step S220, based on one of rows, columns and blocks, multiple levels of weight groups are provided, each level includes multiple level weight groups, and the level weight group includes multiple weight elements, and the weight element is the next level weight group or weight value.

[0058] In this embodiment, see Figure 3 , step S220 includes the following steps.

[0059] In step S221, the weight matrix is ​​divided into a plurality of zero-th level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values.

[0060] Specifically, the weight matrix is ​​divided into any one of rows, columns and blocks, and each zero-th level weight group includes multiple weight values. Four weight values ​​are taken as an example for explanation, but it is not limited to this.

[0061] In step S222, the plurality of zero-level weight groups are divided into a plurality of first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zero-level weight groups.

[0062] Specifically, each zeroth-level weight group is taken as a weight element, multiple zeroth-level weight groups constitute a zeroth-level weight matrix, the zeroth-level weight matrix is ​​divided into any one of rows, columns and blocks, and each first-level weight group includes multiple zeroth-level weight values.

[0063] In step S223, by analogy, the K-1th level weight group is divided into multiple K-th level weight groups according to any one of rows, columns and blocks, and the weight elements are the K-1th level weight groups, where K is a positive integer.

[0064] In some embodiments, any level weight group can be divided based on the rows of the weight matrix, that is, each row of the weight matrix or each row of the previous level weight group matrix is ​​divided into at least one any level weight group to obtain multiple any level weight groups. In the multiple any level weight groups obtained based on row division, the weight elements contained in the same weight group are arranged in ascending order of columns, that is, any level weight groups obtained by the same row division are arranged in ascending order of columns, and any level weight groups obtained by different row divisions are arranged in ascending order of rows.

[0065] In other embodiments, any level weight group can be divided based on the columns of the weight matrix, that is, each column of the weight matrix or each column of the previous level weight group matrix is ​​divided into at least one any level weight group to obtain multiple any level weight groups. In the multiple any level weight groups obtained based on column division, the weight elements contained in the same weight group are arranged in ascending order of rows, that is, any level weight groups obtained by the same column division are arranged in ascending order of rows, and any level weight groups obtained by different column divisions are arranged in ascending order of columns.

[0066] In some other embodiments, any level weight group can be divided based on the block of the weight matrix, that is, the weight matrix or the weight group matrix of the previous level is divided into at least one any level weight group according to the size of the block to obtain multiple any level weight groups. In the multiple any level weight groups obtained based on the block division, the weight elements contained in the same weight group are arranged in ascending order of the rows and columns of the block.

[0067] In other embodiments, the weight groups of each level can be divided based on any one of rows, columns or blocks. The weight groups of different levels adopt the same or different division methods. For example, the weight groups of all levels are divided by rows, or the weight groups of all levels are divided by columns, or the weight groups of all levels are divided by blocks; or the weight groups of some levels are divided by rows, and the weight groups of the remaining levels are divided by columns; or the weight groups of some levels are divided by rows, and the weight groups of the remaining levels are divided by blocks; or the weight groups of some levels are divided by columns, and the weight groups of the remaining levels are divided by blocks; or the weight groups of some levels are divided by rows, the weight groups of some levels are divided by columns, and the weight groups of the remaining levels are divided by blocks.

[0068] The data length of each zero-level weight group can be determined based on the bit width of the acceleration unit 200. The bit width of the acceleration unit 200 is the number of bits of data (binary) that the acceleration unit 200 can process at one time. Since the data length of each zero-level weight group is determined based on the number of weight values ​​contained in the weight group and the number of bits required to store the weight values, the number of weight values ​​contained in each zero-level weight group can be determined based on the bit width of the acceleration unit 200 and the number of bits required to store the weight values. For example, the data length of each zero-level weight group is equal to the bit width of the acceleration unit 200, and the number of weight values ​​contained in each zero-level weight group can be obtained by dividing the bit width of the acceleration unit 200 by the number of bits required to store the weight values. The number of bits required to store the weight values ​​is determined by the data type of the weight values.

[0069] Assuming that the obtained weight matrix is ​​an N×M matrix, the number of weight values ​​contained in each zero-level weight group is obtained based on the bit width of the acceleration unit 200 and the number of bits required to store the weight values, which is k. If the bit width of the acceleration unit 200 is 32 bits, and the data type of the weight value is int8, the number of bits for storing the weight value is 8 bits, and the number of weight values ​​contained in the zero-level weight group is k=4; if the bit width of the acceleration unit 200 is 64 bits, and the data type of the weight value is int4, the number of bits for storing the weight value is 4 bits, and the number of weight values ​​contained in the zero-level weight group is k=16.

[0070] If the hierarchical weight groups are divided based on rows, when M is an integer multiple of k, each row can be divided into M / k zero-level weight groups, and the entire weight matrix can be divided into N*M / k zero-level weight groups. When M is not an integer multiple of k, it is necessary to expand the weight matrix columns and then divide the expanded weight matrix into multiple zero-level weight groups. For example, kb specified columns (b is the remainder of M divided by k) can be added after the last column of the weight matrix, and the N elements of each specified column are all 0. In this way, each row can be divided into a+1 zero-level weight groups (a is the quotient of M divided by k), and the entire weight matrix can be divided into N*(a+1) zero-level weight groups.

[0071] If the hierarchical weight groups are divided based on columns, when N is an integer multiple of k, each column can be divided into N / k zero-level weight groups, and the entire weight matrix can be divided into M*N / k zero-level weight groups. When N is not an integer multiple of k, it is necessary to expand the rows of the weight matrix and then divide the expanded weight matrix into multiple zero-level weight groups. For example, kd specified rows (d is the remainder of N divided by k) can be added after the last row of the weight matrix, and the M elements of each specified row are all 0. In this way, each column can be divided into c+1 zero-level weight groups (c is the quotient of N divided by k), and the entire weight matrix can be divided into N*(c+1) zero-level weight groups.

[0072] If the hierarchical weight group is divided based on blocks, the size of the block is n*m, where n and m are both positive integers greater than 1, and k=n*m; if N is an integer multiple of n and M is an integer multiple of m, the entire weight matrix can be divided into N*M / (n*m) zero-level weight groups. If N is not an integer multiple of n and / or M is not an integer multiple of m, the weight matrix is ​​expanded in corresponding rows and / or columns, and the expanded weight matrix is ​​divided into multiple zero-level weight matrices. For example, add a specified row of ne (e is the remainder of N divided by n) and / or a specified column of mf (f is the remainder of M divided by m) to the last row and / or last column of the weight matrix.

[0073] In some embodiments, the above-mentioned method can be used to expand the weight matrix when dividing each level of weight groups.

[0074] Figure 4 The following is a schematic diagram of dividing hierarchical weight groups based on rows according to an embodiment of the present invention. This is illustrated by taking k=4 as an example, but is not limited to this. Assuming that the weight matrix obtained is an N×16 matrix, at the zeroth level L0, each row of the weight matrix is ​​divided into a weight group every 4, and a total of 4N ordered zeroth level weight groups GL0 are divided. 1,1 , GL0 1,2 , GL0 1,3 , GL0 1,4 , ..., GL0 N,1 , GL0 N,2 , GL0 N,3 , GL0 N,4 , 4N zero-level weight groups form an N×4 zero-level weight group matrix. Each row of the weight matrix can be divided into 4 zero-level weight groups, and each zero-level weight group includes 4 weight values. At the first level L1, each row of the zero-level weight group matrix is ​​divided into one weight group every 4, and N ordered first-level weight groups GL1 are obtained. 1,1 , GL1 2,1 , ..., GL1N,1 .

[0075] Figure 5 A schematic diagram of dividing hierarchical weight groups based on columns according to an embodiment of the present invention is shown. This is illustrated by taking k=4 as an example, but is not limited to this. Assuming that the acquired weight matrix is ​​a 16×M matrix, at the zeroth level L0, each column of the weight matrix is ​​divided into a weight group every 4, and a total of 4M ordered zeroth level weight groups GL0 are divided. 1,1 , GL0 2,1 , GL0 3,1 , GL0 4,1 , ..., GL0 1,M , GL0 2,M , GL0 3,M , GL0 4,M , 4M zero-level weight groups form a 4×M zero-level weight group matrix. Each row of the weight matrix can be divided into 4 zero-level weight groups, and each zero-level weight group includes 4 weight values. At the first level L1, each column of the zero-level weight group matrix is ​​divided into one weight group every 4, and M ordered first-level weight groups GL1 are obtained. 1,1 , GL1 1,2 , ..., GL1 1,M .

[0076] Figure 6 A schematic diagram of dividing hierarchical weight groups based on blocks according to an embodiment of the present invention is shown. Take k=4 as an example, and the size of the block is 2×2, but it is not limited to this. Assuming that the obtained weight matrix is ​​an N×16 matrix, at the zeroth level L0, the weight matrix is ​​divided into a weight group according to the size of the block, and is divided into 4N ordered zeroth level weight groups GL0 1,1 , GL0 1,2 , GL0 1,3 , GL0 1,4 , GL0 1,5 , GL0 1,6 , GL0 1,7 , GL0 1,8 , ..., GL0 N / 2,1 , GL0 N / 2,2 , GL0 N / 2,3 , GL0 N / 2,4 , GL0 N / 2,5 , GL0 N / 2,6 , GL0 N / 2,7 , GL0 N / 2,8, 4N zero-level weight groups form an N / 2×8 zero-level weight group matrix. Every two rows of the weight matrix can be divided into 8 zero-level weight groups, and each zero-level weight group includes 4 weight values. At the first level L1, the zero-level weight group matrix divides a weight group according to the size of the block, and obtains N ordered first-level weight groups GL1 1,1 , GL1 1,2 , GL1 1,3 , GL1 1,4 , ..., GL1 N / 4,1 , GL1 N / 4,2 , GL1 N / 4,3 , GL1 N / 4,4 , N zero-th level weight groups form an N / 4×4 first-level weight group matrix. At the second level L2, the first-level weight group matrix is ​​divided into a weight group according to the size of the block, and N / 4 ordered second-level weight groups GL2 are obtained. 1,1 , GL2 1,2 , ..., GL1 N / 8,1 , GL1 N / 8,2 , N / 4 zero-level weight groups form an N / 8×2 second-level weight group matrix.

[0077] In step S230, the weight groups of each level are sparsified in sequence according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0078] In some embodiments, see Figure 7 , step S230 includes the following steps.

[0079] In step S231, the weight group of each level is sparsely processed in sequence according to the level order to obtain a sparse weight matrix.

[0080] In this embodiment, the level order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

[0081] In some embodiments, the hierarchy order is from the zeroth level to the Kth level, that is, the weight groups of each level are thinned out in turn from the zeroth level to the Kth level, and the level number of the next level is greater than the level number of the current level.

[0082] In some embodiments, the hierarchy order is from the Kth level to the zeroth level, that is, the weight groups of each level are thinned out in turn from the Kth level to the zeroth level, and the number of levels of the next level is less than the number of levels of the current level.

[0083] In this embodiment, fine-grained sparsity is used as the method of sparse processing. This embodiment is illustrated by taking k=4 and sparsity 2:4 as an example, but is not limited to this. Specifically, when the weight group of each level is sparse, 2 weight elements out of the 4 weight elements in each level weight group are set to 0. Sparse processing includes structured sparsity or unstructured sparsity, for example, the positions of 2 weight elements are selected to be the same among the 4 weight elements to achieve structured sparsity, or the positions of 2 weight elements are selected to be randomly different among every 4 weight elements to achieve unstructured sparsity.

[0084] In step S232, the accuracy and / or compression ratio of the neural network model is obtained after the weight group of each level is sparsely processed. In step S233, it is determined whether the accuracy of the neural network model after the weight group of the current level is sparsely processed is less than a first threshold.

[0085] In this embodiment, when the accuracy of the neural network model after the weight group sparsification processing of the current level is less than the first threshold, step S234 is executed; if the accuracy of the neural network model after the weight group sparsification processing of the current level is not less than the first threshold, step S235 or step S237 is executed.

[0086] In step S234, the sparse processing of the weight group of the next level is stopped, and the sparse weight matrix obtained after the sparse processing of the weight group of the previous level is used as the target weight matrix.

[0087] In step S235, it is determined whether the compression ratio of the neural network model after the weight group sparsification processing of the current level is greater than the second threshold.

[0088] In this embodiment, if the compression ratio of the neural network model after the current level of weight group sparse processing is greater than the second threshold, step S236 is executed; if the compression ratio of the neural network model after the current level of weight group sparse processing is not greater than the second threshold, step S237 is executed.

[0089] In step S236, the sparse processing of the weight group of the next level is stopped, and the sparse weight matrix obtained after the sparse processing of the weight group of the current level is used as the target weight matrix.

[0090] In step S237, the thinning process is continued for the weight group of the next level.

[0091] In some embodiments, considering both the accuracy and compression ratio of the neural network model, steps S233, S234, S235, S236, and S237 are performed. That is, when the accuracy of the neural network model is lower than the first threshold, the recursive sparse compression is stopped. When the accuracy of the neural network model is not lower than the first threshold, the recursive sparse compression is continued until the compression ratio is not less than the second threshold.

[0092] In some embodiments, when only the accuracy of the neural network model is considered, step S233, step S234 are executed, and step S235 and step S236 are skipped to directly execute step S237. That is, when the accuracy of the neural network model is lower than the first threshold, the recursive sparse compression is stopped. When the accuracy of the neural network model is not lower than the first threshold, the recursive sparse compression is continued.

[0093] In some embodiments, when only the compression ratio of the neural network model is considered, steps S233 and S234 are skipped, and steps S235, S236, and S237 are directly executed. That is, when the compression ratio of the neural network model is not less than the second threshold, the recursive sparse compression is stopped. When the compression ratio of the neural network model is less than the second threshold, the recursive sparse compression is continued.

[0094] The compression method of the neural network model provided by the present invention divides the weight matrix into multiple hierarchical weight groups based on rows or columns, and sparsifies each hierarchical weight group in turn, which can improve the sparsity of the weight matrix and take into account the model accuracy and compression ratio that can be achieved by fine-grained sparsification, thereby reducing the storage and computing requirements of the neural network model.

[0095] Furthermore, the sparse model is also structured, which is more friendly to hardware deployment and can take into account both accuracy and performance.

[0096] Figure 8 FIG. 1 is a flowchart of a neural network model compression method according to an embodiment of the present invention. Figure 8 As shown, the neural network model compression method includes the following steps.

[0097] In step S310, a weight matrix of the neural network model is obtained.

[0098] In this embodiment, the neural network model includes but is not limited to a deep neural network model (DKK), a convolutional neural network model (CKK), a recursive neural network model (RKK), etc. The neural network model includes multiple neural network layers, and the weight data of each neural network layer can be represented in the form of a matrix, that is, a weight matrix.

[0099] The compression method of the neural network model according to the embodiment of the present invention can obtain the weight matrix of any neural network layer and compress it independently. Preferably, in some embodiments, the weight matrix of a neural network layer with severe parameter redundancy can be obtained, such as a fully connected layer in a convolutional neural network model.

[0100] In step S320, the weight matrix is ​​divided into a plurality of weight block matrices.

[0101] In this embodiment, a weight matrix can be divided into multiple weight block matrices, for example, the weight matrix is ​​divided into L weight block matrices. The data formats of different weight block matrices are the same or different. The data format includes data type and data precision. Data types include floating point numbers (float) and fixed point numbers (int). Data precision includes 4-bit precision, 8-bit precision, half precision (16-bit), single precision (32-bit) and double precision (64-bit), etc., but are not limited to this. Data formats include, for example, int4, int8, int16, fp16, fp32, etc., but are not limited to this. Weight block matrices that are sensitive to model accuracy can choose high-precision data representation, and conversely, weight block matrices that are insensitive to model accuracy can choose low-precision data representation. Flexible data type and data precision selection strategies help optimize model performance and resource utilization efficiency. Model accuracy sensitivity refers to the high accuracy requirements for model output results, that is, the model needs to predict or output results as accurately as possible. In this case, the accuracy requirements for input data and weight parameters are also high to ensure that the output results of the model have a high degree of accuracy and reliability. Therefore, when the model accuracy is sensitive, high-precision data representation is usually chosen to improve the accuracy and performance of the model. In some embodiments, the sparsity of different weight block matrices can be the same or different, and different sparsities can be selected according to data accuracy or the density of valid data, such as 1:2, 2:4, 1:8, 1:16, 1:32, etc.

[0102] In some embodiments, the weight block matrix can also be quantized to reduce the number of bits required to store the weight values ​​to further compress the neural network model. Generally speaking, the parameters of the neural network model are represented by 32-bit floating-point numbers, and quantization can store the weight values ​​originally represented as floating-point numbers with a lower number of bits (e.g., 4-8 bits).

[0103] In step S330, the number of levels for division of each weight block matrix is ​​determined according to the density of valid data in the matrix.

[0104] In this embodiment, the number of hierarchical divisions of the L weight block matrices is K1, K2, ..., K. L. The valid data in each weight block matrix is ​​the weight value greater than the preset threshold. The density of valid data is the ratio of the number of valid data in a weight block matrix to the number of all weight values ​​in a weight block matrix. The more valid data there are, the greater its density is, and accordingly, the fewer the number of levels to be divided. That is, the number of levels K is inversely correlated with the density of valid data in the weight block matrix.

[0105] In step S340, each weight block matrix is ​​divided into a corresponding number of hierarchical weight groups according to the number of levels of each weight block matrix based on one of rows, columns and blocks.

[0106] In this embodiment, for the jth weight block matrix, the weight block matrix is ​​divided into a plurality of zero-level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values; the plurality of zero-level weight groups are divided into a plurality of first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zero-level weight groups, and so on. j-1 The hierarchical weight group is divided into multiple K-th j hierarchical weight group, the weight element is the Kth j-1 Hierarchical weight group, where K j is a positive integer, j=1,2,…,L.

[0107] In some embodiments, in the same weight block matrix, the weight groups of each level can be divided based on any of rows, columns or blocks. Weight groups of different levels adopt the same or different division methods. For example, the weight groups of all levels are divided by rows, or the weight groups of all levels are divided by columns, or the weight groups of all levels are divided by blocks; or the weight groups of some levels are divided by rows, and the weight groups of the remaining levels are divided by columns; or the weight groups of some levels are divided by rows, and the weight groups of the remaining levels are divided by blocks; or the weight groups of some levels are divided by columns, and the weight groups of the remaining levels are divided by blocks; or the weight groups of some levels are divided by rows, the weight groups of some levels are divided by columns, and the weight groups of the remaining levels are divided by blocks.

[0108] In some embodiments, weight groups at the same level of different weight block matrices are divided in the same or different ways.

[0109] In step S350, the multiple levels of weight groups divided by each weight block matrix are sparsely processed in turn according to the sparsity of each weight block matrix until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0110] In some embodiments, the hierarchical weight groups of each weight block matrix are sparsified in turn according to the sparsity of each weight block matrix in hierarchical order to obtain a sparse weight matrix; the accuracy and / or compression ratio of the neural network model is obtained after the sparsification processing of the weight groups at each level; it is determined whether the accuracy and / or compression ratio of the neural network model after the sparsification processing of the weight groups at the current level reaches a predetermined target; if the accuracy and / or compression ratio of the neural network model after the sparsification processing of the weight groups at the current level is less than the predetermined target, the sparsification processing of the weight groups at the next level continues; if the accuracy and / or compression ratio of the neural network model after the sparsification processing of the weight groups at the current level is greater than or equal to the predetermined target, the sparse weight matrix obtained after the sparsification processing of the weight groups at the current level is used as the target weight matrix.

[0111] In this embodiment, fine-grained sparsity is used as the method of sparse processing. This embodiment is illustrated by taking k=4 and sparsity 2:4 as an example, but is not limited to this. Specifically, when the weight group of each level is sparse, 2 weight elements out of the 4 weight elements in each level weight group are set to 0. Sparse processing includes structured sparsity or unstructured sparsity, for example, the positions of 2 weight elements are selected to be the same among the 4 weight elements to achieve structured sparsity, or the positions of 2 weight elements are selected to be randomly different among every 4 weight elements to achieve unstructured sparsity.

[0112] In some embodiments, the weight groups of each level may be thinned out in sequence from the zeroth level to the Kth level, and the number of levels of the next level is greater than the number of levels of the current level.

[0113] In some embodiments, the weight groups of each level are thinned out in sequence from the zeroth level to the Kth level, and the number of levels of the next level is less than the number of levels of the current level.

[0114] The compression method of the neural network model provided by the present invention divides the weight matrix into multiple weight block matrices. Different weight block matrices can achieve different levels of sparsity, retain more effective weight values, and improve calculation accuracy.

[0115] Fig. 9 FIG. 1 is a schematic diagram showing a compression device for a neural network model according to an embodiment of the present invention. Fig. 9 As shown, the compression device 400 of the neural network model includes an acquisition module 410, a hierarchical division module 420 and a sparse module 430.

[0116] The acquisition module 410 is used to acquire the weight matrix of the neural network model.

[0117] The hierarchical division module 420 is used to divide the weight matrix into multiple hierarchical weight groups based on any one of rows, columns, and blocks, each hierarchical level includes multiple hierarchical weight groups, and the hierarchical weight groups include multiple weight elements, which are the weight groups or weight values ​​of the next hierarchical level.

[0118] In some embodiments, the hierarchical division module 420 is also used to divide the weight matrix into multiple zero-level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values; divide multiple zero-level weight groups into multiple first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zero-level weight groups; and so on, divide the K-1-th level weight group into multiple K-th level weight groups according to any one of rows, columns and blocks, and the weight elements are K-1-th level weight groups, where K is a positive integer. Weight groups of different levels use the same or different division methods. The data length of the zero-th level weight group is determined based on the bit width of the acceleration unit, and the acceleration unit is used to perform operations related to the neural network model.

[0119] The sparse module 430 is used to perform sparse processing on the weight group of each level in sequence according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0120] In this embodiment, the sparse module 430 includes a sparse unit 431, an acquisition unit 432, and a judgment unit 433, wherein the sparse unit 431 is used to perform sparse processing on the weight group of each level in turn according to the hierarchical order to obtain a sparse weight matrix; the acquisition unit 432 is used to obtain the accuracy and / or compression ratio of the neural network model after the sparse processing of the weight group of each level; the judgment unit 433 is used to judge whether the accuracy of the neural network model after the sparse processing of the weight group of the current level is less than a first threshold; the sparse unit 431 is also used to stop continuing to perform sparse processing on the weight group of the next level if the accuracy of the neural network model after the sparse processing of the weight group of the current level is less than the first threshold, and perform sparse processing on the weight group of the previous level. The sparse weight matrix obtained after the sparse processing is used as the target weight matrix; the judgment unit 433 is also used to judge whether the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than the second threshold if the accuracy of the neural network model after the sparse processing of the weight group of the current level is greater than the first threshold; the sparse unit 431 is also used to stop the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than or equal to the second threshold, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; and if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold, continue to sparse the weight group of the next level. The hierarchical order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

[0121] In some embodiments, the judgment unit 433 is used to judge whether the accuracy of the neural network model after the sparsification processing of the weight group of the current level is less than a first threshold; the sparse unit 431 is also used to stop the sparsification processing of the weight group of the next level if the accuracy of the neural network model after the sparsification processing of the weight group of the current level is less than the first threshold, and use the sparse weight matrix obtained after the sparsification processing of the weight group of the previous level as the target weight matrix, and continue the sparsification processing of the weight group of the next level if the accuracy of the neural network model after the sparsification processing of the weight group of the current level is not less than the first threshold.

[0122] In some embodiments, the judgment unit 433 is used to judge whether the compression ratio of the neural network model after the current level of weight group sparse processing is greater than a second threshold; the sparse unit 431 is also used to stop the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the current level of weight group sparse processing is greater than or equal to the second threshold, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; and continue the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold.

[0123] In some embodiments, the compression device 400 further includes a block division module 440 and a level determination module 450. The block division module 440 is used to divide the weight matrix into a plurality of weight block matrices. The level determination module 450 is used to determine the number of levels of division according to the density of valid data in each weight block matrix.

[0124] The hierarchical division module 420 is also used to divide each weight block matrix into a corresponding number of hierarchical weight groups based on one of rows, columns and blocks according to the number of levels of each weight block; wherein the number of levels is inversely correlated with the density of valid data in the weight block matrix.

[0125] The sparse module 430 is also used to determine the sparsity of each weight block matrix according to the accuracy of the weight values ​​in the matrix; and to perform sparse processing on the multiple hierarchical weight groups divided by each weight block matrix in turn according to the sparsity of each weight block matrix until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix.

[0126] For detailed processing logic and implementation process of each module in the compression device 400 of the neural network model, please refer to the above combined Figure 1-Figure 8 The description of the compression method of the neural network model will not be repeated here.

[0127] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0128] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0130] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for compressing a neural network model, characterized in that: include: Get the weight matrix of the neural network model; Based on any one of rows, columns, and blocks, the weight matrix is ​​divided into multiple levels of weight groups, the multiple levels include the zeroth level to the Kth level, K is a positive integer, and each level of weight group includes multiple weight elements, wherein when the level is the zeroth level, the weight element is a weight value; when the level is any one of the first level to the Kth level, the weight element is a previous level weight group, and multiple Kth level weight groups are obtained by dividing multiple K-1th level weight groups according to any one of rows, columns, and blocks; Performing sparse processing on each level weight group in turn according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix; Among them, the data length of the zeroth-level weight group is determined based on the bit width of the acceleration unit, the acceleration unit is used to perform operations related to the neural network model, and the number of weight elements in the zeroth-level weight group is determined by the data length of the zeroth-level weight group and the data type of the weight value.

2. The compression method according to claim 1, characterized in that: Based on one of the rows, columns and blocks, the weight matrix is ​​divided into multiple levels of weight groups including: Divide the weight matrix into a plurality of zero-level weight groups according to any one of rows, columns and blocks, wherein the weight elements are weight values; Dividing the plurality of zero-level weight groups into a plurality of first-level weight groups by any one of rows, columns and blocks, wherein the weight elements are zero-level weight groups; By analogy, the K-1th level weight group is divided into multiple K-th level weight groups according to any one of rows, columns and blocks, and the weight element is the K-1th level weight group, where K is a positive integer; Among them, weight groups at different levels adopt the same or different division methods.

3. The compression method according to claim 2, characterized in that: The process of sparsifying each level weight group according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target includes: The weight groups of each level are sparsely processed in order of levels to obtain a sparse weight matrix; Obtaining the accuracy and / or compression ratio of the neural network model after sparsification of the weight groups at each level; Determine whether the accuracy of the neural network model after the weight group sparsification processing of the current level is less than a first threshold; If the accuracy of the neural network model after the sparse processing of the weight group of the current level is less than the first threshold, the sparse processing of the weight group of the next level is stopped, and the sparse weight matrix obtained after the sparse processing of the weight group of the previous level is used as the target weight matrix; If the accuracy of the neural network model after the weight group sparsification processing of the current level is greater than the first threshold, then determine whether the compression ratio of the neural network model after the weight group sparsification processing of the current level is greater than the second threshold; If the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than or equal to the second threshold, stop the sparse processing of the weight group of the next level, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; If the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold, the sparse processing of the weight group of the next level will continue; The hierarchical order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

4. The compression method according to claim 1, characterized in that: Also includes: Dividing the weight matrix into a plurality of weight block matrices; The number of levels of division is determined according to the density of valid data in each weight block matrix; Based on one of rows, columns and blocks, each weight block matrix is ​​divided into a corresponding number of level weight groups according to the number of levels of each weight block matrix; Determine the sparsity of each weight block matrix according to the precision of the weight values; According to the sparsity of each weight block matrix, the multiple levels of weight groups divided by each weight block matrix are sequentially subjected to sparse processing until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix; Among them, the number of levels is inversely correlated with the density of valid data in the weight block matrix.

5. The compression method according to claim 4, characterized in that: The data formats of different weight block matrices are the same or different, and the data formats include data type and data precision; Among them, the weight block matrix that is sensitive to the model accuracy can choose high-precision data representation, and the weight block matrix that is insensitive to the model accuracy can choose low-precision data representation.

6. The compression method according to claim 5, characterized in that: In each weight block matrix, the weight groups at each level are divided into rows, columns and blocks, and the weight groups at different levels adopt the same or different division methods; The weight groups at the same level of different weight block matrices adopt the same or different partitioning methods.

7. The compression method according to claim 1, characterized in that: The sparse processing includes structured sparse processing or unstructured sparse processing.

8. A compression device for a neural network model, characterized in that: include: An acquisition module is used to obtain the weight matrix of the neural network model; A hierarchical division module, for dividing the weight matrix into a plurality of hierarchical weight groups based on any one of rows, columns, and blocks, wherein the plurality of hierarchical levels include the zeroth level to the Kth level, K is a positive integer, and each hierarchical weight group includes a plurality of weight elements, wherein when the level is the zeroth level, the weight element is a weight value; when the level is any one of the first level to the Kth level, the weight element is a previous level weight group, and a plurality of Kth level weight groups are obtained by dividing a plurality of K-1th level weight groups according to any one of rows, columns, and blocks; A sparse module is used to perform sparse processing on the weight group of each level in turn according to the level order until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix; Among them, the data length of the zeroth-level weight group is determined based on the bit width of the acceleration unit, the acceleration unit is used to perform operations related to the neural network model, and the number of weight elements in the zeroth-level weight group is determined by the data length of the zeroth-level weight group and the data type of the weight value.

9. The compression device according to claim 8, characterized in that The hierarchical division module is also used to divide the weight matrix into multiple zero-level weight groups according to any one of rows, columns and blocks, and the weight elements are weight values; divide multiple zero-level weight groups into multiple first-level weight groups according to any one of rows, columns and blocks, and the weight elements are zero-level weight groups; and so on, divide the K-1-th level weight group into multiple K-th level weight groups according to any one of rows, columns and blocks, and the weight elements are K-1-th level weight groups, wherein K is a positive integer; wherein weight groups of different levels adopt the same or different division methods.

10. The compression device according to claim 9, characterized in that The sparse module includes: The sparse unit is used to perform sparse processing on the weight group of each level in turn according to the level order to obtain a sparse weight matrix; An acquisition unit, used for acquiring the accuracy and / or compression ratio of the neural network model after the weight group of each layer is sparsely processed; A judging unit, used to judge whether the accuracy of the neural network model after the weight group sparsification processing of the current level is less than a first threshold; The sparse unit is also used to stop the sparse processing of the weight group of the next level if the accuracy of the neural network model after the sparse processing of the weight group of the current level is less than the first threshold, and use the sparse weight matrix obtained after the sparse processing of the weight group of the previous level as the target weight matrix; The judgment unit is further configured to judge whether the compression ratio of the neural network model after the weight group sparse processing of the current level is greater than a second threshold value if the accuracy of the neural network model after the weight group sparse processing of the current level is greater than the first threshold value; The sparse unit is also used to stop the sparse processing of the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is greater than or equal to the second threshold, and use the sparse weight matrix obtained after the sparse processing of the weight group of the current level as the target weight matrix; and continue to sparse the weight group of the next level if the compression ratio of the neural network model after the sparse processing of the weight group of the current level is less than the second threshold; The hierarchical order includes from the zeroth level to the Kth level or from the Kth level to the zeroth level.

11. The compression device according to claim 8, characterized in that Also includes: A block partitioning module, used for partitioning the weight matrix into a plurality of weight block matrices; A level determination module is used to determine the number of levels of division according to the density of valid data in each weight block matrix; The hierarchical division module is further used to divide each weight block matrix into a corresponding number of hierarchical weight groups based on one of rows, columns and blocks according to the number of hierarchies of each weight block; The sparse module is also used to determine the sparsity of each weight block matrix according to the accuracy of the weight value; and performing sparse processing on the multiple hierarchical weight groups divided by each weight block matrix in turn according to the sparsity of each weight block matrix until the accuracy and / or compression ratio of the neural network model reaches a predetermined target, so as to obtain a target weight matrix; Among them, the number of levels is inversely correlated with the density of valid data in the weight block matrix.

12. The compression device according to claim 11, characterized in that The data formats of different weight block matrices are the same or different, and the data formats include data type and data precision; Among them, the weight block matrix that is sensitive to the model accuracy can choose high-precision data representation, and the weight block matrix that is insensitive to the model accuracy can choose low-precision data representation.

13. The compression device according to claim 12, characterized in that In each weight block matrix, the weight groups at each level are divided into rows, columns and blocks, and the weight groups at different levels adopt the same or different division methods; The weight groups at the same level of different weight block matrices adopt the same or different partitioning methods.

14. A computing system, characterized in that: It includes system memory, processor and acceleration unit which are coupled to each other. The system memory includes a program and a neural network model, and the program includes instructions of a compression method of a neural network model according to any one of claims 1 to 7; The processor is used to execute the instruction to compress the neural network model, and write the calculation data of the compressed neural network model into the acceleration unit; And the acceleration unit is used to perform operations related to the neural network model.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images