Neural network compression method, device and equipment and readable storage medium

Abstract

The application discloses a neural network compression method, device and equipment and a readable storage medium. The application obtains a multi-level deep learning neural network to be compressed and a network loss; determines sparse coding and sparse weight corresponding to the multi-level deep learning neural network, so that redundant parameters and weight in the multi-level deep learning neural network can be removed; determines a sparse coding loss and a sparse weight loss; fuses the network loss, the sparse coding loss and the sparse weight loss to obtain a compression loss; and compresses the multi-level deep learning neural network by using the compression loss to obtain a compressed multi-level deep learning neural network. It can be seen that the application can realize double compression of the multi-level deep learning neural network to be compressed, reduce redundant parameters and reduce the calculation performance loss of the neural network. The application performs sparse coding and sparse weight on each level of the multi-level deep learning neural network, so that the strong crosstalk problem in the deep learning neural network can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, more particularly, to a neural network compression method, device and equipment and readable storage medium. BACKGROUND

[0002] With the development of Internet technology, deep learning neural networks have received extensive attention due to their excellent performance in decision-making tasks in complex environments. For example, deep learning neural networks can be applied to medical image segmentation to provide end-to-end image segmentation results to help doctors diagnose diseases more accurately; deep learning neural networks can also be used for vulnerability analysis of smart grids to identify vulnerable links in smart grids; deep learning neural networks can also be used for robot motion control; deep learning neural networks can also be used for resource allocation decisions and task offloading of Internet of Things devices; deep learning neural networks can also be used for autonomous driving technology, etc.

[0003] However, in the prior art, deep learning neural networks generally have high redundancy, which leads to a decline in the control performance of deep learning neural networks and greatly affects the practicality of deep learning neural networks. How to effectively reduce the redundant parameters in deep learning neural networks and reduce the computational performance loss of neural networks is a problem worthy of attention.

[0004] In view of the above, there is an urgent need for a neural network compression method for compressing redundant parameters in deep learning neural networks to reduce the computational performance loss of neural networks. SUMMARY

[0005] Therefore, the present application provides a neural network compression method, device and equipment and readable storage medium for compressing redundant parameters in deep learning neural networks to reduce the computational performance loss of neural networks.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following scheme:

[0007] A neural network compression method comprises:

[0008] obtaining a multi-level deep learning neural network to be compressed and a corresponding network loss thereof;

[0009] determining a sparse coding and a sparse weight corresponding to the multi-level deep learning neural network;

[0010] determining a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weight;

[0011] fusing the network loss, the sparse coding loss and the sparse weight loss to obtain a fused compression loss;

[0012] Compress parameters in an output process of each level of the multi-level deep learning neural network by using the compression loss, to obtain a compressed multi-level deep learning neural network.

[0013] Optionally, determining the sparse code corresponding to the multi-level deep learning neural network comprises:

[0014] Determining an output distribution of each level of the multi-level deep learning neural network and an encoding expectation value corresponding to the sparse code;

[0015] Obtaining a distribution difference value by using the SKL divergence and based on the output distribution and the encoding expectation value;

[0016] Obtaining the sparse code of the multi-level deep learning neural network based on the distribution difference value and the output distribution.

[0017] Optionally, obtaining the distribution difference value by using the SKL divergence and based on the output distribution and the encoding expectation value comprises:

[0018] Determining a distribution difference value calculation formula by using a definition of the SKL divergence;

[0019] Calculating the distribution difference value of each level of the multi-level deep learning neural network based on the output distribution, the encoding expectation value and the distribution difference value calculation formula.

[0020] Optionally, determining the sparse weight corresponding to the multi-level deep learning neural network comprises:

[0021] Obtaining a weight coefficient set corresponding to each level of the multi-level deep learning neural network;

[0022] Determining a weight adjustment parameter corresponding to the sparse weight;

[0023] Reducing each weight coefficient in each weight coefficient set by using a regularization technique and the weight adjustment parameter, to obtain the sparse weight corresponding to the multi-level deep learning neural network.

[0024] Optionally, reducing each weight coefficient in each weight coefficient set by using the regularization technique and the weight adjustment parameter comprises:

[0025] Reducing each weight coefficient in each weight coefficient set by using a logarithmic regularization technique and the weight adjustment parameter.

[0026] Optionally, compressing parameters in an output process of each level of the multi-level deep learning neural network by using the compression loss comprises:

[0027] update each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network based on the compression loss and using a stochastic gradient descent method;

[0028] prune the weight coefficient set corresponding to each level of the multi-level deep learning neural network based on the compression loss and using a proximal operator of a regularizer;

[0029] wherein the update process of each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network and the pruning process of the weight coefficient set corresponding to each level of the multi-level deep learning neural network are alternately iterated.

[0030] Optionally, determining the sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight comprises:

[0031] determining a coding coefficient corresponding to the sparse coding;

[0032] determining a weight parameter corresponding to the sparse weight;

[0033] multiplying the coding coefficient with the sparse coding of each level to obtain a sparse coding loss;

[0034] multiplying the weight parameter with the sparse weight of each level to obtain a sparse weight loss;

[0035] fusing the network loss, the sparse coding loss, and the sparse weight loss to obtain a fused compression loss, comprising:

[0036] summing the network loss, the sparse coding loss, and the sparse weight loss to obtain a compression loss.

[0037] A neural network compression device, comprising:

[0038] a loss acquisition unit configured to acquire a multi-level deep learning neural network to be compressed and a network loss corresponding thereto;

[0039] a sparsity determination unit configured to determine sparse coding and sparse weight corresponding to the multi-level deep learning neural network;

[0040] a loss determination unit configured to determine a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weight;

[0041] a loss fusion unit configured to fuse the network loss, the sparse coding loss, and the sparse weight loss to obtain a fused compression loss;

[0042] a network compression unit configured to compress parameters in a process of outputting each level of the multi-level deep learning neural network using the compression loss, to obtain a compressed multi-level deep learning neural network.

[0043] A neural network compression device, comprising a memory and a processor;

[0044] The memory is configured to store a program.

[0045] The processor is configured to execute the program to implement each step of the neural network compression method.

[0046] A readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement each step of the neural network compression method.

[0047] As can be seen from the above technical solutions, the neural network compression method provided by the present application can obtain a multi-level deep learning neural network to be compressed and a corresponding network loss. Then, sparse coding and sparse weights corresponding to the multi-level deep learning neural network can be determined. In this way, the sparse coding and the sparse weights can be used to remove redundant parameters and redundant weights in the multi-level deep learning neural network. Next, a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weights can be determined. Then, the network loss, the sparse coding loss, and the sparse weight loss can be fused to obtain a fused compression loss. Finally, the compression loss can be used to compress parameters in a process of outputting each level of the multi-level deep learning neural network, to obtain a compressed multi-level deep learning neural network. Even if the multi-level deep learning neural network is compressed, the analysis function of the multi-level deep learning neural network can still be realized. It can be seen that the present application can remove redundant parameters in the multi-level deep learning neural network and remove redundant weights in the multi-level deep learning neural network, that is, the present application can double compress the multi-level deep learning neural network to be compressed, reduce unnecessary parameters in the multi-level deep learning neural network, and reduce the computational performance loss of the neural network.

[0048] In addition, the application performs sparse coding and sparse weight on each level of the multi-level deep learning neural network, and the compressed multi-level deep learning neural network of the application performs sparse coding and sparse weight on each level, and double sparse is performed, which has great sparsity. On this basis, the compressed multi-level deep learning neural network of the application can effectively avoid strong crosstalk in the multi-level deep learning neural network, effectively reduce the overlap between non-local inputs, and improve the orthogonality, which improves the discrimination of the multi-level deep learning neural network for different inputs. Therefore, the compressed multi-level deep learning neural network of the application can effectively capture important attributes of states or instances in deep learning and simplify the analysis process of the multi-level deep learning neural network, thereby improving the analysis performance of the multi-level deep learning neural network. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0050] Figure 1 A flow chart of a neural network compression method disclosed by the embodiments of the application;

[0051] Figure 2 A schematic diagram of a neural network compression result disclosed by the embodiments of the application;

[0052] Figures 3a-3c A schematic diagram of different regularizers reducing the speed of each weight coefficient disclosed by the embodiments of the application;

[0053] Figure 4 A structural block diagram of a neural network compression device disclosed by the embodiments of the application;

[0054] Figure 5 A hardware structural block diagram of a neural network compression device disclosed by the embodiments of the application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0056] The neural network compression method, device and equipment and readable storage medium provided by the application can be applied in multiple fields, such as natural language field, computer vision field and audio processing field.

[0057] Next, the neural network compression method of the application is described in detail, including the following steps: Figure 1 The neural network compression method of the application is described in detail, including the following steps:

[0058] Step S1, obtaining a multi-level deep learning neural network to be compressed and a corresponding network loss thereof.

[0059] The multi-level deep learning neural network can be a multi-level deep learning neural network with an analysis function, for example, the multi-level deep learning neural network can be a neural network applied to medical image segmentation, which analyzes the best segmentation method of a medical image and provides an end-to-end image segmentation result for a doctor to make a more accurate diagnosis; for another example, the multi-level deep learning neural network can also be a neural network for analyzing the vulnerability of a smart grid according to the network topology of the smart grid, which can check the vulnerable link of the smart grid; for another example, the multi-level deep learning neural network can be a neural network for controlling the action of a robot according to coordinate information; for another example, the multi-level deep learning neural network can be a neural network for realizing resource allocation decision and task offloading of an Internet of Things device through edge computing; for another example, the multi-level deep learning neural network can be a neural network for realizing automatic driving technology according to regional road conditions.

[0060] Specifically, a multi-level deep learning neural network to be compressed can be obtained. First, the state and reward sequence in the multi-level deep learning neural network to be compressed can be represented as:

[0061]

[0062] wherein the state is represented by S and the reward is represented by r, i represents a certain moment, and t represents the number of sample data sets. S i and S i+1 represent the input of the multi-level deep learning neural network at i and i+1 moments respectively, n (0) in the above formula is a number, which is the dimension of the input vector of the multi-level deep learning neural network, and can also be regarded as the number of neurons of the 0th layer of the multi-level deep learning neural network, R is a real number set, and based on this, is a set of real number vectors with a dimension of n (0) .

[0063] Secondly, a state-action value function Q(S i , a i), the state action value function can be calculated through the state S i the final value of performing action a t in the state.

[0064] Again, the parameter set of the multi-level deep learning neural network can be set as:

[0065]

[0066] wherein, W (1) represents the weight coefficient set of the first layer of the multi-level deep learning neural network, b (1) represents the bias of the first layer of the multi-level deep learning neural network. W (L) represents the weight coefficient set of the Lth layer of the multi-level deep learning neural network, and L represents the last layer of the multi-level deep learning neural network. b (L) represents the bias of the Lth layer of the multi-level deep learning neural network, and L can be any number other than one, for example, it can be 5 or 30. Thus, the parameter set of the multi-level deep learning neural network is composed of the weight coefficient set of each layer of the multi-level deep learning neural network and the bias of each layer. n (L) and n (L-1) represent the number of neurons of the Lth and L-1th layers of the multi-level deep learning neural network, represents a set of real number vectors with dimension n (L) . represents a set of real number matrices with dimension n (L) rows and n (L-1) columns.

[0067] When the input of the multi-level deep learning neural network is the state S, the output of the lth layer of the multi-level deep learning neural network can be denoted as wherein, l represents any layer in the multi-level deep learning neural network with total layers L.

[0068] The weight coefficient set of the last layer of the multi-level deep learning neural network, i.e., the output layer, can be denoted as W a From the above description, it can be seen that corresponds to different actions in the multi-level deep learning neural network (DQN). The output layer uses linear activation, and the output of the multi-level deep learning neural network is the estimated value of the action value:

[0069]

[0070] wherein, a represents the total action of the multi-level deep learning neural network, Q π (S i , a) represents the accurate value of the state value of the multi-level deep learning neural network. represents an estimated value using a multi-level deep learning neural network to estimate state value. represents the transpose of the output of the Lth layer of the multi-level deep learning neural network when the input is S represents another expression of the estimated value, indicating that the estimated value is the product of the transpose of the output of the Lth layer of the multi-level deep learning neural network and the weight W a of the last layer.

[0071] Specifically, a sample set labeled with a label can be input into the multi-level deep learning neural network to be compressed to obtain the output of each level of the multi-level deep learning neural network to be compressed. Subsequently, based on the output of each level and the label of the sample set corresponding to each level, the overall loss of the multi-level deep learning neural network to be compressed can be calculated as the network loss.

[0072] In order to obtain an accurate action value estimation in the multi-level deep learning neural network, a mean square temporal difference error can be used to define the network loss F, which takes into account the reward difference between adjacent states in the time scale, and is defined as follows:

[0073]

[0074] wherein r t+1 represents the reward at time t+1, γ represents a reward decay factor, i.e., a discount rate, and γ t+1 represents the discount rate at time t+1. γ t+1 = 0, S t represents a terminal state, and L represents the last layer of the multi-level deep learning neural network.

[0075] represents the output of the Lth layer of the multi-level deep learning neural network when the input is S t . represents the transpose of the output of the Lth layer of the multi-level deep learning neural network when the input is S t . represents the transpose of the output of the Lth layer of the multi-level deep learning neural network when the input is S t+1 .

[0076] Step S2, determining the sparse coding and sparse weight corresponding to the multi-level deep learning neural network.

[0077] Specifically, the outputs of all hidden layers in the multi-level deep learning neural network, i.e., the distribution constraints of each level of representation in the multi-level deep learning neural network, can be determined as a sparse distribution set G, so as to determine the sparse coding. Further, the sparsity of the sparse distribution set G is controlled by a set parameter β, so as to control the sparsity of the sparse coding. Thus, the sparse coding corresponding to the multi-level deep learning neural network can be determined by determining the definition of the sparse distribution set G.

[0078] In the present application, by determining the sparse coding corresponding to the multi-level deep learning neural network to be compressed, the output of each level in the multi-level deep learning neural network is constrained, so as to make the output distribution of each level in the multi-level deep learning neural network sparse, i.e., the output of each level in the multi-level deep learning neural network tends to be a sparse vector, and the redundant parameters in the multi-level deep learning neural network are removed. The output distribution refers to that the output of each level in the multi-level deep learning neural network can be regarded as a random variable, and the random variable conforms to a certain distribution rule, and the distribution rule contains the data range of the random variable.

[0079] Specifically, the sparse weight corresponding to the multi-level deep learning neural network can be determined in various ways. For example, the sparse weight can be determined by setting a threshold and training the multi-level deep learning neural network, or the sparse weight can be determined by pruning the weight coefficient set of the multi-level deep learning neural network.

[0080] The weight coefficient set of each level in the multi-level deep learning neural network can be regarded as a matrix. By making the weight coefficient set of each level in the multi-level deep learning neural network tend to be a sparse matrix, the matrix can be made sparse, so as to determine the sparse weight corresponding to the multi-level deep learning neural network to be compressed, and to remove the redundant weight of the multi-level deep learning neural network.

[0081] Referring to Figure 2 , Figure 2 The double sparsity in the above formula refers to the sparse weight and the sparse coding. t The input of the multi-level deep learning neural network is denoted as S. t The output of the multi-level deep learning neural network is denoted as Q(S). Figure 2 The circle in the above formula indicates the value of the output vector of the level in a certain dimension, and the dashed circle indicates that the value of the dimension is zero. Before compression, the output vector of each layer has a value in each dimension, i.e., each layer does not contain a dashed circle. After compression by the sparse coding, the value of the output vector of each layer in some dimensions is zero, i.e., a dashed circle appears in each layer as shown in Figure 2

[0082] ​Figure 2 W (1) , W (2) and W (3) are respectively a weight coefficient set of the first level deep learning neural network, a weight coefficient set of the second level deep learning neural network and a weight coefficient set of the third level deep learning neural network. The weight coefficient set of each level is a coefficient matrix, and in Figure 2 , three coefficient matrices of the first level, the second level and the third level are represented by three large boxes, and each small box in the three large boxes represents each coefficient in the coefficient matrix. The line segment connecting the input and the first level, the line segment connecting the first level and the second level and the line segment connecting the second level and the third level also represent the coefficients in the respective coefficient matrix. The small boxes in the large box corresponding to each level and the line segment between each level correspond one by one, that is, the white small box in the large box indicates that the coefficient is zero, and the line segment between each level corresponding to the coefficient is a dashed line. When not compressed, the line segment between each level is a solid line, so the coefficient matrix corresponding to each level is more complex, and after processing the weight coefficient set of each level, a sparse weight can be obtained, as shown in Figure 2 , the large box representing the coefficient matrix becomes sparse. Based on this, the calculation process of the multi-level deep learning neural network becomes simple.

[0083] Step S3, determine the sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight.

[0084] Specifically, the loss caused by adding sparse coding to each level of the multi-level deep learning neural network can be determined as the sparse coding loss. The loss caused by adding sparse weight to each level of the multi-level deep learning neural network can be determined as the sparse weight loss.

[0085] Step S4, fuse the network loss, the sparse coding loss and the sparse weight loss to obtain a fused compression loss.

[0086] Specifically, the network loss F, the sparse coding loss and the sparse weight loss can be added, and the result after addition can be taken as the compression loss. According to the importance of the network loss F, the sparse coding and the sparse weight in the actual training process, the performance weight of the network loss F, the performance weight of the sparse coding and the performance weight of the sparse weight can be determined. Then, the compression loss is obtained by weighting according to the performance weight of the network loss F, the performance weight of the sparse coding and the performance weight of the sparse weight.

[0087] Step S5, compress the parameters in the output process of each level of the multi-level deep learning neural network by using the compression loss to obtain a compressed multi-level deep learning neural network.

[0088] Specifically, the parameters of each level of the multi-level deep learning neural network can be modified according to the compression loss, and a compressed multi-level deep learning neural network is obtained.

[0089] As can be seen from the technical solutions described above, the neural network compression method provided in the present application can obtain a multi-level deep learning neural network to be compressed and a corresponding network loss. Then, the sparse coding and sparse weight corresponding to the multi-level deep learning neural network can be determined. In this way, the redundant parameters and the redundant weight in the multi-level deep learning neural network can be removed through the sparse coding and the sparse weight. Next, the sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight can be determined. Then, the network loss, the sparse coding loss and the sparse weight loss can be fused to obtain a fused compression loss. Finally, the parameters in the output process of each level of the multi-level deep learning neural network can be compressed by using the compression loss, and a compressed multi-level deep learning neural network is obtained, so that the analysis function of the multi-level deep learning neural network can be realized even if the multi-level deep learning neural network is compressed. It can be seen that the present application can remove the redundant parameters in the multi-level deep learning neural network and remove the redundant weight in the multi-level deep learning neural network, that is, the multi-level deep learning neural network to be compressed is compressed twice, so that unnecessary parameters in the multi-level deep learning neural network are reduced, and the calculation performance loss of the neural network is reduced.

[0090] In addition, the multi-level deep learning neural network is subjected to sparse coding and sparse weight, and the compressed multi-level deep learning neural network is also subjected to sparse coding and sparse weight, so that double sparseness is achieved and great sparseness is obtained. On this basis, the compressed multi-level deep learning neural network can effectively avoid strong crosstalk in the multi-level deep learning neural network, effectively reduce the overlap between non-local inputs, and improve the orthogonality, so that the discrimination of the multi-level deep learning neural network for different inputs is improved. Therefore, the compressed multi-level deep learning neural network can effectively capture important attributes of states or instances in deep learning, and simplify the analysis process of the multi-level deep learning neural network, thereby improving the analysis performance of the multi-level deep learning neural network.

[0091] Further, in some embodiments of the present application, the process of determining the sparse coding corresponding to the multi-level deep learning neural network in step S2 is described in detail, and the specific steps are as follows:

[0092] S10, determining the output distribution of each level of the multi-level deep learning neural network and the coding expectation value corresponding to the sparse coding.

[0093] Specifically, the output distribution of each level in the multi-level deep learning neural network can be determined by determining the output distribution function corresponding to each level in the multi-level deep learning neural network. The output distribution function can be denoted as G, and the sparsity of G can be controlled by a set parameter β. The parameter β is a parameter value for adjusting the encoding expectation value.

[0094] S11, obtaining a distribution difference value based on the output distribution and the encoding expectation value by using the SKL divergence.

[0095] Specifically, the target output distribution of each level in the compressed multi-level deep learning neural network can be determined as the encoding expectation value. Then, the distribution difference value of each level in the multi-level deep learning neural network before and after compression can be determined by using the SKL divergence.

[0096] S12, obtaining the sparse coding of the multi-level deep learning neural network based on the distribution difference value and the output distribution.

[0097] Specifically, the output distribution can be obtained by inputting the sample set labeled with labels into the multi-level deep learning neural network.

[0098] The sparse coding of the multi-level deep learning neural network can be obtained by calculating the distribution difference value and the output distribution.

[0099] As can be seen from the above technical solution, compared with the previous embodiment, the present embodiment provides an optional way to determine the sparse coding of each level in the multi-level deep learning neural network, which is to determine the sparse coding of each level by using the SKL divergence. As can be seen, the sparse coding of each level can be well determined by the above steps, thereby realizing the compression of the multi-level deep learning neural network.

[0100] Further, in some embodiments of the present application, the process of S11, obtaining a distribution difference value based on the output distribution and the encoding expectation value by using the SKL divergence, is described in detail, and the specific steps are as follows:

[0101] S110, determining a distribution difference value calculation formula by using the definition of the SKL divergence.

[0102] p η and p B are the output distribution of a certain level in the multi-level deep learning neural network and the target output distribution of the level, respectively, p η and p B are distribution sets, and the value of η can be determined according to the output mean of the level. For p η and p B , the SKL divergence between them, i.e. the distribution difference value, can be defined as D SKL (pη ||p B ), the SKL divergence expression is as follows:

[0103]

[0104] where, B = [η1, η2], p B = {p b : b ∈ B}. η1 and η2 are natural parameters describing the properties of p B , which are used to determine the fitting target of the output distribution, and vary with different p B . B refers to the real number interval. p B = {p b : b ∈ B} indicates that each element p B in p b is an exponential distribution, and the mean value b of the exponential distribution is located in the interval B. P and Q are two different probability distribution functions used to explain the calculation formula of the KL divergence. p is any element in the target output distribution.

[0105] And for the exponential family, the KL divergence has a generalized form of Bregman divergence:

[0106]

[0107] where p i is the mean value of the p distribution, and q i is the mean value of the q distribution.

[0108] Based on this, the non-negative and convex form of the SKL divergence in formula (3) can be rewritten as follows:

[0109]

[0110] where β = -η -1 is the parameter value for adjusting the expected value of the code. And is the output distribution corresponding to each level in the multi-level deep learning neural network, which is an exponential distribution. B = (0, β] is the expected value of the sparse code corresponding to each level in the multi-level deep learning neural network. is the distribution difference.

[0111] Based on this, β can determine the properties of an exponential distribution. When the value of β is small, the exponential distribution will randomly output more 0 values, that is, more 0 values will appear in the output of each level. Therefore, an exponential distribution B with a small β is used as the target output distribution of a certain level of the multi-level deep learning neural network. In the training process of the multi-level deep learning neural network, the output distribution of the level is made to approach the exponential distribution B using the SKL regularization method. After the training is completed, the output distribution of the level is considered to conform to the exponential distribution B, and the output of the level can be a sparse vector, obtaining the sparse coding of the level.

[0112] S111, based on the output distribution, the encoding expectation value and the distribution difference calculation formula, the distribution difference of each level of the multi-level deep learning neural network is calculated.

[0113] Specifically, the output distribution of the lth level of the multi-level deep learning neural network can be set as Wherein, the lth level can be any level of the multi-level deep learning neural network. Based on the above determined distribution difference, the distribution function G of the multi-level deep learning neural network, that is, the expression of the sparse coding, is as follows:

[0114]

[0115] Wherein, is the output distribution corresponding to the lth level in the multi-level deep learning neural network, and the calculation formula is as follows:

[0116]

[0117] It has been proved in the above that the output distribution of a certain level in the multi-level deep learning neural network can approach the exponential distribution B through the distribution difference, and after the training is completed, the output of the level can be a sparse vector, obtaining the sparse coding of the level. Through formula (5), the output distribution of each level in the multi-level deep learning neural network can approach the corresponding exponential distribution B, and after the training is completed, the outputs of the multi-levels can all be sparse vectors, obtaining the sparse coding of the multi-levels.

[0118] It should be noted that the exponential distribution B corresponding to each level can be the same or different.

[0119] As can be seen from the above technical solution, compared with the previous embodiment, the present embodiment provides an optional way of determining the sparse coding of each level of the multi-level deep learning neural network using the SKL divergence, the output distribution and the encoding expectation value. The specific way is to determine the sparse coding of each level using the definition of SKL divergence, the output distribution and the encoding expectation value. It can be seen that the above steps can well realize the compression of the multi-level deep learning neural network.

[0120] Further, in some embodiments of the present application, the process of determining the sparse weights corresponding to the multi-level deep learning neural network in step S2 is described in detail, and the specific steps are as follows:

[0121] S20, obtaining the weight coefficient set corresponding to each level of the multi-level deep learning neural network.

[0122] Specifically, each level in the multi-level deep learning neural network has a corresponding weight coefficient set, and the weight coefficient set corresponding to each level is extracted, which is a coefficient matrix.

[0123] S21, determining the weight adjustment parameter corresponding to the sparse weight.

[0124] Specifically, the weight adjustment parameter is a parameter for adjusting the sparsity of the sparse weight, that is, the size of the weight adjustment parameter determines the overall sparsity of the sparse weight.

[0125] S22, using a regularization technique and the weight adjustment parameter to reduce each weight coefficient in each weight coefficient set to obtain the sparse weight corresponding to the multi-level deep learning neural network.

[0126] Specifically, there are various regularization techniques, and three optional ones are provided here, such as L1 norm regularization technique, L2 norm regularization technique, and logarithmic regularization technique.

[0127] According to the definition of the selected regularization technique, a penalty term can be formulated to reduce each weight coefficient in each weight coefficient set to obtain the sparse weight corresponding to the multi-level deep learning neural network.

[0128] Referring to Figure 2 , Figures 3a-3c Three kinds of regularization techniques for reducing weight sparsity are recorded in

[0129] As shown in Figures 3a-3c , the L1 norm regularization technique reduces the values of each weight coefficient in the weight coefficient set at the same division speed; as shown in Figure 3a , the L2 norm regularization technique reduces the values of each weight coefficient in the weight coefficient set at the same combination speed; and as shown in Figure 3b , the logarithmic regularization technique shrinks the weight coefficients smaller than the threshold value faster than the L1 norm regularization technique, and shrinks the weight coefficients larger than the threshold value slower.

[0130] From the above technical solution, the embodiment provides an optional way of determining sparse weights corresponding to a multi-level deep learning neural network, specifically, using a regularization technique to reduce each weight coefficient, thereby realizing sparse weights. It can be seen that the embodiment can better determine sparse weights and realize compression of a multi-level deep learning neural network.

[0131] Further, in some embodiments of the present application, the process of reducing each weight coefficient in each weight coefficient set using the regularization technique and the weight adjustment parameter in step S22 is described in detail as follows:

[0132] S220, using a logarithmic regularization technique and the weight adjustment parameter, each weight coefficient in each weight coefficient set is reduced.

[0133] Specifically, the logarithmic regularization technique has a faster shrinkage speed for weight coefficients less than the threshold value, and a slower shrinkage speed for weight coefficients greater than the threshold value. The weight coefficients less than the threshold value can be considered as weight coefficients with small values, and the weight coefficients greater than the threshold value can be considered as weight coefficients with large values. Thus, using the logarithmic regularization technique can reduce the weight coefficients with small values as much as possible, and retain the weight coefficients with large values as much as possible. The weight coefficients with small values can be considered as redundant weights, and thus the purpose of removing redundant weights can be achieved.

[0134] After obtaining each weight coefficient set and the weight adjustment parameter, a penalty term formula can be set to adjust the parameters of the multi-level deep learning neural network to obtain the reduced weight coefficient set, thereby realizing the determination of sparse weights. The penalty term formula is as follows:

[0135]

[0136] Where δ is the weight adjustment parameter, used to adjust the sparsity of the sparse weights, i.e., the degree of reduction of each weight coefficient. As δ decreases, the sparsity of the sparse weights increases. l is any layer of the multi-level deep learning neural network, W (l) is the weight coefficient set of any layer of the multi-level deep learning neural network. is the element in the kth row and jth column of W (l) (l) has n (l) columns and n (l-1) rows.

[0137] ​From the above technical solution, compared with the above embodiment, the present embodiment provides an optional way of determining sparse weights by using log regularization technology. By using the log regularization technology, the weight coefficients in each weight coefficient set can be reduced, and during the reduction process, the reduction degree of the weight coefficients less than the threshold is greater, and the reduction degree of the weight coefficients greater than the threshold is smaller, which can remove redundant weight coefficients, obtain more sparse weight coefficient sets, and better determine the sparse weights of the multi-level deep learning neural network.

[0138] Further, in some embodiments of the present application, the process of step S3, determining the sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight, is described in detail as follows:

[0139] S30, determining the coding coefficient corresponding to the sparse coding.

[0140] Specifically, after the target output distribution is determined, the coding coefficient controls the sparse adjustment range of the sparse coding on the basis of the target output distribution. The larger the coding coefficient, the larger the sparse adjustment range of the sparse coding, and the greater the sparsity of the sparse coding corresponding to the multi-level deep learning neural network.

[0141] S31, determining the weight parameter corresponding to the sparse weight.

[0142] Specifically, after the penalty term is determined, the weight parameter controls the sparse adjustment range of the sparse weight on the basis of pruning the weight coefficient set of the multi-level deep learning neural network by using the penalty term. The larger the weight parameter, the more the weight coefficients of the multi-level deep learning neural network are pruned, and the greater the sparsity of the sparse weight corresponding to the multi-level deep learning neural network.

[0143] S32, multiplying the coding coefficient by the sparse coding of each level to obtain the sparse coding loss.

[0144] Specifically, the coding coefficient can be represented by λ, λ is multiplied by the sparse coding of each level of the multi-level deep learning neural network, and the results of each level after multiplication are added, and the result after addition is the sparse coding loss.

[0145] S33, multiplying the weight parameter by the sparse weight of each level to obtain the sparse weight loss.

[0146] Specifically, the weight parameter can be represented by μ, μ is multiplied by the sparse weight of each level of the multi-level deep learning neural network, and the results of each level after multiplication are added, and the result after addition is the sparse weight loss.

[0147] On this basis, the process of step S4, fusing the network loss, the sparse coding loss and the sparse weight loss to obtain the fused compression loss, is described in detail, and the steps are as follows:

[0148] S34, summing the network loss, the sparse coding loss and the sparse weight loss to obtain the compression loss.

[0149] Specifically, from the expression of the network loss F, the expression of the sparse coding and the sparse weight penalty term of each level of the multi-level deep learning neural network in the above, the expression of the compression loss can be obtained as follows:

[0150]

[0151] It can be seen that the compression loss is composed of the network loss in the entire time period, the sum of the sparse coding loss of each level in the multi-level deep learning neural network, and the sum of the sparse weight loss of each level.

[0152] From the above technical solution, it can be seen that the embodiment provides an optional way of fusing the compression loss. Through the above steps, the network loss, the sparse coding loss and the sparse weight loss can be well fused, so as to compress the multi-level deep learning neural network.

[0153] In some embodiments of the present application, the process of step S4 of compressing the parameters in the output process of each level of the multi-level deep learning neural network using the compression loss is described in detail. The updating process of each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network in step S40 and the pruning process of the weight coefficient set corresponding to each level of the multi-level deep learning neural network in step S41 are alternately iterated. The specific processes of step S40 and step S41 are as follows:

[0154] S40, based on the compression loss, and using the stochastic gradient descent method, updating each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network.

[0155] Specifically, after obtaining the output of all hidden layers in the multi-level deep learning neural network and the final output result of the multi-level deep learning neural network by using the forward propagation of the labeled sample set, the compression loss can be obtained. Then, the back propagation stochastic gradient descent method and the updating of each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network can be used.

[0156] The partial derivative of the network loss in the compression loss is as follows:

[0157]

[0158] The gradient of the hidden layer neuron is composed of two parts, one part is from the network loss F, which only produces a gradient in the L hidden layer neurons of the L deep learning neural network, and the calculation expression of the gradient is as follows:

[0159]

[0160] The other part of the gradient of the hidden layer neuron comes from the sparse coding loss in the compression loss, and the calculation expression of the gradient is as follows:

[0161]

[0162] Wherein, means taking the sign of , if then

[0163] After obtaining the gradient of the hidden layer neuron through expressions (9) and (10), each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network can be updated by using the stochastic gradient descent method.

[0164] S41, based on the compression loss, and using the proximal operator of the regularizer, pruning the weight coefficient set corresponding to each level of the multi-level deep learning neural network.

[0165] Specifically, only focusing on the weight coefficient set of the multi-level deep learning neural network, the compression loss expression is as follows:

[0166]

[0167] Wherein, W is each weight coefficient set of the pruned multi-level deep learning neural network, and U is each weight coefficient set in the parameter set of each level of the multi-level deep learning neural network updated by using the stochastic gradient descent method.

[0168] Based on this, all weight coefficient sets of the pruned multi-level deep learning neural network can be obtained by using the proximal operator of the regularizer, and the calculation method is as follows:

[0169]

[0170] Wherein, u = u kj is the element in the kth row and jth column of U. is the element in the kth row and jth column of W (l) The proximal operator Prox logAn atomic closed-form solution for the logarithmic regularizer is presented, which is highly efficient. It reduces weights to varying degrees depending on the weight coefficient, keeping weights with values ​​greater than a threshold almost unchanged (i.e., large weights remain largely constant) while minimizing weights with values ​​less than a threshold (i.e., small weights). Based on this, sparse weights with higher sparsity can be induced. Furthermore, μ is key to sparse weights, controlling the rate at which weight coefficients decrease in each iteration.

[0171] The neural network compression apparatus provided in this application is described below. The neural network compression apparatus described below can be referred to in correspondence with the neural network compression method described above.

[0172] First, combined Figure 3c This section introduces neural network compression devices, such as... Figure 4 As shown, the neural network compression device may include:

[0173] Network loss acquisition unit 1 is used to acquire the multi-level deep learning neural network to be compressed and its corresponding network loss;

[0174] Sparse determination unit 2 is used to determine the sparse encoding and sparse weights corresponding to the multi-level deep learning neural network.

[0175] Sparse loss determination unit 3 is used to determine the sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight.

[0176] Loss fusion unit 4 is used to fuse the network loss, the sparse coding loss and the sparse weight loss to obtain the fused compression loss;

[0177] The network compression unit 5 is used to compress the parameters in the output process of each layer of the multi-level deep learning neural network using the compression loss, so as to obtain a compressed multi-level deep learning neural network.

[0178] Furthermore, the sparse determination unit 2 may include:

[0179] The encoding determination unit is used to determine the output distribution of each layer in the multi-level deep learning neural network and the expected encoding value corresponding to the sparse encoding.

[0180] The divergence utilization unit is used to utilize SKL divergence and obtain the distribution difference based on the output distribution and the expected value of the encoding.

[0181] The encoding unit is used to obtain the sparse encoding of the multi-level deep learning neural network based on the distribution difference and the output distribution.

[0182] Further, the divergence utilizing unit can comprise:

[0183] A calculation formula determining unit is configured to determine a distribution difference value calculation formula by using a definition of the SKL divergence;

[0184] A distribution calculating unit is configured to calculate a distribution difference value of each level of the multi-level deep learning neural network based on the output distribution, the encoding expectation value and the distribution difference value calculation formula.

[0185] Further, the sparsity determining unit 2 can comprise:

[0186] A set obtaining unit is configured to obtain a weight coefficient set corresponding to each level of the multi-level deep learning neural network;

[0187] A parameter determining unit is configured to determine a weight adjustment parameter corresponding to the sparse weight;

[0188] A weight obtaining unit is configured to reduce each weight coefficient in each weight coefficient set by using a regularization technique and the weight adjustment parameter, to obtain a sparse weight corresponding to the multi-level deep learning neural network.

[0189] Further, the weight obtaining unit can comprise:

[0190] A coefficient reducing unit is configured to reduce each weight coefficient in each weight coefficient set by using a logarithmic regularization technique and the weight adjustment parameter.

[0191] Further, the sparse loss determining unit 3 can comprise:

[0192] A first determining unit is configured to determine an encoding coefficient corresponding to the sparse encoding;

[0193] A second determining unit is configured to determine a weight parameter corresponding to the sparse weight;

[0194] A third determining unit is configured to multiply the encoding coefficient and the sparse encoding of each level to obtain a sparse encoding loss;

[0195] A fourth determining unit is configured to multiply the weight parameter and the sparse weight of each level to obtain a sparse weight loss;

[0196] On this basis, the loss fusing unit 4 can comprise:

[0197] A first loss fusing unit is configured to sum the network loss, the sparse encoding loss and the sparse weight loss to obtain a compression loss.

[0198] Further, the network compression unit 5 can comprise:

[0199] a parameter updating unit, configured to update each weight coefficient set and bias set in a parameter set of each level of the multi-level deep learning neural network based on the compression loss and by using a stochastic gradient descent method;

[0200] a weight updating unit, configured to prune the weight coefficient set corresponding to each level of the multi-level deep learning neural network based on the compression loss and by using a proximal operator of a regularizer;

[0201] wherein the updating process of each weight coefficient set and bias set in the parameter set of each level of the multi-level deep learning neural network and the pruning process of the weight coefficient set corresponding to each level of the multi-level deep learning neural network are iterated alternately.

[0202] The neural network compression device provided by the embodiments of the present application can be applied to a neural network compression device, such as a mobile phone, an IPAD, a PC terminal, etc. Optionally, Figure 4 A hardware structure block diagram of the neural network compression device is shown, and the hardware structure of the neural network compression device can include at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4, as shown in Figure 5 Figure 5

[0203] In the embodiments of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete communication with each other through the communication bus 4;

[0204] The processor 1 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0205] The memory 3 can include a high-speed RAM memory, and can also include a non-volatile memory, etc., such as at least one disk memory.

[0206] The memory stores a program, and the processor can invoke the program stored in the memory, and the program is used to:

[0207] obtain a multi-level deep learning neural network to be compressed and a network loss corresponding to the multi-level deep learning neural network;

[0208] determine a sparse code and a sparse weight corresponding to the multi-level deep learning neural network;

[0209] determine a sparse code loss corresponding to the sparse code and a sparse weight loss corresponding to the sparse weight; ​

[0210] The network loss, the sparse coding loss and the sparse weight loss are fused to obtain a fused compression loss.

[0211] The parameters in the output process of each level of the multi-level deep learning neural network are compressed by using the compression loss to obtain a compressed multi-level deep learning neural network.

[0212] Optionally, the refinement function and the expansion function of the program can refer to the description above.

[0213] The application further provides a storage medium which can store a program suitable for processor execution, and the program is used for:

[0214] Obtaining a multi-level deep learning neural network to be compressed and a corresponding network loss thereof;

[0215] Determining sparse coding and sparse weight corresponding to the multi-level deep learning neural network;

[0216] Determining a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weight;

[0217] The network loss, the sparse coding loss and the sparse weight loss are fused to obtain a fused compression loss.

[0218] The parameters in the output process of each level of the multi-level deep learning neural network are compressed by using the compression loss to obtain a compressed multi-level deep learning neural network.

[0219] Finally, it should be noted that in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0220] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment can be referred to each other.

[0221] The above description of disclosed embodiments enables one of ordinary skill in the art to make and use various embodiments of the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Various embodiments of the present application can be combined with each other. Accordingly, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A neural network compression method, characterized in that, The method comprises the following steps: inputting a sample set labeled with a label into a multi-level deep learning neural network to be compressed to obtain an output of each level of the multi-level deep learning neural network to be compressed; calculating an overall loss of the multi-level deep learning neural network to be compressed based on the output of each level and the label of the sample set corresponding to each level, as a network loss, wherein the multi-level deep learning neural network is a medical image analysis network; determining sparse coding and sparse weights corresponding to the multi-level deep learning neural network; determining a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weights; fusing the network loss, the sparse coding loss and the sparse weight loss to obtain a fused compression loss; compressing parameters in an output process of each level of the multi-level deep learning neural network by using the compression loss to obtain a compressed multi-level deep learning neural network; an expression of the compression loss is as follows: ； wherein is the compression loss; is a parameter set of a multi-level deep learning neural network; is an output of the Lth layer of the multi-level deep learning neural network, L representing a last layer of the multi-level deep learning neural network; is a weight coefficient set of the output layer; r i+1 represents a reward at i+1 time; represents a discount rate at t+1 time; F is a network loss; is a weight adjustment parameter for adjusting the sparsity of the sparse weights; denotes the encoding coefficients; G is a sparse distribution set; represents a sparse weight penalty term; is a weight coefficient set of the Lth layer of a multi-level deep learning neural network; is a weight coefficient set of the Lth layer of a multi-level deep learning neural network; represents an output of the Lth layer of a multi-level deep learning neural network when the input is s i represents an output of the Lth layer of a multi-level deep learning neural network when the input is s represents a weight parameter; represents an output of the Lth layer of a multi-level deep learning neural network when the input is s represents a transpose of an output of the Lth layer of a multi-level deep learning neural network when the input is s represents a transpose of an output of the Lth layer of a multi-level deep learning neural network when the input is s represents a transpose of an output of the Lth layer of a multi-level deep learning neural network when the input is s is an output distribution corresponding to the Lth layer in a multi-level deep learning neural network, and the output distribution is an exponential distribution; is a parameter value for adjusting an encoding expectation value; is a distribution difference value; is an element in the kth row and jth column in has a total of columns, rows.​​ 2. The method of claim 1, wherein, determining sparse coding corresponding to the multi-level deep learning neural network comprises the following steps: determining an output distribution of each level of the multi-level deep learning neural network and an encoding expectation value corresponding to the sparse coding; obtaining a distribution difference value by using the SKL divergence and based on the output distribution and the encoding expectation value; obtaining the sparse coding of the multi-level deep learning neural network based on the distribution difference value and the output distribution.

3. The method of claim 2, wherein, obtaining a distribution difference value by using the SKL divergence and based on the output distribution and the encoding expectation value comprises the following steps: determining a distribution difference value calculation formula by using the definition of the SKL divergence; calculating a distribution difference value of each level of the multi-level deep learning neural network based on the output distribution, the encoding expectation value and the distribution difference value calculation formula.

4. The method of claim 1, wherein, determining sparse weights corresponding to the multi-level deep learning neural network comprises the following steps: obtaining a weight coefficient set corresponding to each level of the multi-level deep learning neural network; determining a weight adjustment parameter corresponding to the sparse weights; reducing each weight coefficient in each weight coefficient set by using a regularization technique and the weight adjustment parameter to obtain the sparse weights corresponding to the multi-level deep learning neural network.

5. The method of claim 4, wherein, reducing each weight coefficient in each weight coefficient set by using a regularization technique and the weight adjustment parameter comprises the following steps: reducing each weight coefficient in each weight coefficient set by using a logarithmic regularization technique and the weight adjustment parameter.

6. The method of claim 4, wherein, compressing parameters in an output process of each level of the multi-level deep learning neural network by using the compression loss comprises the following steps: updating each weight coefficient set and a bias set in a parameter set of each level of the multi-level deep learning neural network based on the compression loss and by using a stochastic gradient descent method; pruning the weight coefficient set corresponding to each level of the multi-level deep learning neural network based on the compression loss and by using a proximal operator of a regularizer; wherein the updating process of each weight coefficient set and the bias set in the parameter set of each level of the multi-level deep learning neural network and the pruning process of the weight coefficient set corresponding to each level of the multi-level deep learning neural network are alternately iterated.

7. The method of claim 1, wherein, The sparse coding loss corresponding to the sparse coding and the sparse weight loss corresponding to the sparse weight are determined, including: determining coding coefficients corresponding to the sparse coding; determining weight parameters corresponding to the sparse weight; multiplying the coding coefficients and the sparse coding of each level to obtain a sparse coding loss; multiplying the weight parameters and the sparse weight of each level to obtain a sparse weight loss; fusing the network loss, the sparse coding loss and the sparse weight loss to obtain a fused compression loss, including: summing the network loss, the sparse coding loss and the sparse weight loss to obtain a compression loss.

8. A neural network compression apparatus, comprising: including: a network loss acquisition unit, configured to input a sample set labeled with a label into a multi-level deep learning neural network to be compressed to obtain an output of each level of the multi-level deep learning neural network to be compressed; and calculate an overall loss of the multi-level deep learning neural network to be compressed based on the output of each level and the label of the sample set corresponding to each level, as a network loss, wherein the multi-level deep learning neural network is a medical image analysis network; a sparse determination unit, configured to determine sparse coding and sparse weight corresponding to the multi-level deep learning neural network; a sparse loss determination unit, configured to determine a sparse coding loss corresponding to the sparse coding and a sparse weight loss corresponding to the sparse weight; a loss fusion unit, configured to fuse the network loss, the sparse coding loss and the sparse weight loss to obtain a fused compression loss; wherein an expression of the compression loss is as follows: ； In the formula, To compress losses; For a multi-level deep learning neural network, the parameter set is used. This represents the output of the Lth layer of a multi-level deep learning neural network, where L represents the last layer of the multi-level deep learning neural network. r is the set of weight coefficients for the output layer. i+1 This represents the reward at time i+1; Let F represent the discount rate at time t+1; F is the network loss. This is a weight adjustment parameter used to adjust the sparsity of the sparse weights; G represents the coding coefficients; G is the set of sparse distributions. This represents the sparse weight penalty term; For the first multi-level deep learning neural network The set of weight coefficients for each layer; This indicates that the input is s i At that time, the output of the Lth layer of a multi-level deep learning neural network; Indicates the weighting parameter; Indicates that the input is At that time, the transpose of the output of the Lth layer of a multi-level deep learning neural network; Indicates that the input is At that time, the transpose of the output of the Lth layer of a multi-level deep learning neural network; For the first in a multi-level deep learning neural network The output distribution corresponding to the layer is an exponential distribution; To adjust the parameter values ​​for the desired encoded value; The difference in distribution; yes The element in the k-th row and j-th column of the array; There are a total List, OK; a network compression unit, configured to compress parameters in an output process of each level of the multi-level deep learning neural network by using the compression loss to obtain a compressed multi-level deep learning neural network.

9. A neural network compression device, comprising: including a memory and a processor; the memory, configured to store a program; the processor, configured to execute the program to implement each step of the neural network compression method according to any one of claims 1-7.

10. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement each step of the neural network compression method according to any one of claims 1-7. The computer program is executed by the processor to implement each step of the neural network compression method according to any one of claims 1-7.

Images