Neural network model compression method

Abstract

The application discloses a neural network model compression method. The method comprises the following steps: setting a mapping relationship of a teacher model and a student model in a depth direction according to scale parameters of the teacher model and the student model; constructing a quantum optimization model expression for one layer of the student model based on weights and outputs of one or more layers of the teacher model for the one layer of the student model and the one or more layers of the teacher model having the mapping relationship; and obtaining model parameters of the one layer of the student model by solving the quantum optimization model expression. Through the neural network model compression method, the model compression is converted into a QUBO or other similar problem, so that a more efficient and rapid neural network model compression scheme can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and deep learning, and in particular to a neural network model compression method. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, neural network models have made significant achievements in multiple sub-technology fields. However, as the complexity of the model increases, the size and computational requirements of the model also increase dramatically, which brings great challenges to the training, storage and deployment of the model.

[0003] In this context, neural network model compression methods have emerged. Neural network model compression methods aim to reduce the size and computational complexity of the model without affecting the performance of the model. Traditional neural network model compression methods mainly include parameter pruning, low-rank approximation and quantization. Parameter pruning reduces model parameters by removing neurons or connections with small weights or little impact on output, but good compression solutions often lead to a decrease in model performance. Low-rank approximation uses the low-rank property of matrices to approximate the original model, but this method is usually only applicable to models with specific structures. Quantization technology reduces the precision of model parameters (such as weights) to achieve model compression, which determines that quantization technology cannot change the overall structure and topology of the model, and the compression effect is constrained, and once the quantization is too much, it may cause unacceptable precision loss.

[0004] With the development of deep learning frameworks and hardware, some new model compression techniques have emerged, and knowledge distillation is one of them. Knowledge distillation trains a smaller student network to imitate the behavior of a larger teacher network, so that the student network can maintain a low computational complexity and storage requirement while achieving performance close to that of the teacher network. However, model compression through knowledge distillation requires a large amount of training time and computing resources, and even compared with training a model of the same size alone, it also increases the resource occupancy intensity.

[0005] Therefore, although there are many existing neural network model compression methods that can reduce the size and computational complexity of the model to some extent, these methods still have their own limitations. SUMMARY

[0006] In order to overcome the problems existing in the prior art, the present application proposes a neural network model compression method, which converts model compression into a quadratic unconstrained binary optimization (QUBO) problem or similar problem, thereby enabling a more efficient and fast neural network model compression technical solution.

[0007] According to an aspect of the present application, a neural network model compression method is provided. The method comprises: setting a mapping relationship of a teacher model and a student model in a depth direction according to scale parameters of the teacher model and the student model; constructing a quantum optimization model expression for a layer of the student model based on weights and outputs of one or more layers of the teacher model, for the layer of the student model and the one or more layers of the teacher model having the mapping relationship; and obtaining model parameters of the layer of the student model by solving the quantum optimization model expression.

[0008] In some embodiments, before the step of setting the mapping relationship of the teacher model and the student model in the depth direction, the method of the present application further comprises: determining whether a scale index of the student model is greater than a threshold number. In the case that the scale index of the student model is greater than the threshold number, the method of the present application performs the step of setting the mapping relationship of the teacher model and the student model in the depth direction; in the case that the scale index of the student model is less than or equal to the threshold number, the step of setting the mapping relationship of the teacher model and the student model in the depth direction is not performed.

[0009] In some embodiments, without performing the step of setting the mapping relationship of the teacher model and the student model in the depth direction, the method of the present application determines the parameters of the student model by: determining a quantum optimization model expression of the student model according to scale parameters of the student model, wherein the quantum optimization model expression includes a true value constraint term corresponding to a distance between an output of an output layer of the student model and a true label of each data sample and a consistency constraint term corresponding to a distance between the output of the output layer of the student model and each label generated by the teacher model; and obtaining the parameters of the student model by solving the quantum optimization model expression.

[0010] In some embodiments, the true value constraint term is,

[0011]

[0012] wherein N represents a number of layers of the student model, a iN represents an output vector of the output layer, i represents an i-th data sample, represents a true label of the i-th data sample;

[0013] The consistency constraint term is,

[0014]

[0015] wherein, represents a soft label generated by the teacher model for the i-th data sample.

[0016] In some embodiments, the scale indicator is based on any one or a combination of the following: the number of weights, the number of neurons, and the number of layers.

[0017] In some embodiments, the scale parameter is one or more of the following: the depth ratio of the model, the width of the layer, the position of the layer in the depth direction, the type of the layer, and the connection relationship between layers.

[0018] In some embodiments, if the first layer and the second layer in the student model have a front-back positional relationship in the depth direction, the layers in the teacher model that have a mapping relationship with the first layer and the layers that have a mapping relationship with the second layer also have the same front-back positional relationship in the depth direction.

[0019] In some embodiments, in the step of constructing the quantum optimization model expression for the one layer of the student model, if the layer having a mapping relationship with the one layer of the student model is multiple layers, the construction is based on one of the following: the weights and outputs of each layer in the multiple layers, the weights of each layer in the multiple layers and the output of the last layer, the weights and output of the last layer in the multiple layers.

[0020] In some embodiments, before the step of constructing the quantum optimization model expression for the one layer of the student model, the method further comprises: for a layer of the student model and one or more layers of the teacher model having a mapping relationship, determining whether the width of the one layer of the student model is the same as the width of the one layer of the teacher model or the width of the last layer in the multiple layers.

[0021] In some embodiments, in the case where the width of the one layer of the student model is the same as the width of the one layer of the teacher model or the width of the last layer in the multiple layers, the step of constructing the quantum optimization model expression for the one layer of the student model comprises: constructing an objective function for the one layer of the student model, wherein the objective function at least includes term and term, wherein W and are the weight matrices corresponding to the one layer of the student model and the one or more layers of the teacher model respectively, x t and are the output vector of the one layer of the student model and the output vector of the last layer in the one or more layers of the teacher model respectively, p is the activation function of the model, and a is the strength coefficient of the weight consistency constraint; according to the objective function, the quantum optimization model expression is constructed.

[0022] In some embodiments, when the width of a layer in the student model differs from the width of a layer in the teacher model or the width of the last layer in the multi-layer structure, the step of constructing a quantum optimization model expression for the layer of the student model includes: constructing an objective function for the layer of the student model, wherein the objective function includes at least |x t -Wx t-1 | 2 Item and Item, where W is the weight matrix corresponding to the layer of the student model, x t and These are the output vectors of the student model's first layer and the last layer of the teacher model's first or multi-layer output vectors, respectively. t-1 ρ is the output vector of the layer above the layer of the student model, ρ is the activation function of the model, and B is the width alignment matrix; the quantum optimization model expression is constructed according to the objective function.

[0023] In some embodiments, the quantum optimization model is a quadratic unconstrained binary optimization (QUBO) model or an Ising model, and in the step of solving the expression of the quantum optimization model, the solution is performed by a quantum annealing machine or a coherent Ising machine.

[0024] The neural network model compression method proposed in this application can transform the problem of neural network model compression into several QUBO problems or similar problems by establishing a layer mapping relationship between the teacher model and the student model, thereby achieving a more efficient and faster neural network model compression technical solution. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating an exemplary neural network structure.

[0026] Figure 2 A schematic flowchart of a neural network model compression method according to an embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the described embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. It should be noted that throughout the accompanying drawings, the same elements are represented by the same or similar reference numerals. In the following description, some specific embodiments are used for descriptive purposes only and should not be construed as limiting this disclosure in any way, but are merely examples of embodiments of this disclosure. Conventional structures or constructions will be omitted where they may cause confusion in understanding this disclosure.

[0028] Neural network model

[0029] Figure 1 A schematic diagram of an exemplary neural network 100 is shown. Figure 1 As shown, the neural network 100 includes an input layer, an output layer, and several hidden layers, each layer containing several neurons, for example... Figure 1 The input layer in the example includes two neurons, and each hidden layer includes three neurons.

[0030] Figure 1 The neural network in this application is a multilayer feedforward neural network, also known as a multilayer perceptron (MLP), where each layer of neurons is fully interconnected with the neurons in the next layer, and there are no connections between neurons within the same layer or across layers. In the embodiments of this application, a multilayer feedforward neural network is used as an example for description without loss of generality. It should be understood that the technical solutions of this application are also applicable to other types of neural networks.

[0031] As can be seen from the neural network structure, for each neuron in a given layer (i.e., the neurons in that current layer), the output from the neurons in the previous layer is input to each neuron in that layer via inter-neuron connections. At the neurons in that layer, the activation function maps the input to the output of that layer, which is then sent to the neurons in the next layer. This can be illustrated by the following expression:

[0032] s t =W t x t-1 +b t

[0033] x t =ρ(s) t )

[0034] Where t represents the current layer, t-1 represents the previous layer, and x t-1 and x tis an output vector composed of outputs of neurons in the previous layer and the current layer respectively; W t is a weight matrix of connections between the previous layer and the current layer (in this application, also referred to as the weight matrix of the current layer), wherein a matrix element represents a weight coefficient of a connection between corresponding neurons between two layers, b t represents a bias vector; p represents an activation function, s t represents an output vector x t before activation.

[0035] Compression of neural network models

[0036] As described above, in order to reduce model complexity and other requirements, it is necessary to compress the neural network model, that is, to construct a smaller scale neural network model on the basis of an existing larger scale neural network model, and the smaller scale neural network model has similar effects to the larger scale neural network model. Specifically, possible results of model compression include reduction of the number of layers, reduction of the number of neurons in a layer, disconnection between neurons, etc.

[0037] In an embodiment of the present application, the larger scale neural network model is referred to as a "teacher model", and the smaller scale neural network model is referred to as a "student model". It should be noted that the use of the terms "teacher model" and "student model" in the present application does not mean that the technical solution of the present patent is limited to a knowledge distillation scheme.

[0038] The neural network model compression method proposed in the present application is described below in conjunction with the accompanying drawings.

[0039] Figure 2 A schematic flowchart of a neural network model compression method 200 according to an embodiment of the present application is shown.

[0040] As shown in Figure 2 , the neural network model compression method 200 includes steps S210-S230.

[0041] In step S210, a mapping relationship between the teacher model and the student model in the depth direction is set according to the scale parameters of the teacher model and the student model.

[0042] The number of layers of the teacher model is generally greater than the number of layers of the student model, and it can be considered that each layer in the student model is compressed from one or more layers in the teacher model, and thus there is a mapping relationship between the teacher model and the student model in the depth direction of the neural network. As for which layer or which layers in the teacher model correspond to a layer in the student model, there can be multiple choices, and different choices will usually result in different compression results. Therefore, when establishing the mapping relationship, many factors can be considered in order to obtain better compression results.

[0043] In some embodiments, the scale parameter includes the depth ratio of the model, for example, when the ratio of the teacher model to the student model is an integer greater than 1 (for example, 2), the layers numbered 2t-1 and 2t in the teacher model can be mapped to the layer numbered t in the student model (t is a positive integer).

[0044] In some embodiments, the scale parameter includes the width of the layer, for example, for two adjacent layers in the teacher model with very wide layer width, they can be respectively mapped to different layers in the student model.

[0045] In some embodiments, the scale parameter includes the position of the layer in the depth direction, for example, more layers in the middle position of the teacher model are mapped to one layer in the student model, and fewer layers at the output and output ends of the teacher model are mapped to one layer in the student model.

[0046] In some embodiments, the scale parameter includes the type of layer, for example, the output layer in the teacher model can be mapped to one layer in the student model.

[0047] In some embodiments, the scale parameter includes the connection relationship between layers, for example, two layers and the layers between them in the teacher model that are connected across layers can be mapped to the same layer in the student model.

[0048] In some embodiments, after mapping the layers in the teacher model to the student model, the mapping will maintain the front-back relationship of different layers. For example, if the first layer and the second layer in the student model have a front-back positional relationship in the depth direction, the layers in the teacher model that have a mapping relationship with the first layer and the layers that have a mapping relationship with the second layer also have the same front-back positional relationship in the depth direction.

[0049] In step S220, for a layer of the student model and one or more layers of the teacher model having a mapping relationship, a quantum optimization model expression for the layer of the student model is constructed based on the weights and outputs of the one or more layers in the teacher model.

[0050] In this step, based on the mapping relationship established in step S210, the construction of each layer in the student model is constrained in the layer in the teacher model that has a mapping relationship with it.

[0051] In the case where the layer in the teacher model corresponding to a layer of the student model includes only one layer, the construction of the quantum optimization model expression for the layer of the student model is based on the weights and outputs of the corresponding layer in the teacher model.

[0052] In the case where the layer in the teacher model corresponding to a layer of the student model includes multiple layers, the construction of the quantum optimization model expression of the layer of the student model can be based on the weights and outputs of each layer in the multiple layers, can be based on the weights of each layer in the multiple layers and the output of the last layer in the multiple layers, or can be based on the weights and output of the last layer in the multiple layers.

[0053] After mapping the teacher model and the student model in the model depth direction, it can also be considered that the two models have correspondence in the width direction.

[0054] Before step S220, the method further includes determining whether the width of a layer in the student model is the same as the width of a layer in the teacher model having a mapping relationship with the layer. If the layer in the teacher model having a mapping relationship with the layer includes multiple layers, the step can determine whether the width of the layer in the student model is the same as the width of the last layer in the multiple layers.

[0055] In the case where the widths are the same, in step S220, a target function corresponding to the layer of the student model can be first constructed according to the following scheme, and then a quantum optimization model expression is constructed according to the target function: the target function at least includes and , where W and are weight matrices corresponding to the layer of the student model and the layer or the multiple layers of the teacher model respectively, x t and are output vectors of the layer of the student model and the layer or the last layer in the multiple layers of the teacher model respectively, and p is an activation function of the model. The strength coefficient of the weight consistency constraint is a. The term represents the consistency of the output of the student model with the teacher model, The term represents the consistency of the weight of the student model with the teacher model. In some embodiments, for example, when the precision of W is low, the strength coefficient a can be 0, that is, is an optional term.

[0056] In the case where the widths are different, in step S220, a target function corresponding to the layer of the student model can be first constructed according to the following scheme, and then a quantum optimization model expression is constructed according to the target function: the target function at least includes t -Wx t-1 | 2 and , where W is a weight matrix corresponding to the layer of the student model, x t and are output vectors of the layer of the student model and the layer or the last layer in the multiple layers of the teacher model respectively, and xt-1 is the output vector of the previous layer of the student model, p is an activation function of the model, and B is a width-aligned matrix. Wherein, |x t -Wx t-1 | 2 represents a feedforward process of the neural network, represents the consistency of the output of the student model with the teacher model.

[0057] It should be pointed out that, in In the presence of the term, high-order terms are generated, which need to be reduced.

[0058] For example, in some embodiments, a penalty term∑h(B ij ,x t,i ,c ij ) can be added in the objective function, where B ij is the matrix element of B, x t,i is the i-th component of the output vector x t , c ij is an auxiliary variable, and the expression of the function h is

[0059] h(B ij ,x t,i ,c ij )=3c ij +B ij x t,i -2B ij c ij -2x t,i c ij

[0060] Wherein, when B ij x t,i =c ij , the function h is 0, otherwise, the function h is greater than 0.

[0061] In some embodiments, the quantum optimization model is a QUBO model. In other embodiments, the quantum optimization model is an Ising model.

[0062] In step S230, the model parameters of the one layer of the student model are obtained by solving the quantum optimization model expression.

[0063] In the case where the quantum optimization model is a QUBO model, the quantum optimization model expression is solved by a quantum annealing machine.

[0064] In the case where the quantum optimization model is an Ising model, the quantum optimization model expression is solved by an Ising machine (such as a coherent Ising machine).

[0065] The above is combined with Figure 2The neural network model compression method 200 of the present application is described. In fact, when the size of the student model is not large, the mapping relationship between the teacher model and the student model can also not be considered, and a new student model can be generated by referring to the parameters in the teacher model at one time.

[0066] Specifically, before step S210, it can be determined whether the size index of the student model is greater than a threshold number, and step S210 is executed only when the size index of the student model is greater than the threshold number.

[0067] In some embodiments, the size index is obtained based on any one or a combination of the following: the number of weights, the number of neurons, and the number of layers.

[0068] In the case where the size index of the student model is less than or equal to the threshold number, the method of the present application does not perform step S210, but directly determines the quantum optimization model expression of the student model according to the size parameter of the student model, and further obtains the parameters of the student model by solving the quantum optimization model expression.

[0069] In an embodiment of the present application, the quantum optimization model expression includes a true value constraint term corresponding to the distance between the output of the output layer of the student model and the true label of each data sample

[0070]

[0071] wherein N represents the number of layers of the student model, a iN represents the output vector of the output layer, i represents the i-th data sample, represents the true label of the i-th data sample.

[0072] In an embodiment of the present application, the quantum optimization model expression further includes a consistency constraint term corresponding to the distance between the output of the output layer of the student model and the label generated by the teacher model

[0073]

[0074] wherein, represents the soft label generated by the teacher model for the i-th data sample.

[0075] In the case where the student model is an MLP, the quantum optimization model expression can be expressed as,

[0076]

[0077] wherein t k represents a slack variable.

[0078] In addition to the above embodiments, the present application can have other implementation manners, and any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope of the present application.

Claims

1. A method for compressing neural network models using a quantum computer, characterized in that, include: Based on the scale parameters of the teacher model and the student model, set the mapping relationship between the teacher model and the student model in the depth direction; For a layer of the student model and one or more layers of the teacher model with a mapping relationship, based on the weights and outputs of the layer or more layers in the teacher model, an objective function is constructed for the layer of the student model, and a quantum optimization model expression is constructed based on the objective function; and The quantum optimization model expression is input into the quantum computer, which solves the expression to obtain the model parameters of the first layer of the student model. The quantum optimization model is a quadratic unconstrained binary optimization (QUBO) model or an Ising model, and the quantum computer is a quantum annealing machine or a coherent Ising machine.

2. The method according to claim 1, further comprising, before the step of setting the mapping relationship between the teacher model and the student model in the depth direction: Determine whether the size index of the student model is greater than a threshold number, and If the size index of the student model is greater than the threshold number, the step of setting the mapping relationship between the teacher model and the student model in the depth direction is performed.

3. The method according to claim 2 further includes, when the size index of the student model is less than or equal to a threshold number, determining the parameters of the student model through the following steps: Based on the scale parameters of the student model, the quantum optimization model expression of the student model is determined, wherein, The quantum optimization model expression includes a truth constraint term corresponding to the distance between the output of the student model's output layer and the true label of each data sample, and a consistency constraint term corresponding to the distance between the output of the student model's output layer and the labels generated by the teacher model. The parameters of the student model are obtained by solving the expression of the quantum optimization model.

4. The method according to claim 3, wherein, The truth constraint term is, Where N represents the number of layers in the student model, a iN This represents the output vector of the output layer, where i represents the i-th data sample. This represents the true label of the i-th data sample; The consistency constraint is, in, This represents the soft label generated by the teacher model for the i-th data sample.

5. The method according to any one of claims 2-4, wherein, The scale metric is based on any one or a combination of the following: number of weights, number of neurons, and number of layers.

6. The method according to claim 1, wherein, The scale parameters are one or more of the following: the model's depth ratio, the layer width, the layer's position in the depth direction, the layer type, and the connection relationships between layers.

7. The method according to claim 1, wherein, If the first and second layers in the student model have a front-to-back positional relationship in the depth direction, then the layers in the teacher model that have a mapping relationship with the first layer and the layers that have a mapping relationship with the second layer also have the same front-to-back positional relationship in the depth direction.

8. The method according to claim 1, wherein, In the step of constructing the quantum optimization model expression for the layer of the student model, if there are multiple layers that have a mapping relationship with the layer of the student model, the construction is based on one of the following: the weights and outputs of each layer in the multiple layers, the weights of each layer in the multiple layers and the output of the last layer, or the weights and outputs of the last layer in the multiple layers.

9. The method according to any one of claims 1 and 6-8, wherein prior to the step of constructing the quantum-optimized model expression for the layer of the student model, the method further comprises: For a layer of the student model and one or more layers of the teacher model that have a mapping relationship, determine whether the width of the layer of the student model is the same as the width of the layer of the teacher model or the width of the last layer in the multiple layers.

10. The method according to claim 9, wherein, When the width of a layer in the student model is the same as the width of a layer in the teacher model or the width of the last layer in the multi-layer structure, The objective function includes at least the following: Item and Items, where W and These are the weight matrices corresponding to the layer of the student model and the layer or layers of the teacher model, respectively. t and These are the output vectors of the first layer of the student model and the last layer of the first or multiple layers of the teacher model, respectively. ρ is the activation function of the model, and α is the strength coefficient of the weight consistency constraint.

11. The method according to claim 9, wherein, When the width of a layer in the student model differs from the width of a layer in the teacher model or the width of the last layer in a multi-layered system, The objective function includes at least |x t -Wx t-1 | 2 Item and Item, where W is the weight matrix corresponding to the layer of the student model, x t and These are the output vectors of the student model's first layer and the last layer of the teacher model's first or multi-layer output vectors, respectively. t-1 ρ is the output vector of the layer above the layer of the student model, ρ is the activation function of the model, and B is the width alignment matrix.

Images