Neural network model compression method

By converting the neural network model into a quadratic unconstrained binary optimization problem, using quantum optimization model expressions to solve the problem of low model compression efficiency and high resource utilization in the existing technology, and an efficient and fast neural network is achieved Model compression.

CN119005261AActive Publication Date: 2024-11-22BEIJING QBOSON QUANTUM TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410765323.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-14
Publication Date
2024-11-22
Estimated Expiration
2044-06-14

AI Technical Summary

Technical Problem

The existing neural network model compression methods have limitations and cannot effectively reduce the model size and computational complexity without affecting the model performance. The knowledge distillation method requires a lot of training time and computing resources.

Method used

The model is compressed and converted into a quadratic unconstrained binary optimization (QUBO) problem. By establishing the layer mapping relationship between the teacher model and the student model, quantum optimization model expression is constructed, and student model parameters are obtained through quantum optimization solutions.

Benefits of technology

It realizes more efficient and fast neural network model compression, reducing computational complexity and storage requirements, while maintaining model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005261A_ABST
    Figure CN119005261A_ABST
Patent Text Reader

Abstract

The invention discloses a neural network model compression method. The method comprises the following steps: setting a mapping relation between a teacher model and a student model in a depth direction according to scale parameters of the teacher model and the student model; aiming at one layer of the student model and one or more layers of the teacher model with a mapping relationship, based on the weight and output of one or more layers in the teacher model, constructing a quantum optimization model expression aiming at one layer of the student model; and solving the quantum optimization model expression to obtain model parameters of one layer of the student model. Through the neural network model compression method provided by the invention, the model is compressed and converted into QUBO or other similar problems, so that a more efficient and faster neural network model compression scheme can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and deep learning, and in particular to a neural network model compression method. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, neural network models have achieved remarkable results in multiple technical segments. However, as the complexity of the model increases, the size and computing requirements of the model also grow dramatically, which brings huge challenges to the training, storage and deployment of the model.

[0003] In this context, neural network model compression methods have emerged. Neural network model compression methods aim to reduce the size and computational complexity of the model without affecting the performance of the model. Traditional neural network model compression methods mainly include parameter pruning, low-rank approximation and quantization. Parameter pruning reduces model parameters by removing neurons or connections with small weights or small impact on the output, but schemes with good compression effects often lead to reduced model performance. Low-rank approximation uses the low-rank property of the matrix to approximate the original model, but this method is usually only applicable to models with specific structures. Quantization technology achieves model compression by reducing the precision of model parameters (such as weights), which determines that quantization technology cannot change the overall structure and topology of the model, and the compression effect is constrained. Once over-quantized, it may cause unacceptable precision loss.

[0004] With the development of deep learning frameworks and hardware, some new model compression techniques have emerged, and knowledge distillation is a representative example. Knowledge distillation trains a smaller student network to imitate the behavior of a larger teacher network, so that the student network can achieve performance close to that of the teacher network while maintaining low computational complexity and storage requirements. However, model compression through knowledge distillation requires a lot of training time and computing resources, and even increases resource usage compared to training a model of the same size alone.

[0005] Thus, although there are many existing neural network model compression methods that can reduce the size and computational complexity of the model to a certain extent, these methods still have their own limitations. Summary of the invention

[0006] In order to overcome the problems existing in the prior art, the present application proposes a neural network model compression method, which converts model compression into a quadratic unconstrained binary optimization (QUBO) problem or a similar problem, thereby achieving a more efficient and faster neural network model compression technical solution.

[0007] According to one aspect of the present application, a neural network model compression method is proposed. The method includes: setting a mapping relationship between the teacher model and the student model in the depth direction according to the scale parameters of the teacher model and the student model; for a layer of the student model and one or more layers of the teacher model having a mapping relationship, constructing a quantum optimization model expression for the layer of the student model based on the weights and outputs of the one or more layers in the teacher model; and obtaining the model parameters of the layer of the student model by solving the quantum optimization model expression.

[0008] In some embodiments, before the step of setting the mapping relationship between the teacher model and the student model in the depth direction, the method of the present application further includes: determining whether the scale index of the student model is greater than a threshold number. In the case where the scale index of the student model is greater than the threshold number, the method of the present application executes the step of setting the mapping relationship between the teacher model and the student model in the depth direction; in the case where the scale index of the student model is less than or equal to the threshold number, the step of setting the mapping relationship between the teacher model and the student model in the depth direction is not executed.

[0009] In some embodiments, without performing the step of setting the mapping relationship between the teacher model and the student model in the depth direction, the present application determines the parameters of the student model through the following steps: determining the quantum optimization model expression of the student model according to the scale parameter of the student model, wherein the quantum optimization model expression includes a truth constraint term corresponding to the distance between the output of the output layer of the student model and the true label of each data sample and a consistency constraint term corresponding to the distance between the output of the output layer of the student model and each label generated by the teacher model; and obtaining the parameters of the student model by solving the quantum optimization model expression.

[0010] In some embodiments, the truth constraint term is,

[0011]

[0012] Where N represents the number of layers of the student model, a iN represents the output vector of the output layer, i represents the i-th data sample, represents the true label of the i-th data sample;

[0013] The consistency constraint is:

[0014]

[0015] in, Represents the soft label generated by the teacher model for the i-th data sample.

[0016] In some embodiments, the scale indicator is based on any one or a combination of the following: the number of weights, the number of neurons, and the number of layers.

[0017] In some embodiments, the scale parameter is one or more of the following: a depth ratio of the model, a width of a layer, a position of a layer in a depth direction, a type of layer, and a connection relationship between layers.

[0018] In some embodiments, if the first layer and the second layer in the student model have a front-to-back position relationship in the depth direction, then the layer in the teacher model that has a mapping relationship with the first layer and the layer that has a mapping relationship with the second layer also have the same front-to-back position relationship in the depth direction.

[0019] In some embodiments, in the step of constructing a quantum optimization model expression for the layer of the student model, when there are multiple layers having a mapping relationship with the layer of the student model, the construction is implemented based on one of the following: the weights and outputs of each layer in the multiple layers, the weights of each layer in the multiple layers and the output of the last layer, and the weights and output of the last layer in the multiple layers.

[0020] In some embodiments, before the step of constructing a quantum optimization model expression for the layer of the student model, the method further includes: for a layer of the student model and one or more layers of the teacher model having a mapping relationship, determining whether the width of the layer of the student model is the same as the width of the layer of the teacher model or the width of the last layer of the multiple layers.

[0021] In some embodiments, when the width of the layer of the student model is the same as the width of the layer of the teacher model or the width of the last layer of the multiple layers, the step of constructing a quantum optimization model expression for the layer of the student model includes: constructing an objective function for the layer of the student model, wherein the objective function at least includes Item and Item, where W and are the weight matrices corresponding to the layer of the student model and the layer or layers of the teacher model, respectively, t and are respectively the output vector of the layer of the student model and the output vector of the layer or the last layer of the multiple layers of the teacher model, ρ is the activation function of the model, and α is the strength coefficient of the weight consistency constraint; according to the objective function, the quantum optimization model expression is constructed.

[0022] In some embodiments, when the width of the layer of the student model is different from the width of the layer of the teacher model or the width of the last layer of the multiple layers, the step of constructing a quantum optimization model expression for the layer of the student model includes: constructing an objective function for the layer of the student model, wherein the objective function at least includes |x t -Wx t-1 | 2 Item and Item, where W is the weight matrix corresponding to the layer of the student model, x t and are the output vector of the first layer of the student model and the output vector of the last layer of the first layer or the last layer of the multi-layer of the teacher model, respectively, t-1 is the output vector of the previous layer of the layer of the student model, ρ is the activation function of the model, and B is the width alignment matrix; according to the objective function, the quantum optimization model expression is constructed.

[0023] In some embodiments, the quantum optimization model is a quadratic unconstrained binary optimization (QUBO) model or an Ising model, and in the step of solving the quantum optimization model expression, the solution is performed by a quantum annealing machine or a coherent Ising machine.

[0024] Through the neural network model compression method proposed in this application, by establishing a layer mapping relationship between the teacher model and the student model, the problem of neural network model compression can be converted into several QUBO problems or similar problems, thereby realizing a more efficient and faster neural network model compression technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram showing the network structure of an exemplary neural network.

[0026] Figure 2 A schematic flow chart of a neural network model compression method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present disclosure. It should be noted that throughout the accompanying drawings, the same elements are represented by the same or similar figure marks. In the following description, some specific embodiments are only used for descriptive purposes and should not be understood as any limitation to the present disclosure, but are only examples of the embodiments of the present disclosure. Conventional structures or constructions will be omitted when they may cause confusion in the understanding of the present disclosure.

[0028] Neural Network Model

[0029] Figure 1 FIG. 1 shows a schematic diagram of a network structure of an exemplary neural network 100. Figure 1 As shown, the neural network 100 includes an input layer, an output layer and several hidden layers, each layer including several neurons, such as Figure 1 The input layer in exemplarily includes two neurons, and each hidden layer exemplarily includes three neurons.

[0030] Figure 1 The neural network in is a multi-layer feedforward neural network, also known as a multi-layer perceptron (MLP), that is, each layer of neurons is fully interconnected with the neurons in the next layer, and there is no same-layer connection or cross-layer connection between neurons. In the embodiments of the present application, without loss of generality, a multi-layer feedforward neural network is used as an example for description. It should be understood that the technical solution of the present application is also applicable to other types of neural networks.

[0031] From the neural network structure, we can see that for each layer of neurons, that is, the neurons in this layer, the output from the neurons in the previous layer is input to each neuron in this layer through the connection between neurons. At the neurons in this layer, the activation function maps the input to the output of the neurons in this layer, and then transmits it to the neurons in the next layer. This can be shown by the following expression:

[0032] s t =W t x t-1 +b t

[0033] x t =ρ(s t )

[0034] Among them, t represents the current layer, t-1 represents the previous layer, and x t-1 and x tare the output vectors composed of the outputs of the neurons in the previous layer and the current layer; W t is the weight matrix of each connection between the previous layer and the current layer (also referred to as the weight matrix of the current layer in this application), where the matrix elements represent the weight coefficients of the connections between the corresponding neurons in the two layers, and b t represents the offset vector; ρ represents the activation function, s t Represents the output vector x t The value before activation.

[0035] Compression of Neural Network Models

[0036] As mentioned above in this application, in order to reduce model complexity and other requirements, it is necessary to compress the neural network model, that is, to build a smaller-scale neural network model based on the existing larger-scale neural network model, and the effect of the smaller-scale neural network model is similar to that of the larger-scale neural network model. Specifically, possible results of model compression include a reduction in the number of layers, a reduction in the number of neurons in a layer, and a disconnection of connections between neurons.

[0037] In the embodiments of the present application, the larger-scale neural network model is represented by the “teacher model”, and the smaller-scale neural network model is represented by the “student model”. It should be noted that the use of the terms “teacher model” and “student model” in this application does not mean that the technical solution of this patent is limited to a knowledge distillation solution.

[0038] The following describes the neural network model compression method proposed in this application in conjunction with the accompanying drawings.

[0039] Figure 2 A schematic flow chart of a neural network model compression method 200 according to an embodiment of the present application is shown.

[0040] like Figure 2 As shown, the neural network model compression method 200 includes steps S210-S230.

[0041] In step S210, a mapping relationship between the teacher model and the student model in the depth direction is set according to the scale parameters of the teacher model and the student model.

[0042] The number of layers in the teacher model is generally greater than that in the student model. It can be considered that each layer in the student model is compressed from one or more layers in the teacher model. Therefore, there is a mapping relationship between the teacher model and the student model in the depth direction of the neural network. As for which layer or layers in the teacher model the layer in the student model corresponds to, there are many options, and different options usually lead to different compression results. Therefore, when establishing a mapping relationship, many factors can be considered in order to obtain better compression results.

[0043] In some embodiments, the scale parameter includes a depth ratio of the model. For example, when the ratio of the teacher model to the student model is an integer greater than 1 (e.g., 2), a mapping relationship can be established between the layers numbered 2t-1 and 2t in the teacher model and the layer numbered t in the student model (t is a positive integer).

[0044] In some embodiments, the scale parameter includes the width of the layer. For example, two adjacent layers with very wide layer widths in the teacher model can be mapped to different layers in the student model, respectively.

[0045] In some embodiments, the scale parameter includes the position of the layer in the depth direction, for example, more layers in the middle of the teacher model are mapped to a layer in the student model, while fewer layers in the teacher model located at the output and the output ends are mapped to a layer in the student model.

[0046] In some embodiments, the scale parameter includes the type of layer, for example, an output layer in the teacher model can be mapped to a layer in the student model.

[0047] In some embodiments, the scale parameter includes the connection relationship between layers. For example, two layers with cross-layer connections in the teacher model and the layers between them can be mapped to the same layer in the student model.

[0048] In some embodiments, after mapping the layers in the teacher model to the student model, the mapping will maintain the front-to-back relationship of different layers. For example, if the first layer and the second layer in the student model have a front-to-back position relationship in the depth direction, then the layer in the teacher model that has a mapping relationship with the first layer and the layer that has a mapping relationship with the second layer also have the same front-to-back position relationship in the depth direction.

[0049] In step S220, for a layer of a student model and one or more layers of a teacher model having a mapping relationship, a quantum optimization model expression for the layer of the student model is constructed based on the weights and outputs of the one or more layers in the teacher model.

[0050] In this step, based on the mapping relationship established in step S210, the construction of each layer in the student model is constrained to the layer in the teacher model that has a mapping relationship with it.

[0051] When the layer in the teacher model corresponding to a certain layer of the student model includes only one layer, the construction of the quantum optimization model expression for the layer of the student model is based on the weights and outputs of the corresponding layer in the teacher model.

[0052] When the layer in the teacher model corresponding to a certain layer of the student model includes multiple layers, the construction of the quantum optimization model expression for the layer of the student model can be based on the weights and outputs of each layer in the multiple layers, based on the weights of each layer in the multiple layers and the output of the last layer in the multiple layers, or based on the weights and output of the last layer in the multiple layers.

[0053] After mapping the teacher model and the student model in the depth direction of the model, we can also consider the correspondence between the two models in the width direction.

[0054] Before step S220, the method further includes determining whether the width of the layer in the student model is the same as the width of the layer in the teacher model that has a mapping relationship with the layer. If the layer in the teacher model that has a mapping relationship with the layer includes multiple layers, this step can determine whether the width of the layer in the student model is the same as the width of the last layer in the multiple layers.

[0055] In the case of the same width, in step S220, the objective function corresponding to the layer of the student model can be constructed according to the following scheme, and then the quantum optimization model expression can be constructed according to the objective function: the objective function at least includes Item and Item, where W and are the weight matrices corresponding to the layer of the student model and the layer or layers of the teacher model, respectively, t and are the output vector of the first layer of the student model and the output vector of the last layer of the first layer or the last layer of the teacher model, ρ is the activation function of the model, and α is the strength coefficient of the weight consistency constraint. The term represents the consistency between the output of the student model and the teacher model. The term represents the consistency of the weight of the student model with the teacher model. In some embodiments, for example, when the accuracy of W is low, the strength coefficient α can be 0, that is, It is actually an optional option.

[0056] In the case of different widths, in step S220, the objective function corresponding to the layer of the student model can be constructed according to the following scheme, and then the quantum optimization model expression can be constructed according to the objective function: the objective function at least includes |x t -Wx t-1 | 2 Item and Item, where W is the weight matrix corresponding to the layer of the student model, x t and are the output vector of the first layer of the student model and the output vector of the last layer of the first layer or the last layer of the multi-layer of the teacher model, respectively,t-1 is the output vector of the previous layer of the student model, ρ is the activation function of the model, and B is the width alignment matrix. t -Wx t-1 | 2 The term represents the feedforward process of the neural network, The term represents the consistency of the output of the student model with the teacher model.

[0057] It should be pointed out that in When the term exists, higher-order terms will be generated and need to be reduced.

[0058] For example, in some embodiments, a penalty term ∑h(B ij ,x t,i ,c ij ), where B ij is the matrix element of B, x t,i is the output vector x t The i-th component of ij is an auxiliary variable, and the expression of function h is

[0059] h(B ij ,x t,i ,c ij )=3c ij +B ij x t,i -2B ij c ij -2x t,i c ij

[0060] Among them, when B ij x t,i =c ij When , the function h is 0, otherwise, the function h is greater than 0.

[0061] In some embodiments, the quantum optimization model is a QUBO model. In other embodiments, the quantum optimization model is an Ising model.

[0062] In step S230, the model parameters of the layer of the student model are obtained by solving the quantum optimization model expression.

[0063] When the quantum optimization model is a QUBO model, the quantum optimization model expression is solved by a quantum annealing machine.

[0064] When the quantum optimization model is an Ising model, the quantum optimization model expression is solved by an Ising machine (eg, a coherent Ising machine).

[0065] Combination of the above Figure 2The neural network model compression method 200 of the present application is described. In fact, when the size of the student model is not large, the mapping relationship between the teacher model and the student model can be ignored, and a new student model can be generated by referring to the parameters in the teacher model at one time.

[0066] Specifically, before step S210, it can be determined whether the scale index of the student model is greater than a threshold number, and step S210 is performed only when the scale index of the student model is greater than the threshold number.

[0067] In some embodiments, the scale indicator is based on any one or a combination of the following: the number of weights, the number of neurons, and the number of layers.

[0068] When the scale index of the student model is less than or equal to the threshold number, the method of the present application does not execute step S210, but directly determines the quantum optimization model expression of the student model according to the scale parameter of the student model, and then obtains the parameters of the student model by solving the quantum optimization model expression.

[0069] In an embodiment of the present application, the quantum optimization model expression includes a truth constraint term corresponding to the distance between the output of the output layer of the student model and the true label of each data sample

[0070]

[0071] Where N represents the number of layers of the student model, a iN represents the output vector of the output layer, i represents the i-th data sample, represents the true label of the i-th data sample.

[0072] In an embodiment of the present application, the quantum optimization model expression also includes a consistency constraint term corresponding to the distance between the output of the output layer of the student model and each label generated by the teacher model

[0073]

[0074] in, Represents the soft label generated by the teacher model for the i-th data sample.

[0075] When the student model is MLP, the quantum optimization model expression can be expressed as,

[0076]

[0077] Among them, t k represents the slack variable.

[0078] In addition to the above embodiments, the present application may also have other implementation modes. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present application.

Claims

1. A neural network model compression method, characterized in that: include: According to the scale parameters of the teacher model and the student model, a mapping relationship between the teacher model and the student model in the depth direction is set; For a layer of the student model and one or more layers of the teacher model having a mapping relationship, constructing a quantum optimization model expression for the layer of the student model based on the weights and outputs of the one or more layers in the teacher model; and The model parameters of the first layer of the student model are obtained by solving the quantum optimization model expression.

2. The neural network model compression method according to claim 1, before the step of setting the mapping relationship between the teacher model and the student model in the depth direction, further comprising: determining whether a size indicator of the student model is greater than a threshold amount, and In the case where the scale index of the student model is greater than a threshold number, the step of setting the mapping relationship between the teacher model and the student model in the depth direction is performed.

3. The neural network model compression method according to claim 2 further comprises, when the scale index of the student model is less than or equal to the threshold number, not performing the step of setting the mapping relationship between the teacher model and the student model in the depth direction, and The parameters of the student model are determined by the following steps: According to the scale parameter of the student model, a quantum optimization model expression of the student model is determined, wherein: The quantum optimization model expression includes a truth constraint term corresponding to the distance between the output of the output layer of the student model and the true label of each data sample and a consistency constraint term corresponding to the distance between the output of the output layer of the student model and each label generated by the teacher model; The parameters of the student model are obtained by solving the quantum optimization model expression.

4. The neural network model compression method according to claim 3, wherein: The truth value constraint term is, Where N represents the number of layers of the student model, a iN represents the output vector of the output layer, i represents the i-th data sample, represents the true label of the i-th data sample; The consistency constraint is: in, Represents the soft label generated by the teacher model for the i-th data sample.

5. The neural network model compression method according to any one of claims 2 to 4, wherein: The scale indicator is based on any one or a combination of the following: the number of weights, the number of neurons, and the number of layers.

6. The neural network model compression method according to claim 1, wherein: The scale parameter is one or more of the following: a depth ratio of the model, a width of a layer, a position of a layer in a depth direction, a type of a layer, and a connection relationship between layers.

7. The neural network model compression method according to claim 1, wherein: If the first layer and the second layer in the student model have a front-to-back position relationship in the depth direction, then the layer in the teacher model that has a mapping relationship with the first layer and the layer that has a mapping relationship with the second layer also have the same front-to-back position relationship in the depth direction.

8. The neural network model compression method according to claim 1, wherein: In the step of constructing a quantum optimization model expression for the layer of the student model, when there are multiple layers having a mapping relationship with the layer of the student model, the construction is implemented based on one of the following: weights and outputs of each layer in the multiple layers, weights of each layer in the multiple layers and output of the last layer, and weights and output of the last layer in the multiple layers.

9. The neural network model compression method according to any one of claims 1 and 6-8, before the step of constructing a quantum optimization model expression for the layer of the student model, the method further comprises: For a layer of the student model and one or more layers of the teacher model having a mapping relationship, determine whether a width of the layer of the student model is the same as a width of the layer of the teacher model or a width of the last layer of the multiple layers.

10. The neural network model compression method according to claim 9, wherein: In the case where the width of the layer of the student model is the same as the width of the layer of the teacher model or the width of the last layer of the multiple layers, the step of constructing a quantum optimization model expression for the layer of the student model comprises: Construct an objective function for the layer of the student model, wherein the objective function at least includes Item and Item, where W and are the weight matrices corresponding to the layer of the student model and the layer or layers of the teacher model, respectively, t and are respectively the output vector of the layer of the student model and the output vector of the layer or the last layer of the multiple layers of the teacher model, ρ is the activation function of the model, and α is the strength coefficient of the weight consistency constraint; According to the objective function, the quantum optimization model expression is constructed.

11. The neural network model compression method according to claim 9, wherein: In the case where the width of the layer of the student model is different from the width of the layer of the teacher model or the width of the last layer of the multiple layers, the step of constructing a quantum optimization model expression for the layer of the student model comprises: Construct an objective function for the layer of the student model, wherein the objective function at least includes |x t -Wx t-1 | 2 Item and Item, where W is the weight matrix corresponding to the layer of the student model, x t and are the output vector of the first layer of the student model and the output vector of the last layer of the first layer or the last layer of the multi-layer of the teacher model, respectively, t-1 is the output vector of the previous layer of the layer of the student model, ρ is the activation function of the model, and B is the width alignment matrix; According to the objective function, the quantum optimization model expression is constructed.

12. The neural network model compression method according to claim 1, wherein: The quantum optimization model is a quadratic unconstrained binary optimization (QUBO) model or an Ising model, and In the step of solving the quantum optimization model expression, the solution is performed by a quantum annealing machine or a coherent Ising machine.

Citation Information

Patent Citations

  • Distributed energy storage robustness evaluation method based on neural network

    CN118171099A

  • Machine learning mapping for quantum processing units

    US11436519B1

  • Systems and methods for quantum processing of data

    US20160321559A1

  • Apparatus and methods for quantum computing and machine learning

    US20200401920A1

  • Quantum computing based deep learning for detection, diagnosis and other applications

    US20230094389A1