Efficient neural network model compression system and method based on differentiable release quantization

Abstract

The application belongs to the technical field of artificial intelligence, and discloses an efficient neural network model compression system and method based on differentiable release quantization. L The method comprises the following steps: step 1, collecting a data set and dividing the data set into a training set, a verification set and a test set; step 2, building a neural network model comprising a plurality of layers and performing initialization processing; step 3, extracting sample data from the data set and inputting the sample data into the neural network model for forward propagation to obtain model output; step 4, calculating a total loss function; step 5, calculating the gradient of the total loss function with respect to each parameter, and performing directional updating on each parameter; and performing dynamic adjustment of the parameters through an exploration-exploitation strategy; step 6, repeating steps 3-5, and stopping iterative training when a first preset target is reached; and step 7, evaluating the performance index of the neural network model until a second preset target is reached, and then stopping training and evaluation; the application realizes more efficient compression under the premise of ensuring model accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to an efficient neural network model compression system and method based on differentiable release quantization. BACKGROUND

[0002] With the wide application and in-depth development of neural networks in various fields such as computer vision, natural language processing, etc., the model size is getting larger and the complexity is increasing.

[0003] A neural network compression method and system based on quantization are disclosed in Chinese Patent No. CN109978135B, which includes: S1, using a reinforcement learning strategy network to mine the interaction between convolution channels in a deep convolutional neural network to obtain correlation information between the convolution channels; S2, modifying and quantizing the original activation values of the deep convolutional neural network according to the correlation information. The invention restores the information loss caused by quantization by mining the interaction between the channels in the convolution and using the correlation information to modify the activation values, thereby solving the problem of information loss caused by the decline in model expression ability.

[0004] Although the above method can meet most scenarios, research and practical application of the above method and prior art have found that the above method and prior art at least have the following defects:

[0005] During the quantization process, especially when quantizing with low bits, large quantization errors are prone to occur, and these errors will be accumulated layer by layer during the forward calculation and back propagation of the neural network, inevitably causing serious precision loss; the quantization strategy is relatively fixed and lacks the ability to adaptively adjust according to different model structures and data distributions.

[0006] In view of this, the present application proposes an efficient neural network model compression system and method based on differentiable release quantization to solve the above problems. SUMMARY

[0007] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned purpose, the present application provides the following technical solution: an efficient neural network model compression method based on differentiable release quantization, comprising the following steps:

[0008] Step 1, collect a data set and divide it into a training set, a validation set and a test set according to a predetermined ratio, and normalize and standardize the data set; the data set includes sample data and corresponding true labels;

[0009] Step 2, build a neural network model containing L layers, and initialize the original parameters of the built neural network model;

[0010] Step 3, sample data is extracted from the data set and input into the neural network model, and forward propagation is performed combined with the initialized original parameters to obtain the model output;

[0011] Step 4, the total loss function is calculated according to the model output and the true label corresponding to the sample data;

[0012] Step 5, the gradient of the total loss function with respect to each parameter in the parameter set is calculated by the chain rule, the parameter set including the original parameters, the quantum ensemble parameters, the release quantization coefficient, the generator parameters and the discriminator parameters; based on the gradient of the total loss function with respect to each parameter in the parameter set, each parameter in the parameter set is updated in a directed manner; and the dynamic adjustment of the parameters is performed through the exploration-exploitation strategy;

[0013] Step 6, repeat steps 3-5 to continuously update the original parameters of the model, the release quantization coefficient, and the generator parameters, and update the exploration probability according to the preset decay rule as the training round progresses; in each iteration process, every interval of a preset time period, the gradient of the network parameters in the generator and the discriminator is calculated based on the total loss function, the importance of the network parameters is evaluated according to the gradient of the network parameters, and the network parameters are discarded according to the importance evaluation result; the performance indicators of the neural network model are evaluated according to the preset time period using the validation set, and the model architecture and each parameter in the parameter set are adjusted according to the evaluation result; when the model reaches the first preset target on the validation set, the iteration training is stopped;

[0014] Step 7, the performance indicators of the neural network model are comprehensively evaluated using the test set, and when the model reaches the second preset target on the test set, the training and evaluation are stopped, otherwise the model is adjusted and returned to step 2.

[0015] Further, the method for initializing the original parameters of the built neural network model includes:

[0016] The original parameters of the lth layer of the neural network model with L layers are initialized by sampling from a normal distribution;

[0017] The quantum ensemble size of the lth layer of the preset neural network model and the number of quantum bits of each quantum state of the lth layer are preset;

[0018] A pure state ensemble is generated by applying a quantum gate sequence to an initial state, and a mixed state ensemble is generated combined with a quantum channel;

[0019] The release quantization coefficient is initialized to 0, and the update step is defined;

[0020] The generator is defined, the original parameters are input, and the quantized original parameters are output, and the generator parameters are randomly initialized;

[0021] Define a discriminator to determine whether the input parameters are the original parameters or the quantized original parameters, and randomly initialize the discriminator parameters.

[0022] Furthermore, methods for obtaining model output include:

[0023] Step 3.1: Extract a batch of samples from the training set and input them into the neural network model;

[0024] Step 3.2: For the l-th layer, where l = 1, ..., L, preset energy spaces corresponding to different data types, select the corresponding energy space according to the data type for quantization operation; map the original parameters to the quantum ensemble, perform quantization operation based on the quantum ensemble to obtain the quantized original parameters, and adjust the original parameters based on the released quantization coefficient to obtain the adjusted original parameters;

[0025] Step 3.3: Use the adjusted original parameters to perform forward propagation calculations for the neural network and obtain the model output.

[0026] Furthermore, the original parameters are mapped to a quantum ensemble by combining the matrix elements of the original parameters and the quantum state.

[0027] Furthermore, the method for obtaining the quantized original parameters includes:

[0028] Calculating the energy of a pure quantum state based on Hamiltonian;

[0029] The energy space is divided into K spaces; the quantized original parameters are calculated based on the correlation between the mixed quantum state and the lower limit of the k-th space and the interval of the k-th interval based on the energy space division, where k = 1, ..., K.

[0030] Furthermore, the quantization matrix obtained based on the quantized original parameters is optimized by low-rank matrix decomposition to obtain an optimized quantization matrix. Based on the released quantization coefficients, the optimized quantization matrix and the original parameters mapped to the quantum ensemble are combined to obtain the adjusted original parameters.

[0031] Furthermore, methods for calculating the total loss function of a neural network model include:

[0032] Calculate the model output loss based on the model output;

[0033] The quantum ensemble quantization loss is calculated based on the parameters reconstructed from the quantized representation;

[0034] During adversarial training, the generator loss and discriminator loss are calculated;

[0035] The total loss function is calculated based on the model output loss, quantum ensemble quantization loss, generator loss, and discriminator loss obtained through computation.

[0036] Furthermore, the gradient of the total loss function with respect to the release quantization coefficients is calculated using the chain rule;

[0037] The release quantization coefficients are updated based on the gradient of the total loss function calculated from the release quantization coefficients, resulting in the updated release quantization coefficients.

[0038] Furthermore, methods for dynamically adjusting parameters include:

[0039] Obtain a random number τ and compare it with the exploration probability. If the random number τ is less than the exploration probability, the generator randomly perturbs the quantum ensemble parameters or the generator parameters, and updates the generator loss and the corresponding gradient. If the random number τ is not less than the exploration probability, the generator is updated according to the conventional gradient descent.

[0040] Furthermore, the method for evaluating the importance of network parameters includes:

[0041] Calculate the gradient norm of the network parameters, compare the gradient norm with a preset gradient norm threshold, and discard the corresponding network parameters when the gradient norm is less than the preset gradient norm threshold.

[0042] Furthermore, when decompressing the neural network model, the data recovered from the adjusted original parameters is used as the input to the trained post-processing model to obtain the repaired original parameters as the output. The post-processing model training method includes: pre-collecting S sets of training data, which include data recovered from the adjusted original parameters and the corresponding repaired original parameters; using the data recovered from the adjusted original parameters as the input to the post-processing model; using the repaired original parameters as the output of the post-processing model; the loss function of the post-processing model is the mean square error between the original parameters and the repaired original parameters; training the post-processing model by minimizing the loss function; and using the post-processing model corresponding to the minimum loss function as the trained post-processing model.

[0043] An efficient neural network model compression system based on differentiable release quantization is implemented, including:

[0044] Data acquisition module: used to collect datasets, divide them into training, validation and test sets according to preset ratios, and normalize and standardize the datasets; the datasets include sample data and corresponding real labels;

[0045] Model initialization module: Used to build a neural network model with L layers and initialize the original parameters of the built neural network model;

[0046] Forward propagation module: used for extracting sample data from the data set and inputting into the neural network model, combining the initialized original parameters to carry out forward propagation, and obtaining the model output;

[0047] Loss calculation module: used for calculating the total loss function according to the model output and the real label corresponding to the sample data;

[0048] Parameter updating module: used for calculating the gradient of each parameter in the parameter set, including the original parameter, the quantum ensemble parameter, the release quantization coefficient, the generator parameter and the discriminator parameter, through the chain rule of the total loss function; based on the gradient of each parameter in the parameter set, the parameter set is updated; and the exploration-exploitation strategy is used for dynamic adjustment of the parameters;

[0049] Model iteration module: used for repeating the steps of the forward propagation module to the parameter updating module, continuously updating the original parameters of the model, the release quantization coefficient, and the generator parameter, and updating the exploration probability according to the preset decay rule as the training round progresses; in each iteration process, every interval of the preset time period, the gradient of the network parameters in the generator and the discriminator is calculated based on the total loss function, the importance of the network parameters is evaluated according to the gradient of the network parameters, and the network parameters are discarded according to the importance evaluation result; the performance index of the neural network model is evaluated according to the preset time period using the validation set, and the model architecture and each parameter in the parameter set are adjusted according to the evaluation result; when the model reaches the first preset target on the validation set, the iteration training is stopped.

[0050] The technical effects and advantages of the efficient neural network model compression system and method based on differentiable release quantization are as follows:

[0051] The neural network is combined with the quantum ensemble, the neural network parameters are mapped to the quantum ensemble, the rich characteristics of the quantum state are used to provide a wider parameter space and potential representation ability for the model, which helps to capture more complex data patterns and relationships, thereby improving the performance and generalization ability of the model to a certain extent; the adaptive quantization scheme based on data characteristics can reduce information loss; the calculation complexity is reduced by low-rank matrix decomposition in the quantization layer; the quantization is combined with the quantum energy, the energy characteristics of the quantum system are fully utilized to guide the quantization process, the quantum energy-based quantization can better adapt to the parameters represented by the quantum state, while preserving the performance of the model, the more efficient quantization compression is realized; the quantization parameters can be more accurately adjusted according to the gradient information in the training process, and the influence of quantization error on accuracy is reduced. In combination with the release of the quantization coefficient, the appropriate quantization strategy can be adaptively selected according to the real-time training of the model, the effect of quantization and the performance of the model are improved, the optimal quantization degree is found in the training process, so that more efficient compression is realized on the premise of ensuring the accuracy of the model; the exploration-exploitation strategy is introduced, the randomness and exploration are increased in the model training, which helps the model to jump out of the local optimal solution and find a better parameter space and model structure; the memory requirement is reduced through the model pruning technique or low-precision training. The unimportant parameters in the network are discarded, and the memory usage is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of the efficient neural network model compression method based on the differentiable release quantization of the present application;

[0053] Figure 2 A flowchart of the method for obtaining the model output of the present application;

[0054] Figure 3 A flowchart of the method for calculating the total loss function of the present application;

[0055] Figure 4 A block diagram of the efficient neural network model compression system based on the differentiable release quantization of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0057] Embodiment 1

[0058] Please refer to Figure 1As shown, the high-efficiency neural network model compression method based on differentiable release quantization described in the embodiment includes the following steps:

[0059] Step 1, used for collecting a data set, and dividing it into a training set, a validation set and a test set according to a preset ratio, and performing normalization and standardization processing on the data set; the data set includes sample data and corresponding real labels.

[0060] The collected data set is used for training, verifying and testing the neural network model, which can adjust the model structure during the model compression process. Normalization and standardization processing can make the distribution of data more stable and regular, and reduce problems such as gradient fluctuation in the training process caused by data scale differences. In the quantization training, stable gradient helps to more accurately update the quantization parameters and other parameters of the model, so that the model can better converge to a better solution in the compression process, avoiding the performance degradation or poor compression effect of the model caused by unstable training. Better training effect also helps to find better model structure and parameter configuration in the compression process, so that the compressed model can better adapt to data in a smaller scale, improving the compactness and efficiency of the model.

[0061] When processing large-scale data sets, distributed file systems such as Hadoop Distributed File System (HDFS) can be used for storage. Different nodes can perform standardization and type labeling operations on the data stored by themselves in parallel, greatly improving the data preprocessing efficiency. For example, in image data set processing, multiple computing nodes can simultaneously perform normalization, image type labeling (such as landscape, person, etc.) and other operations on different parts of the image data. After the data set is normalized and standardized, the data can be mapped to memory through memory mapping technology. In this way, in the subsequent model training process, there is no need to frequently perform disk I / O operations to read data, and data is accessed directly from memory, improving data reading speed.

[0062] Step 2, used for building a neural network model containing L layers, and performing initialization processing on the original parameters of the built neural network model;

[0063] The method for initializing the original parameters of the built neural network model includes:

[0064] The original parameters of the lth layer of the neural network model with L layers are initialized by sampling from a normal distribution N(0, σ 2 ), where σ is a constant;

[0065] The preset quantum ensemble size m l of the lth layer of the neural network model and the number of qubits n l of each quantum state of the lth layer are determined according to the following formula:

[0066] Acting on an initial state by a sequence of quantum gates Generating a pure state ensemble where |0> represents that the quantum bit is in the 0 state; is a tensor product symbol; represents an initial quantum state composed of n l |0> by tensor product, combined with quantum channels to generate a mixed state ensemble where, represents the i-th pure state quantum state; represents a set composed of m l pure state quantum states; and initialize the probability distribution is a uniform distribution where, represents the i-th mixed state quantum state; represents a set composed of m l mixed state quantum states; represents the probability of the i-th mixed state quantum state; represents a set of probability distributions corresponding to m l mixed state quantum states; by applying different sequences of quantum gates to the initial state, a plurality of different pure states can be obtained, which constitute a pure state ensemble, m l , that is, the number of pure states in the pure state ensemble, each pure state has a specific quantum state vector representation, for example, |ψ1>=ζ1|0>+ε1|1>, |ψ2>=ζ2|00>+ε2|01>+∈2|10>+ξ2|11>, wherein ζ1, ε1, ζ2, ε2, ∈2 and ξ2 are complex numbers; these pure state ensembles can be used to map with the parameters of the neural network, and the characteristics of the quantum state are used to realize the construction and training of the model. The superposition and entanglement of quantum states make the model space more abundant. By using the correlation and special properties between quantum states, a more compact way can be found to represent the model parameters, which can reduce the number of independent parameters that need to be stored and processed under the premise of ensuring the performance of the model. Combined with the efficiency of quantum computing, the model compression is realized efficiently.

[0067] Initialize the released quantization coefficient to 0, and preset the update step size;

[0068] Define the generator, input the original parameter, output the quantized original parameter, and randomly initialize the generator parameter θ G ;

[0069] Define the discriminator, which discriminates whether the input parameter is the original parameter or the quantized original parameter, and randomly initialize the parameter θ D ;

[0070] Step 3, sample data is extracted from the data set and input into the neural network model, and forward propagation is performed combined with the initialized original parameters to obtain the model output; in the neural network model training process, different layers of the neural network model can be distributed to different computing nodes for calculation. For example, in a multi-layer neural network, the first few layers are calculated by node A, the middle layers are calculated by node B, and the last few layers are calculated by node C. The nodes communicate with each other through the network to pass on intermediate results, so as to complete the entire forward propagation and back propagation process.

[0071] Referring to Figure 2 , the method for obtaining the model output comprises:

[0072] Step 3.1, extracting a batch of samples from the training set into the neural network model, wherein M is the batch size; by extracting a batch of samples from the training set, data input is provided for the model so as to perform subsequent forward propagation, loss calculation, and parameter updating operations. Batch training can improve training efficiency and also take advantage of the parallel computing capabilities of hardware such as GPUs to speed up the model training process.

[0073] Step 3.2, for the l-th layer, wherein l = 1, …, L, a preset energy space corresponding to different data types is selected according to the data type for quantization operation; the original parameter W l is mapped to a quantum ensemble, a quantization operation is performed based on the quantum ensemble to obtain a quantized original parameter, and the original parameter is adjusted based on the release quantization coefficient to obtain an adjusted original parameter; the neural network is usually composed of multiple layers, and the above operations are sequentially performed on each layer of the neural network, which can ensure that each layer of the model can be processed according to the designed process, thereby realizing the training and optimization of the entire neural network.

[0074] The original parameter W l is mapped to a quantum ensemble, a quantization operation is performed based on the quantum ensemble to obtain a quantized original parameter, and the original parameter is adjusted based on the release quantization coefficient to obtain an adjusted original parameter; the neural network is usually composed of multiple layers, and the above operations are sequentially performed on each layer of the neural network, which can ensure that each layer of the model can be processed according to the designed process, thereby realizing the training and optimization of the entire neural network.

[0075]

[0076] wherein u and v are index parameter matrix elements; is the element in the u-th row and v-th column of the original parameter matrix of the l-th layer; is a quantum state under an observable O l , tr is a trace operation; and are learnable coefficients; is a coefficient related to the i-th mixed state quantum state; is a bias term related to the row index u; the original parameter W lThe method of mapping to a quantum ensemble introduces concepts and characteristics of quantum mechanics, and provides a basis for subsequent model optimization and compression using quantum characteristics, helps the model to capture more complex data patterns and relationships, and improves the representation ability and performance of the model.

[0077] The method for obtaining the quantized original parameters comprises:

[0078] The energy of the i-th pure state quantum state of the l-th layer is calculated

[0079]

[0080] wherein H l is the Hamiltonian corresponding to the l-th layer; is the dual state of .

[0081] The energy space is divided into K energy intervals wherein, is the lower limit value of the k-th energy interval of the l-th layer energy space division; is the upper limit value of the k-th energy interval of the l-th layer energy space division; the quantized original parameters are calculated as follows:

[0082]

[0083] wherein, is the quantized original parameter; round is the rounding function; is the correlation degree of the i-th mixed state quantum state and the lower limit value of the energy interval; ΔE l is the energy interval interval of the l-th layer energy space division; the above quantization operation considers the physical characteristics of the quantum state itself, which can reduce the storage space and calculation amount of the parameters while compressing the model parameters, while maintaining the performance of the model.

[0084] The quantization matrix obtained based on the quantized original parameters is subjected to low-rank matrix decomposition to obtain an optimized quantization matrix.

[0085] The method for obtaining the optimized quantization matrix comprises:

[0086] The quantization matrix LH with a dimension of m*n is approximately expressed in the form of the product of two matrices, i.e. LH≈UV T , wherein U is an m*r matrix; V is an r*n matrix, and r is a preset low-rank value, r satisfies r solving U and V, wherein ||·||F F is the Frobenius norm; in the calculation process, the SVD solution of the quantization matrix LH is first performed, i.e. LH=A∑BT wherein A is an orthogonal matrix with dimension m x m; ∑ is a diagonal matrix with dimension m x n; B is an orthogonal matrix with dimension n x n; T is the transpose of a matrix; all diagonal elements in the diagonal matrix ∑ are obtained, arranged in descending order, and the first r diagonal elements are taken as singular values, the first r columns in the diagonal matrix A are selected to form the matrix U, the first r columns in the diagonal matrix B are selected to form the matrix V, and the product of the matrix U and the transpose matrix of the matrix V is taken as an optimized quantization matrix, and the subsequent operation is performed by using the optimized quantization matrix instead of the quantization matrix LH.

[0087] obtaining an adjusted original parameter The method comprises the following steps:

[0088]

[0089] wherein a is a release quantization coefficient;

[0090] The release quantization coefficient can control the degree of influence of quantization on the model, and by adjusting the value of the release quantization coefficient, a balance between model compression and model performance can be found to achieve better training effect and compression efficiency.

[0091] Step 3.3, using the adjusted original parameter to perform forward propagation calculation of the neural network, and obtaining model output Using the parameter adjusted by the release quantization coefficient to perform forward propagation, that is, starting from the input layer, sequentially passing through the calculation of each layer according to the structure and calculation rules of the neural network, and finally obtaining the output of the model to the sample data. This is one of the core steps of neural network training and prediction. Through forward propagation, the prediction result of the model to the input data under the current parameters can be calculated, which provides a basis for subsequent loss calculation and parameter update, so as to continuously optimize the model, make it better fit the training data, and improve the accuracy and generalization ability of the model.

[0092] In the above method of obtaining model output, the training data can be divided into multiple subsets, and each computing node is responsible for the training of a subset. In order to improve the training speed of the neural network model. In the process of model training, the number of parameters of the model is large. Similarly, the memory mapping technology can be used to store the model parameters in the file on the disk and map to the memory. In this way, when updating parameters and performing forward and backward propagation calculations, parameters can be directly operated in memory, reducing memory occupation, while quickly accessing parameters. The intermediate results generated in the process of model training (such as the output of the intermediate layer, the gradient, etc.) can also be stored and accessed using the memory mapping technology. This helps different nodes to more efficiently share and transfer intermediate results in a distributed computing environment, while reducing memory pressure.

[0093] Step 4, calculate the total loss function according to the model output and the real label corresponding to the sample data;

[0094] With reference to Figure 3 , the method for calculating the total loss function comprises:

[0095] Step 4.1, calculate the model output loss L task ;

[0096]

[0097] Wherein, C is the number of categories; is the probability of the cth category in the model output ; y j (c) is the real label corresponding to the model output The probability of the cth category in the model output loss can measure the difference between the model output and the real label, so as to make the model learn the correct classification decision boundary, so that the probability distribution of the model output is as close as possible to the distribution of the real label, thereby improving the accuracy of the model.

[0098] Step 4.2, calculate the quantum ensemble quantization loss L q :

[0099]

[0100] Wherein, is the approximate parameter restored from the adjusted original parameter;

[0101] The quantum ensemble quantization loss can ensure that the quantization process of the quantum ensemble is as accurate as possible. By minimizing the difference between the original parameter and the approximate parameter restored after quantization, the quantization operation can be constrained to compress the model parameters while minimizing the loss of too much information, maintaining the accuracy and effectiveness of the model parameters, and avoiding a significant decline in model performance due to quantization.

[0102] Step 4.3, calculate the generator loss and discriminator loss in the adversarial training process;

[0103] The generator loss L G is calculated as follows:

[0104]

[0105] Wherein, is the expectation of the original parameter W l ; G(W l ) is the output generated by the generator after processing the original parameter W l ; D(G(W l )) is the data generated by the generator G(W lthe discriminative probability of the discriminator; by minimizing the generator loss, the generator can continuously learn and improve to generate data that better conforms to the real data distribution, thereby mutually promoting the discriminator in the adversarial training, improving the overall performance and generalization ability of the model, and possibly helping to generate better quantization parameters or quantum state representations, further optimizing the model compression effect.

[0106] The discriminator loss L D is calculated as follows:

[0107]

[0108] where D(W l ) is the discriminative probability of the discriminator for the real original parameter W l ; by minimizing the discriminator loss, the discriminator is trained to accurately distinguish between real data and generated data, thereby providing effective feedback to the generator to guide the generator to continuously improve. This adversarial training mechanism can enhance the learning ability and robustness of the model, enabling the model to better capture the characteristics and distribution of the data when processing complex data and performing model compression, improving the performance and adaptability of the model.

[0109] Step 4.4, the total loss function is calculated as follows:

[0110] L total = L task + β1×L q + β2(L G + L D );

[0111] where β1 and β2 are balance weights. The total loss combines the task loss, quantum ensemble quantization loss, and loss in the adversarial training process, and by adjusting the balance weights β1 and β2, the relative importance of different loss terms in the total loss can be controlled. In this way, the task accuracy, quantum ensemble quantization accuracy, and adversarial training effect can be considered simultaneously during model training, enabling the model to balance various requirements during optimization, achieving efficient neural network model training and compression, while achieving a good balance and optimization in terms of task performance, model compression, and generalization ability.

[0112] Step 5, calculate the gradient of the total loss function with respect to each parameter in the parameter set, including the original parameter, the quantum ensemble parameter, the release quantization coefficient, the generator parameter and the discriminator parameter, by the chain rule; based on the gradient of the total loss function with respect to each parameter in the parameter set, update each parameter in the parameter set; and explore-exploit strategy is used for dynamic adjustment of the parameters; the chain rule is an important method for calculating the derivative of a composite function. In neural network training, the total loss function is a complex function of model parameters (including original parameters, quantum ensemble parameters, release quantization coefficients, generator parameters and discriminator parameters, etc.). By the chain rule, the gradient of the total loss function with respect to a certain parameter (such as the release quantization coefficient) can be decomposed into the product of multiple intermediate derivatives, so that the gradient of the parameter can be accurately calculated.

[0113] The method comprises:

[0114]

[0115] The method comprises: updating the release quantization coefficient α to obtain an updated release quantization coefficient;

[0116] The method for obtaining the updated release quantization coefficient comprises:

[0117]

[0118] wherein α' is the updated release quantization coefficient; η α is the update step; other parameters (original parameters, quantum ensemble parameters, generator parameters and discriminator parameters, etc.) are also updated in a similar manner, and through continuous iteration of this process, the model can gradually learn better parameter values, so as to better fit the data, achieve effective model compression (through quantization, etc.) and improve the performance of the generator and the discriminator in the adversarial training.

[0119] The method for dynamically adjusting the parameters comprises:

[0120] obtain a random number τ, compare the random number τ with a preset exploration probability P explore , if the random number τ is less than the exploration probability P explore , then the generator randomly disturbs the quantum ensemble parameter or the generator parameter, and updates L G ​and the corresponding gradient; when the exploration condition is met, the generator is randomly disturbed in its parameters (quantum ensemble parameters or self-parameters), which helps the generator jump out of the current possible local optimal solution, explore a wider parameter space, and find a possibly better model structure or parameter configuration, thereby increasing the diversity and generalization ability of the model. At the same time, the generator loss and the corresponding gradient are updated so that subsequent parameter updates can be performed according to the new gradient information. If the random number τ is not less than the exploration probability P explore , the generator is updated by the conventional gradient descent. When the exploration condition is not met, the generator parameters are updated according to the conventional gradient descent method, i.e., based on the previously calculated gradient information, the update is performed in the direction that reduces the loss function. This is to utilize the current available information and the learning achievements of the model to gradually optimize the generator, so that it continues to move in the current optimal direction and improves the quality of the data generated by the generator and the ability to confront the discriminator.

[0121] Step 6, repeat the iteration of steps 3-5 to continuously update the original parameters of the model, the release of the quantization coefficients, and the generator parameters, and update the exploration probability according to the preset decay rule as the training round progresses; in each iteration process, every interval of a preset time period, the gradients of the network parameters in the generator and the discriminator are calculated based on the total loss function, the importance of the network parameters is evaluated according to the gradients of the network parameters, and the network parameters are discarded according to the importance evaluation results;

[0122] The method for evaluating the importance of the network parameters includes:

[0123] The gradient norm of the network parameters is calculated, and the gradient norm is compared with a preset gradient norm threshold value; when the gradient norm is less than the preset gradient norm threshold value, the corresponding network parameter is discarded.

[0124] The performance indicators (such as accuracy, recall rate, and F1 value, etc.) of the neural network model are evaluated according to the validation set at a preset time period, and the model architecture and each parameter in the parameter set are adjusted according to the evaluation results; when the model reaches the first preset target (such as the accuracy is above 95% and does not improve for 5 consecutive iterations) on the validation set, the iteration training is stopped;

[0125] Step 7, the performance indicators (such as accuracy, stability, and generalization, etc.) of the neural network model are evaluated using the test set, and when the model reaches the second preset target on the test set, it indicates that the model training is successful, and the training and evaluation can be stopped, and the subsequent deployment and application link is entered; if it does not meet the standard, the training process needs to be analyzed, the problem source is found out, the model or the training strategy is adjusted, and the training is returned to step 2.

[0126] Step 8, when decompressing the neural network model, the data recovered from the adjusted original parameters is taken as the input of the trained post-processing model, and the output of the repaired original parameters is obtained, the post-processing model training method comprising: pre-collecting S groups of training data, the training data including data recovered from the adjusted original parameters and corresponding repaired original parameters, taking the data recovered from the adjusted original parameters as the input of the post-processing model, taking the repaired original parameters as the output of the post-processing model, the loss function of the post-processing model being the mean square error of the original parameters and the repaired original parameters, and the post-processing model being trained by minimizing the loss function so as to learn the repair method of quantization distortion, thereby improving the quality of the recovered data. The post-processing model corresponding to the minimum loss function is taken as the trained post-processing model.

[0127] Embodiment 2

[0128] Please refer to Figure 4 The high-efficiency neural network model compression system based on differentiable release quantization described in the embodiment includes:

[0129] The data acquisition module is used to collect the data set, divide it into a training set, a validation set and a test set according to a preset ratio, and normalize and standardize the data set; the data set includes sample data and corresponding true labels.

[0130] The model initialization module is used to build a neural network model containing L layers, and to initialize the original parameters of the built neural network model.

[0131] The forward propagation module is used to extract sample data from the data set and input it into the neural network model, combine the initialized original parameters to perform forward propagation, and obtain the model output.

[0132] The loss calculation module is used to calculate the total loss function according to the model output and the true labels corresponding to the sample data.

[0133] The parameter update module is used to calculate the gradient of each parameter in the parameter set, including the original parameters, the quantum ensemble parameters, the release quantization coefficients, the generator parameters and the discriminator parameters, with respect to the total loss function by the chain rule, to perform directional update of each parameter in the parameter set based on the gradient of each parameter in the parameter set with respect to the total loss function, and to perform dynamic adjustment of the parameters by the exploration-exploitation strategy.

[0134] The model iteration module is configured to repeat the steps of the forward propagation module to the parameter updating module, continuously update the original parameters of the model, release the quantization coefficients, update the generator parameters, and update the exploration probability according to a preset decay rule as the training round progresses, evaluate the performance indicators of the neural network model according to a preset period using a validation set, and adjust the model architecture and each parameter in the parameter set according to the evaluation results, and stop the iterative training when the model reaches a first preset target on the validation set.

[0135] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0136] Finally, the above merely describes preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An efficient neural network model compression method based on differentiable release quantization, characterized in that, The method comprises the following steps: Step 1, collecting an image data set, dividing the image data set into a training set, a validation set and a test set according to a preset ratio, and performing normalization and standardization processing on the image data set; the image data set comprises sample image data and corresponding true labels; Step 2, build a neural network model comprising layers, and initialize the original parameters of the built neural network model; Step 3, extracting sample image data from the image data set and inputting the sample image data into the neural network model, performing forward propagation in combination with the initialized original parameters to obtain an image recognition result output by the model; Step 4, calculating a total loss function according to the image recognition result output by the model and the true labels corresponding to the sample image data; Step 5, calculating the gradient of the total loss function with respect to each parameter in the parameter set, including the original parameters, the quantum ensemble parameters, the release quantization coefficient, the generator parameters and the discriminator parameters, based on the chain rule; performing directional updating of each parameter in the parameter set based on the gradient of the total loss function with respect to each parameter in the parameter set; and performing dynamic adjustment of the parameters through an exploration-exploitation strategy; Step 6, repeating steps 3-5 to continuously update the original parameters of the model, the release quantization coefficient, and the generator parameters, and updating the exploration probability according to a preset decay rule as the training round progresses; In each iteration process, every interval of a preset time period, the gradient of the network parameters in the generator and the discriminator is calculated based on the total loss function, the importance of the network parameters is evaluated according to the gradient of the network parameters, and the network parameters are discarded according to the importance evaluation result; The performance indicators of the neural network model are evaluated using the validation set according to a preset time period, and the model architecture and each parameter in the parameter set are adjusted according to the evaluation result; when the model reaches a first preset target on the validation set, the iteration training is stopped; Step 7, using the test set to comprehensively evaluate the performance indicators of the neural network model, and stopping training and evaluation when the model reaches a second preset target on the test set, otherwise adjusting the model and returning to step 2.

2. The high efficient neural network model compression method based on differentiable release quantization according to claim 1, characterized in that, The method for initializing the original parameters of the built neural network model comprises: For the neural network model with the first layer of the original parameters are initialized in a manner sampled from a normal distribution; the first preset neural network model the size of the quantum ensemble of the first layer and the number of qubits of each quantum state of the first layer the size of the quantum ensemble of the first layer and the number of qubits of each quantum state of the first layer Generating a pure state ensemble by acting on an initial state through a quantum gate sequence, and generating a mixed state ensemble through a quantum channel; Initializing the release quantization coefficient to 0 and defining an update step; Defining a generator that inputs original parameters and outputs quantized original parameters, and randomly initializing the generator parameters; Defining a discriminator that discriminates whether the input parameters are original parameters or quantized original parameters, and randomly initializing the discriminator parameters.

3. The high efficient neural network model compression method based on differentiable release quantization according to claim 2, characterized in that, The method for obtaining the model output comprises: Step 3.1, extracting a batch of samples from the training set and inputting them into the neural network model; Step 3.2, for the first layer, wherein, , preset energy spaces corresponding to different data types, selecting a corresponding energy space according to the data type to perform a quantization operation; mapping the original parameter to a quantum ensemble, performing a quantization operation based on the quantum ensemble to obtain a quantized original parameter, and adjusting the original parameter based on a release quantization coefficient to obtain an adjusted original parameter; Step 3.3, performing forward propagation calculation of the neural network using the adjusted original parameters to obtain the model output.

4. The high-efficiency neural network model compression method based on differentiable release quantization according to claim 3, characterized in that, Mapping the original parameters to a quantum ensemble in combination with the matrix elements of the original parameters and quantum states.

5. The method of claim 4, wherein, The method for obtaining the quantized original parameters comprises: Calculating the energy of a pure state quantum state based on a Hamiltonian; dividing the energy space into K spaces; obtaining a quantized original parameter based on a correlation degree of the mixed state quantum state and a lower limit value of the Kth space and a calculation based on an interval spacing of the Kth space obtained by the energy space division, wherein the Kth space is the space with the highest energy value in the energy space. the Kth space is the space with the highest energy value in the energy space. the Kth space is the space with the highest energy value in the energy space.

6. The method of claim 5, wherein, Optimizing the quantization matrix obtained based on the quantized original parameters through low-rank matrix decomposition to obtain an optimized quantization matrix, and obtaining adjusted original parameters based on the release quantization coefficient, in combination with the optimized quantization matrix and the original parameters mapped into the quantum ensemble.

7. The high efficient neural network model compression method based on differentiable release quantization according to claim 6, characterized in that, The method for calculating the total loss function of the neural network model comprises: calculating a model output loss based on the model output; calculating a quantum ensemble quantization loss based on the restored parameters from the quantized representation; calculating a generator loss and a discriminator loss in the adversarial training process; calculating a total loss function based on the calculated model output loss, quantum ensemble quantization loss, generator loss and discriminator loss.

8. The high efficient neural network model compression method based on differentiable release quantization according to claim 7, characterized in that, calculating the gradient of the total loss function with respect to the released quantization coefficient by the chain rule; updating the released quantization coefficient based on the calculated gradient of the total loss function with respect to the released quantization coefficient to obtain an updated released quantization coefficient.

9. The method of claim 8, wherein, The method for performing dynamic adjustment of the parameters comprises: Obtaining a random number , the random number is compared with an exploration probability, if the random number is less than the exploration probability, the generator randomly perturbs the quantum ensemble parameter or the generator parameter, the generator loss and the corresponding gradient are updated; if the random number is not less than the exploration probability, the generator is updated according to the normal gradient descent.

10. The method of claim 9, wherein, The method for evaluating the importance of the network parameters comprises: calculating the gradient norm of the network parameters, comparing the gradient norm with a preset gradient norm threshold, and discarding the corresponding network parameters when the gradient norm is less than the preset gradient norm threshold.

11. The method of claim 10, wherein, When the neural network model is decompressed, the data restored from the adjusted original parameters is used as the input of the trained post-processing model to obtain the output of the repaired original parameters, and the post-processing model training method comprises: pre-collecting S groups of training data, the training data comprising data restored from the adjusted original parameters and corresponding repaired original parameters, using the data restored from the adjusted original parameters as the input of the post-processing model, using the repaired original parameters as the output of the post-processing model, the loss function of the post-processing model being the mean square error of the original parameters and the repaired original parameters, and training the post-processing model by minimizing the loss function, and the post-processing model corresponding to the minimum loss function being used as the trained post-processing model.

12. A high-efficiency neural network model compression system based on differentiable release quantization, implementing the high-efficiency neural network model compression method based on differentiable release quantization according to any one of claims 1-11, characterized in that, comprise: a data acquisition module for acquiring an image dataset, dividing the image dataset into a training set, a validation set and a test set according to a preset ratio, and performing normalization and standardization processing on the image dataset; the image dataset comprises sample image data and corresponding true labels; The model initialization module is configured to build a neural network model comprising layers, and to initialize original parameters of the built neural network model. a forward propagation module for extracting sample image data from the image dataset and inputting the sample image data into the neural network model, performing forward propagation in combination with the initialized original parameters, and obtaining image recognition results output by the model; a loss calculation module for calculating a total loss function according to the image recognition results output by the model and the true labels corresponding to the sample image data; a parameter updating module for calculating the gradient of each parameter in a parameter set with respect to the total loss function by the chain rule, the parameter set comprising the original parameters, the quantum ensemble parameters, the released quantization coefficient, the generator parameters and the discriminator parameters, performing directional updating of each parameter in the parameter set based on the gradient of each parameter in the parameter set with respect to the total loss function, and performing dynamic adjustment of the parameters by an exploration-exploitation strategy; a model iteration module for repeatedly iterating the steps of the forward propagation module to the parameter updating module, continuously updating the original parameters of the model, the released quantization coefficient, and the generator parameters, and updating the exploration probability according to a preset decay rule as the training round progresses; In each iteration process, the gradient of the network parameters in the generator and the discriminator is calculated based on the total loss function every preset time interval, the importance of the network parameters is evaluated according to the gradient of the network parameters, and the network parameters are discarded according to the importance evaluation result; The performance indicators of the neural network model are evaluated according to the preset period using the validation set, and the model architecture and each parameter in the parameter set are adjusted according to the evaluation result; the iteration training is stopped when the model reaches the first preset target on the validation set.

Images