Deep neural network parameter dynamic quantization compression method and system

Abstract

The invention provides a deep neural network parameter dynamic quantization compression method and system, belongs to the technical field of parameter quantization, and aims to solve the problem that in the prior art, division is performed based on values of parameters, and the effect of the parameters with small values in a network is ignored. Comprising the steps of 1, establishing a cascade failure model corresponding to a deep neural network model based on a graph neural network; 2, initializing the cascade failure model, and selecting important nodes in the deep neural network model based on the initialized cascade failure model; and step 3, calculating the dispersion degree f of the parameters of the important nodes, and selecting different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

Description

Technical Field

[0001] The present invention belongs to the technical field of parameter quantization, and specifically relates to a method and system for dynamic quantization compression of deep neural network parameters. Background Art

[0002] The main purpose of parameter quantization is to reduce the storage space occupied by the model and then accelerate the calculation through special hardware. There are currently two mainstream quantization methods: weight sharing and low-bit model parameter quantization. HashedNets proposed by Chen et al. maps the weight parameters of the model to multiple hash buckets through a hash function. The parameters in each hash bucket share a weight value. This method greatly reduces the number of unique parameters that need to be stored, thereby significantly reducing the storage requirements of the model. Han et al. proposed a k-means clustering quantization method, which first performs K-means clustering on the model parameters, and each cluster center represents a set of weight parameters. After clustering is completed, the parameters are quantized to the nearest cluster center. Then Huffman coding is used to compress the quantized weights to further reduce the storage size. As an improvement on the traditional K-means algorithm, Sculley proposed the Mini-batch K-means algorithm, which accelerates the convergence of the algorithm by using a small batch of data to update the center point each time. The new algorithm has made modifications to the center point update process to make it more suitable for processing large-scale data.

[0003] However, the above methods are all based on the value of the parameter, ignoring the role of small parameters in the network. In addition, the division range is based on a certain layer in the network, without considering the connection between layers. The binarization and ternary methods, which have good compression effects, are not enough to express complex models. Summary of the invention

[0004] In view of this, the present invention provides a method and system for dynamic quantization compression of deep neural network parameters to solve the problem that the existing methods are based on the value of parameters, ignoring the role of small parameters in the network. In addition, the division range is based on a certain layer in the network, without considering the connection between layers.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A method for dynamic quantization and compression of deep neural network parameters, characterized by comprising:

[0007] Step 1: Establish a cascading failure model corresponding to the deep neural network model based on the graph neural network;

[0008] Step 1 specifically includes the following steps:

[0009] Step 1.1: In the initial state, all nodes remain normal, the initial load of node vi is Di(0), and the capacity is Ci;

[0010] Step 1.2: invalidate the node vi, i.e. delete the node;

[0011] Step 1.3: Get a set of nodes within X adjacent to the failed node vi, and distribute the load to its adjacent nodes vj according to the load redistribution strategy;

[0012] Step 1.4: Determine whether the adjacent node vj is invalid. If the load exceeds the threshold of the adjacent node vj after load redistribution, a new invalid node vj is generated. At this time, return to step 1.3 to redistribute the load of the new invalid node. If it does not exceed, go to step 5;

[0013] Step 1.5: When no new failed nodes are generated, the network reliability index is calculated and the simulation ends.

[0014] Step 2: Initialize the cascading failure model, and select important nodes in the deep neural network model based on the initialized cascading failure model;

[0015] In step 2, the cascading failure model is initialized, specifically including:

[0016] Step A: Set the weight of the node in the model to a, as shown below:

[0017]

[0018] Where v is the parameter value of the neuron corresponding to the node in the model, g is the gradient value during training, k and b are constants, and n is the total number of training times;

[0019] Step B: Based on the weight a, calculate the attention score between the nodes corresponding to each edge in the model, as shown in the following formula:

[0020]

[0021] Where g is the gradient value, n is the total number of training times, h is the feature space after node mapping, || represents the vector concatenation operation, i and j represent nodes i and j respectively, and c is the weight of the edge in the model;

[0022] Step C: When the faulty node propagates the crisis, the weights of the nodes related to the faulty node will decay. The directly adjacent nodes will be more affected, and the indirectly adjacent nodes will be less affected. When the weight of the node is zero, the node also becomes a faulty node. The state transition formula is as follows:

[0023]

[0024] Where m represents the distance between nodes. When deta il is equal to 0, the node becomes a fault node.

[0025] In step 2, selecting important nodes in the deep neural network model based on the initialized cascading failure model specifically includes:

[0026] Select d nodes as faulty nodes, select adjacent nodes in random order for crisis propagation, and the propagation range is the nodes within 5 of the faulty node. The detail value is continuously updated, and finally the proportion of nodes below the threshold t is counted.

[0027] Step 3: Calculate the dispersion degree f of the parameters of important nodes, and select different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

[0028] In step 3, the parameters are quantified, which specifically includes the following steps:

[0029] Step 3.1: Select the quantization strategy, which is divided into post-quantization and quantization-aware training;

[0030] Step 3.2: During post-quantization, a set of calibration data is used to estimate the distribution of model weights and activations;

[0031] Step 3.3: Quantize each weight value according to the selected quantization strategy and representation, calculate the scaling factor and zero point by calibrating the activation range, and then map the floating-point value to a fixed-point number.

[0032] Step 3.5: Quantize activations. Similar to quantizing weights, quantize activations. Activation quantization can be done dynamically at runtime or predetermined using calibration data. It is important to note the differences in activation distributions between layers in order to accurately choose quantization parameters.

[0033] Step 3.6: Retraining. The model is retrained to adapt to the errors introduced by quantization. The model uses quantized weights and activations during forward propagation, and still uses floating point precision to calculate gradients during backpropagation.

[0034] Step 3.7: Accuracy evaluation. After quantization is completed, the quantized model is evaluated to detect the impact of quantization on model accuracy. The performance on different tasks and datasets can be evaluated by comparing with the original floating-point model.

[0035] Step 3.8: Optimization and debugging: If the accuracy of the model drops significantly after quantization, you may need to adjust the quantization parameters, choose a different quantization strategy, or add quantization-aware training steps.

[0036] A deep neural network parameter dynamic quantization compression system, comprising:

[0037] Model building module: Building a cascading failure model corresponding to the deep neural network model based on the graph neural network;

[0038] Initialization module: initialize the cascading failure model and select important nodes in the deep neural network model based on the initialized cascading failure model;

[0039] Quantization module: Calculate the dispersion degree f of the parameters of important nodes, and select different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

[0040] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0041] 1. Compared with the traditional quantization method for parameter values, the present invention fully considers the relationship between neurons in the network, and the selected parameters play a greater role in the network.

[0042] 2. In the present invention, the quantization range is expanded from local to global, that is, all parameters of a certain layer in the network are quantized separately, and then the whole network is quantized once, which fully considers the connection between network layers.

[0043] 3. In the present invention, the quantization parameter width is dynamically selected to perform targeted compression on different problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0045] Figure 1 A diagram of the parameter screening process in the deep neural network of the present invention;

[0046] Figure 2 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0052] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0053] Example

[0054] like Figure 1-Figure 2 As shown, an embodiment of the present invention discloses a method for dynamic quantization and compression of deep neural network parameters, which is characterized by comprising:

[0055] Step 1: Establish a cascading failure model corresponding to the deep neural network model based on the graph neural network;

[0056] Step 1 specifically includes the following steps:

[0057] Step 1.1: In the initial state, all nodes remain normal, the initial load of node vi is Di(0), and the capacity is Ci;

[0058] Step 1.2: invalidate the node vi, i.e. delete the node;

[0059] Step 1.3: Get a set of nodes within X adjacent to the failed node vi, and distribute the load to its adjacent nodes vj according to the load redistribution strategy;

[0060] Step 1.4: Determine whether the adjacent node vj is invalid. If the load exceeds the threshold of the adjacent node vj after load redistribution, a new invalid node vj is generated. At this time, return to step 1.3 to redistribute the load of the new invalid node. If it does not exceed, go to step 5;

[0061] Step 1.5: When no new failed nodes are generated, the network reliability index is calculated and the simulation ends.

[0062] Step 2: Initialize the cascading failure model, and select important nodes in the deep neural network model based on the initialized cascading failure model;

[0063] In step 2, the cascading failure model is initialized, specifically including:

[0064] Step A: Set the weight of the node in the model to a, as shown below:

[0065]

[0066] Where v is the parameter value of the neuron corresponding to the node in the model, g is the gradient value during training, k and b are constants, and n is the total number of training times;

[0067] Step B: Based on the weight a, calculate the attention score between the nodes corresponding to each edge in the model, as shown in the following formula:

[0068]

[0069] Where g is the gradient value, n is the total number of training times, h is the feature space after node mapping, || represents the vector concatenation operation, i and j represent nodes i and j respectively, and c is the weight of the edge in the model;

[0070] Step C: When the faulty node propagates the crisis, the weights of the nodes related to the faulty node will decay. The directly adjacent nodes will be more affected, and the indirectly adjacent nodes will be less affected. When the weight of the node is zero, the node also becomes a faulty node. The state transition formula is as follows:

[0071]

[0072] Where m represents the distance between nodes. When deta il is equal to 0, the node becomes a fault node.

[0073] In step 2, selecting important nodes in the deep neural network model based on the initialized cascading failure model specifically includes:

[0074] Select d nodes as faulty nodes, select adjacent nodes in random order for crisis propagation, and the propagation range is the nodes within 5 of the faulty node. The detail value is continuously updated, and finally the proportion of nodes below the threshold t is counted.

[0075] Step 3: Calculate the dispersion degree f of the parameters of important nodes, and select different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

[0076] In step 3, the parameters are quantized. Taking ResNet-50 as an example, the data set is ImageNet, which specifically includes the following steps:

[0077] Step 3.1: According to which layer of the neural network model the important nodes selected in step 2 correspond to, classify the important nodes, calculate the maximum and minimum values ​​of each layer, and then take the mean u of the maximum values ​​of all layers as the activation upper limit, and the mean l of the minimum values ​​of all layers as the activation lower limit. The activation upper limit u = 6.8, the activation lower limit l = -6.2;

[0078] Step 3.2: Calculate the dispersion degree f of all activated weight values, and select the bit width of the quantized parameter based on the value of f. The value of f is between 0.3-0.5, so a 4-bit quantization scheme is selected. The first and last layers use 8-bit quantization to avoid excessive information loss;

[0079] Step 3.3: Determine the proportional factor based on the activation upper and lower limits u and l in step 3.1, and then determine the zero point based on the value of f. The proportional factor and zero point offset are calculated to be 0.15 and 85 respectively.

[0080] Finally, the model size before quantization is 98M, and after quantization it is 25M, which is 25.5% of the original model. After quantization, the Top-1 accuracy only drops by 0.6% to 75.8%.

[0081] A deep neural network parameter dynamic quantization compression system, comprising:

[0082] Model building module: Building a cascading failure model corresponding to the deep neural network model based on the graph neural network;

[0083] Initialization module: initialize the cascading failure model and select important nodes in the deep neural network model based on the initialized cascading failure model;

[0084] Quantization module: Calculate the dispersion degree f of the parameters of important nodes, and select different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

[0085] The circuits, electronic components and modules involved are all prior art and can be fully implemented by those skilled in the art. Needless to say, the content protected by the present invention does not involve improvements to software and methods.

[0086] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0087] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dynamic quantization and compression of deep neural network parameters, characterized in that: include: Step 1: Establish a cascading failure model corresponding to the deep neural network model based on the graph neural network; Step 2: Initialize the cascading failure model, and select important nodes in the deep neural network model based on the initialized cascading failure model; Step 3: Calculate the dispersion degree f of the parameters of important nodes, and select different quantization widths to quantize the parameters according to the interval to which the calculated f value belongs.

2. A method for dynamic quantization and compression of deep neural network parameters according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1.1: In the initial state, all nodes in the cascading failure model remain normal, the initial load of node vi is Di(0), and the capacity is Ci; Step 1.2: invalidate the node vi, i.e. delete the node; Step 1.3: Get a set of nodes within X adjacent to the failed node vi, and distribute the load to the adjacent nodes vj of vi according to the load redistribution strategy; Step 1.4: Determine whether the adjacent node vj is invalid. If the load exceeds the threshold of the adjacent node vj after load redistribution, a new invalid node vj is generated. At this time, return to step 1.3 to redistribute the load of the new invalid node. If it does not exceed, go to step 1.5; Step 1.5: When no new failed nodes are generated, the network reliability index is calculated and the simulation ends.

3. A method for dynamic quantization and compression of deep neural network parameters according to claim 1, characterized in that: In step 2, the cascading failure model is initialized, specifically including: Step A: Set the weight of the node in the cascading failure model to a, as shown in the following formula: Where v is the parameter value of the neuron corresponding to the node in the model, g is the gradient value during training, k and b are constants, and n is the total number of training times; Step B: Based on the weight a, calculate the attention score between the nodes corresponding to each edge in the model, as shown in the following formula: Where g is the gradient value, n is the total number of training times, h is the feature space after node mapping, || represents the vector concatenation operation, i and j represent nodes i and j respectively, and c is the weight of the edge in the model; Step C: When the faulty node propagates the crisis, the weights of the nodes related to the faulty node will decay. The directly adjacent nodes will be more affected, and the indirectly adjacent nodes will be less affected. When the weight of the node is zero, the node also becomes a faulty node. The state transition formula Delta is as follows: Where m represents the distance between nodes. When Delta is less than or equal to 0, the node becomes a faulty node.

4. A method for dynamic quantization and compression of deep neural network parameters according to claim 1, characterized in that: In step 2, selecting important nodes in the deep neural network model based on the initialized cascading failure model specifically includes: Select d nodes as faulty nodes, and select adjacent nodes in random order for crisis propagation. The propagation range is the nodes within 5 of the faulty node. The Δ value is continuously updated, and finally the proportion of nodes below the threshold t is counted.

5. A method for dynamic quantization and compression of deep neural network parameters according to claim 1, characterized in that: In step 3, the parameters are quantized by post-training quantization, which specifically includes the following steps: Step 3.1: According to which layer of the neural network model the important nodes selected in step 2 correspond to, classify the important nodes, calculate the maximum and minimum values ​​of each layer respectively, and then take the mean u of the maximum values ​​of all layers as the activation upper limit of the parameter, and the mean l of the minimum values ​​of all layers as the activation lower limit of the parameter; Step 3.2: Calculate the dispersion degree f of the activated weight values ​​of all parameters, and select the quantization strategy according to the value of f; Step 3.3: Determine the proportional factor based on the activation upper and lower limits u and l in step 3.1, and then determine the zero point based on the value of f, and then perform quantization.

6. A deep neural network parameter dynamic quantization compression system, characterized in that: include: Model building module: Building a cascading failure model corresponding to the deep neural network model based on the graph neural network; Initialization module: initialize the cascading failure model and select important nodes in the deep neural network model based on the initialized cascading failure model; Quantization module: Calculate the dispersion degree f of the parameters of important nodes, and select different widths in the parameter quantization process according to the interval to which the calculated f value belongs.

Images