Inference acceleration optimization method based on quantization neural network weight reuse

By generating data flow diagrams in deep neural networks and optimizing the calculation diagrams, the problem of low weight repetitive computing efficiency on general-purpose processors is solved, and the effect of improving the DNN computing speed without relying on customized hardware accelerators is achieved.

CN119047571APending Publication Date: 2024-11-29ZHEJIANG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410994001.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, deep neural networks (DNNs) fail to effectively utilize the repetitive characteristics of quantized weights on general-purpose processors, resulting in inefficient computing efficiency and lack of methods to achieve optimization without relying on customized hardware accelerators.

Method used

By generating data flow graphs in the offline stage, optimizing the calculation process, using data flow graphs to perform inference operations in the online stage, converting irregular indexes of repeated weights into node connection relationships, reducing calculation irregularities, and creating optimized calculation graphs and splits Common subformats, reduce addition and multiplication operations.

Benefits of technology

On a general-purpose processor, it reduces the number of calculations and the number of weight data readings, improves the computing speed of neural networks, is universal and practical, and is suitable for resource-constrained environments such as cloud computing and mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047571A_ABST
    Figure CN119047571A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of deep learning, and discloses a reasoning acceleration optimization method based on quantization neural network weight reuse, which comprises an offline stage and an online stage. In the off-line stage, a data flow diagram is generated, and the data flow diagram optimizes the calculation process by multiplexing repeated sub-calculation in the calculation process; in the online stage, the data flow diagram is used for executing reasoning operation, the reasoning process is modeled into traversal of the data flow diagram, irregular indexes with repeated weights are converted into connection relations of nodes in the data flow diagram when the data flow diagram is constructed, and the problem of calculation irregularity is solved. According to the technology, the calculation times of addition and multiplication can be reduced, and the reading times of weight data are reduced, so that the calculation speed of the neural network is improved. Compared with a technical scheme of customizing a hardware accelerator, the method can realize benefits on a general processor without depending on the hardware accelerator, and has practicability and universality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to an inference acceleration optimization method based on quantized neural network weight reuse. Background Art

[0002] With the development of artificial intelligence, deep neural networks (DNNs) have been widely used due to their excellent performance in many fields such as image recognition, speech processing, and natural language processing. Improving the efficiency of DNN reasoning is particularly important for real-time applications and resource-constrained environments. In the existing technology, DNN quantization schemes are used to reduce storage requirements, but quantization causes the same weight value to be repeated in the convolution kernel. This phenomenon is called weight repetition. Weight repetition will lead to many repeated calculations. Using weight repetition can reduce the amount of calculation in the reasoning process, but it also introduces irregularity in the calculation.

[0003] On general-purpose processors, neural network calculations often fail to take into account the repetitive nature of quantized weights, and the calculation process involves many repetitive calculations, resulting in low computational efficiency. Although some technologies recognize the repetitive nature of quantized neural network weights, they require the use of custom hardware accelerators to achieve net benefits, which limits the practicality of weight repetition optimization techniques on general-purpose processors. Summary of the invention

[0004] The purpose of the present invention is to provide an inference acceleration optimization method based on quantized neural network weight reuse to solve the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the specific technical solution of the inference acceleration optimization method based on quantized neural network weight reuse of the present invention is as follows: A method for accelerating reasoning and optimizing based on the reuse of quantized neural network weights includes an offline phase and an online phase. In the offline phase, a data flow graph is generated, and the data flow graph optimizes the calculation process by reusing repeated sub-calculations in the calculation process. In the online phase, the data flow graph is used to perform reasoning operations. The method models the reasoning process as a traversal of the data flow graph. When constructing the data flow graph, the irregular index of the repeated weights is converted into the connection relationship of the nodes in the data flow graph to solve the problem of irregularity in calculation.

[0006] Furthermore, the unique weight in a network is called a unique weight, and its number is recorded as U. Therefore, in the 8-bit weight, U ≤ 2 8 = 256, each u represents a unique weight value, the same subscript represents the same weight value, there are at most U such u, during DNN calculation, each input data and convolution kernel data are multiplied and accumulated in turn.

[0007] Furthermore, the offline phase creates an optimized computational graph, which describes the computational process of the DNN and uses the weight repetition feature to perform computational optimization.

[0008] Furthermore, the offline phase includes creating an original computation graph and splitting common sub-formulas.

[0009] Furthermore, the creating of the original computation graph includes: The trained DNN convolution layer is used as input. The layer is represented as a two-dimensional array containing weight values. For each convolution kernel, all input activation positions multiplied by the same weight value are grouped into the same group. Since each layer has at most U different weights and K convolution kernels, U × K groups are formed. Each group becomes a node in the computational graph, representing a sum to be calculated. A directed edge connects the sum node to the corresponding weight value to indicate that the sum is multiplied by the weight value. There are common sub-formulas between these sum nodes. For further optimization, these sum nodes will be split into common sub-formulas later.

[0010] Furthermore, the split common subformula includes: For the sum nodes in the original computational graph, calculate the intersection of the index sets they contain. This intersection represents the common minors that can be shared. According to the size of the intersection and the number of times it can be reused, select the intersection with the most potential for splitting. Add the selected intersection as a new node to the computational graph and remove these indexes from the original nodes. For the original nodes containing the new nodes, add a directed edge from the new nodes to them to indicate the direction of data flow. Repeat the steps of common minor splitting until no more intersections are found or the preset stopping condition is reached.

[0011] Furthermore, the online stage includes: The computational graph created in the above offline phase will be executed to perform inference calculations on the DNN, copy the input activation values ​​to a data structure that stores the activation values, traverse the data flow graph, perform all addition operations, and for each node in the graph, accumulate the values ​​of all nodes according to the direction of the directed edges. Finally, a multiplication operation will be performed to multiply the partial sum formed in the graph traversal by the corresponding weight.

[0012] The inference acceleration optimization method based on quantized neural network weight reuse of the present invention has the following advantages: The present invention proposes a weight reuse algorithm suitable for quantized neural networks and its implementation technology. Compared with the calculation method of conventional neural networks, the technology of the present invention can reduce the number of addition and multiplication calculations and the number of weight data readings, thereby improving the calculation speed of the neural network. Compared with the technical solution of customizing hardware accelerators, the present invention can achieve benefits on general processors without relying on hardware accelerators, and has practicality and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of DNN calculation; Figure 2 It is a schematic diagram of optimized DNN calculation of the present invention; Figure 3 Schematic diagram of the process of creating the original computational graph and splitting common minor expressions. DETAILED DESCRIPTION

[0014] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an inference acceleration optimization method based on quantized neural network weight reuse of the present invention in conjunction with the accompanying drawings.

[0015] The present invention proposes a novel algorithm and implementation technology that utilizes weight reuse, which improves the reasoning efficiency by reconstructing the convolution operation of DNN as a traversal of a data flow graph. The algorithm of the present invention includes two main modules: an offline stage and an online stage. In the offline stage, the algorithm generates a data flow graph, which optimizes the calculation process by reusing repeated sub-calculations in the calculation process. In the online stage, the algorithm uses the data flow graph to perform reasoning operations, reducing the number of required arithmetic operations. The present invention models the reasoning process as a traversal of a data flow graph, and the structure of these data flow graphs depends on the distribution of repeated weights. When constructing a data flow graph, the algorithm of the present invention converts the irregular index of the repeated weights into the connection relationship of the nodes in the data flow graph, thereby solving the problem of irregularity in calculation.

[0016] The characteristic of quantized neural networks is that the weights of convolution kernels have a large number of repeated values. We call the unique weights in a network unique weights, and the number of them is recorded as U. Therefore, in the 8-bit weights, U ≤ 2 8 = 256. Figure 1 In the convolution operation shown, each u represents a unique weight value, and the same subscript represents the same weight value. There are at most U such u. During DNN calculation, each input data and convolution kernel data are multiplied and accumulated in turn.

[0017] The present invention utilizes the characteristics of weight duplication and explores the opportunity of computing reuse. The DNN computing process of the present invention includes two major stages: one is the offline stage; the other is the online stage.

[0018] The first is the offline phase. In this phase, this solution creates an optimized computational graph that describes the computational process of the DNN and optimizes the computation using the weight repetition feature. The computational graph created is as follows: Figure 2 shown.

[0019] The offline phase of this algorithm specifically includes the following steps: 1. Create the original computational graph The algorithm takes as input a trained DNN convolutional layer represented as a two-dimensional array containing weight values. For each convolution kernel, all input activation positions (indices) multiplied by the same weight value are grouped together. Since each layer has at most U different weights and K convolution kernels, U × K groups are formed. Each group becomes a node in the computational graph, representing a sum to be calculated. The sum node is connected to the corresponding weight value by a directed edge, which indicates that the sum is multiplied by the weight value. There are common sub-formulas between these sum nodes, which can be further optimized. These sum nodes will be split into common sub-formulas later.

[0020] 2. Split common minor For the sum nodes in the original computational graph, calculate the intersection of the index sets they contain. This intersection represents the common minors that can be shared. Based on the size of the intersection and the number of times it can be reused, select the intersection with the most potential for splitting. Add the selected intersection as a new node to the computational graph and remove the indexes from the original node. For the original nodes that contain the new nodes, add a directed edge from the new node to them to indicate the direction of data flow. Repeat the steps of common minor splitting until no more intersections are found or the preset stopping condition is reached.

[0021] The process of creating the original calculation and splitting the common sub-formula is shown below: Figure 3 shown.

[0022] Next is the online phase. In this phase, the present invention will execute the computation graph created in the offline phase and perform inference calculations on the DNN. The input activation value is copied to a data structure that stores the activation value. The data flow graph is traversed and all addition operations are performed. For each node in the graph, the values ​​of all nodes are accumulated according to the direction of the directed edge. Finally, the multiplication operation is performed to multiply the partial sum formed in the graph traversal by the corresponding weight.

[0023] The technical solution of the present invention is advanced, accelerates DNN calculations compared to conventional technologies, and is highly efficient on general-purpose processors. The technical solution of the present invention is applicable to a variety of computing environments such as cloud computing, mobile devices, and the Internet of Things, and is particularly beneficial for application scenarios that have special requirements for real-time reasoning performance. In addition, the present invention provides a new direction for further research and development, which may promote the deployment and application of deep learning models on resource-constrained devices.

[0024] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for accelerating inference optimization based on quantized neural network weight reuse, characterized in that: It includes an offline phase and an online phase. In the offline phase, a data flow graph is generated, which optimizes the computational process by reusing repeated sub-computations in the computational process. In the online stage, the data flow graph is used to perform reasoning operations. The method models the reasoning process as a traversal of the data flow graph. When constructing the data flow graph, the irregular index of repeated weights is converted into the connection relationship of the nodes in the data flow graph to solve the irregularity problem of calculation.

2. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 1 is characterized in that: The unique weight in a network is called a unique weight, and its number is recorded as U. Therefore, in the 8-bit weight, U ≤ 2 8 = 256, each u represents a unique weight value, the same subscript represents the same weight value, there are at most U such u, during DNN calculation, each input data and convolution kernel data are multiplied and accumulated in turn.

3. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 1, characterized in that: The offline phase creates an optimized computational graph that describes the computational flow of the DNN and optimizes the computation by utilizing the weight repetition feature.

4. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 1, characterized in that: The offline phase includes creating the original computation graph and splitting common minor expressions.

5. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 4 is characterized in that: The creating of the original computation graph includes: The trained DNN convolution layer is used as input. The layer is represented as a two-dimensional array containing weight values. For each convolution kernel, all input activation positions multiplied by the same weight value are grouped into the same group. Since each layer has at most U different weights and K convolution kernels, U × K groups are formed. Each group becomes a node in the computational graph, representing a sum to be calculated. A directed edge connects the sum node to the corresponding weight value to indicate that the sum is multiplied by the weight value. There are common sub-formulas between these sum nodes. For further optimization, these sum nodes will be split into common sub-formulas later.

6. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 4, characterized in that: The split common minor formula includes: For the sum nodes in the original computational graph, calculate the intersection of the index sets they contain. This intersection represents the common minors that can be shared. According to the size of the intersection and the number of times it can be reused, select the intersection with the most potential for splitting. Add the selected intersection as a new node to the computational graph and remove these indexes from the original nodes. For the original nodes containing the new nodes, add a directed edge from the new nodes to them to indicate the direction of data flow. Repeat the steps of common minor splitting until no more intersections are found or the preset stopping condition is reached.

7. The inference acceleration optimization method based on quantized neural network weight reuse according to claim 1, characterized in that: The online phase includes: The computational graph created in the above offline phase will be executed to perform inference calculations on the DNN, copy the input activation values ​​to a data structure that stores the activation values, traverse the data flow graph, perform all addition operations, and for each node in the graph, accumulate the values ​​of all nodes according to the direction of the directed edges. Finally, a multiplication operation will be performed to multiply the partial sum formed in the graph traversal by the corresponding weight.

Citation Information

Cited By

  • Calculation acceleration method based on cooperation of fast Fourier transform and neural network reasoning

    CN120950266A