On-chip Execution Method, Device, Equipment and Medium of Quantized Neural Network Model

Abstract

The present invention discloses an in-chip execution method, device, equipment and medium for a quantized neural network model. The method includes: obtaining an original operator graph matching a quantized neural network model to be loaded into a target AI acceleration chip; reconstructing the original operator graph according to the computing unit characteristics and / or instruction set characteristics of the target AI acceleration chip to obtain a first operator graph; determining a target node in a second quantized sensitive node set of the first operator graph and replacing it with a branch network; replacing the computing nodes in a first quantized sensitive node set in the first operator graph with non-quantized nodes; reconstructing the first operator graph according to the storage hierarchy characteristics of the target AI acceleration chip to obtain a second operator graph; retraining an original quantized reconstruction model corresponding to the second operator graph and loading it into the target AI acceleration chip for execution. Through the technical solution of the present invention, on the basis of improving the accuracy of the quantized neural network model, the computing performance of the AI acceleration chip can be exerted to the optimal level.

Description

Technical Field

[0001] The present invention relates to the field of in-chip execution of neural network models, and particularly to an in-chip execution method, device, equipment and medium for quantized neural network models. Background Art

[0002] With the gradual development of the field of neural network model technology, there are higher requirements for the compression deployment and computing acceleration of neural network models.

[0003] In the prior art, in order to effectively reduce the amount of stored data in an AI (Artificial Intelligence) acceleration chip and improve the operating performance of the AI acceleration chip, the neural network model is usually compressed by using the method of low-bit model quantization and then deployed on the AI acceleration chip. However, the method of using a low-bit quantization model will cause a problem of accuracy degradation during the actual deployment of the model. At the same time, the existing quantization deployment methods generally adopt a static quantization deployment method and do not actually consider the actual hardware characteristics of the AI acceleration chip. Therefore, how to effectively solve the problem of accuracy degradation of the neural network model deployed by the low-bit quantization model, and how to give full play to the computing performance of the AI acceleration chip after the quantization deployment of the neural network model are problems that need to be solved urgently at present. Summary of the Invention

[0004] Embodiments of the present invention provide an in-chip execution method, device, equipment and medium for quantized neural network models, so as to improve the accuracy of the quantized neural network model while achieving precise adaptation between the quantized neural network model and the deployed AI acceleration chip.

[0005] According to one aspect of the present invention, an in-chip execution method for a quantized neural network model is provided, including:

[0006] Obtaining an original operator graph that matches a quantized neural network model to be loaded into a target AI acceleration chip, where the quantized neural network model is obtained by post-training quantization of the target neural network model;

[0007] Performing a first reconstruction on the original computational operator graph according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph;

[0008] Identifying a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes among the computing nodes in the first operator graph, and determining a set number of target nodes in the second set of quantization-sensitive nodes; where the quantization sensitivity of the first quantization-sensitive nodes is higher than that of the second quantization-sensitive nodes;

[0009] In the first operator graph, each computing node in the first set of quantization-sensitive nodes is replaced with a non-quantized node, and each target node is replaced with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weighted-fit the calculation result of the replaced node;

[0010] According to the memory hierarchy characteristics of the target AI acceleration chip, the first operator graph is re-constructed twice to obtain a second operator graph;

[0011] The original quantization reconstruction model corresponding to the second operator graph is retrained to obtain a target quantization reconstruction model, and the target quantization reconstruction model is loaded into the target AI acceleration chip for execution.

[0012] According to another aspect of the present invention, there is provided an in-chip execution device for a quantized neural network model, the device comprising:

[0013] An original operator graph acquisition module, configured to acquire an original operator graph matching the quantized neural network model to be loaded into the target AI acceleration chip, where the quantized neural network model is obtained by post-training quantization of the target neural network model;

[0014] A first operator graph reconstruction module, configured to perform a first reconstruction on the original operator graph according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph;

[0015] A sensitive node identification module, configured to identify a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes among the computing nodes in the first operator graph, and determine a set number of target nodes in the second set of quantization-sensitive nodes; wherein the quantization sensitivity of the first quantization-sensitive node is higher than that of the second quantization-sensitive node;

[0016] A node processing module, configured to replace each computing node in the first set of quantization-sensitive nodes with a non-quantized node in the first operator graph, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weighted-fit the calculation result of the replaced node;

[0017] A second operator graph reconstruction module, configured to perform a second reconstruction on the first operator graph according to the memory hierarchy characteristics of the target AI acceleration chip to obtain a second operator graph;

[0018] A reconstruction model acquisition module, configured to retrain the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model, and load the target quantization reconstruction model into the target AI acceleration chip for execution.

[0019] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein

[0022] the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the on-chip execution method of the quantization neural network model according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the on-chip execution method of the quantization neural network model according to any embodiment of the present invention when executed.

[0024] The technical solution of the embodiment of the present invention performs a first reconstruction on the original operator graph matching the quantization neural network model to be loaded into the target AI acceleration chip according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph; further, in each computing node of the first operator graph, a first quantization-sensitive node set and a second quantization-sensitive node set are identified, and a set number of target nodes are determined in the second quantization-sensitive node set; further, in the first operator graph, each computing node in the first quantization-sensitive node set is replaced with a non-quantized node, and each target node is replaced with a branch network; finally, according to the storage hierarchy characteristics of the target AI acceleration chip, a second reconstruction is performed on the first operator graph to obtain a second operator graph; the original quantization reconstruction model corresponding to the second operator graph is retrained to obtain a target quantization reconstruction model, and the target quantization reconstruction model is loaded into the target AI acceleration chip for execution, which solves the problem of the accuracy decline of the neural network model in the deployment of the low-bit quantization model, can suppress the loss of feature representation of the quantization neural network model, improves the accuracy of the quantization neural network model, and at the same time, on the basis of improving the accuracy of the quantization neural network model, can achieve the optimal adaptation of the quantization neural network model on the hardware and give full play to the computing performance of the AI acceleration chip.

[0025] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0027] Figure 1 is a flowchart of an in-chip execution method for a quantization neural network model according to Embodiment 1 of the present invention;

[0028] Figure 2a is a flowchart of an in-chip execution method for a quantization neural network model according to Embodiment 2 of the present invention;

[0029] Figure 2b [[ID=I2]]is a schematic diagram of a branch network replacement process according to Embodiment 2 of the present invention;

[0030] Figure 2c is a schematic diagram of the process of adjusting the parameters of the original quantization reconstruction model according to Embodiment 2 of the present invention;

[0031] Figure 2d is a schematic diagram of a preferred in-chip execution method for a quantization neural network model according to Embodiment 2 of the present invention;

[0032] Figure 3 is a schematic diagram of the structure of an in-chip execution device for a quantization neural network model according to Embodiment 3 of the present invention;

[0033] Figure 4 is a schematic diagram of the structure of an electronic device for implementing the in-chip execution method of the quantization neural network model of the embodiments of the present invention. Detailed Embodiments

[0034] To enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0035] It should be noted that the terms "first", "second", "target", "original", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment 1

[0037] Figure 1 FIG. 1 is a flowchart of an in-chip execution method for quantifying a neural network model provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of low-bit quantization of a neural network model. This method can be executed by an in-chip execution device for a quantized neural network model. The in-chip execution device for the quantized neural network model can be implemented in the form of hardware and / or software, and the in-chip execution device for the quantized neural network model can be configured in an electronic device. As Figure 1 shown, the method includes:

[0038] S110. Obtain an original operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, where the quantized neural network model is obtained by post-training quantization of the target neural network model.

[0039] Among them, the target artificial intelligence acceleration chip may refer to the artificial intelligence acceleration chip that bears the quantized neural network model. The original operator graph may refer to the operator graph corresponding to the quantized neural network model, and may include each operator of the quantized neural network model and the connection relationship between each operator. Each operator is used to execute a set operation logic, for example, addition, subtraction, or convolution, etc.

[0040] Among them, quantization may refer to the process of approximating the original data to the data after the precision change. Exemplarily, a certain quantization standard is to quantize all data between 0.3 and 0.4 to 0.3. Then, if the data is 0.351, the corresponding quantized data is 0.3. Post-training quantization may refer to quantizing and training a neural network model. The target neural network model may refer to the neural network model that needs to be post-trained and quantized, such as a dense neural network model (Dense model).

[0041] S120. Reconstruct the original operator graph once according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph.

[0042] Among them, the computing unit features can refer to the features of each computing unit in the target AI acceleration chip. Among them, the above-mentioned computing unit can be understood as the arithmetic logic unit (ALU) in the target AI acceleration chip, and this arithmetic logic unit is the computing unit that is the smallest unit for implementing logical operations in the target AI acceleration chip. Among them, the target AI acceleration chip may include one or more computing cores, and each computing core contains one or more ALUs.

[0043] In this embodiment, the computing unit features may include the number of computing units, the data bit width that each computing unit can calculate (for example, float32 or int16, etc.), the computing cores to which each computing unit belongs, the parallelism between each computing unit, and the efficient operation operators corresponding to each computing unit respectively. Among them, the efficient operation operator can be understood as an operator that can be efficiently run by a computing unit after configuring an operator in the computing unit.

[0044] The instruction set refers to a set of commands used to instruct the hardware in the target AI acceleration chip to execute set operations or processing functions. Among them, the instruction set can be a RISC instruction set, a RISC-V instruction set, or an ARM instruction set, etc., and this embodiment does not limit this.

[0045] The instruction set features can be understood as the operation performance parameters of each instruction in the instruction set determined after running tests after loading the instruction set in the target AI acceleration chip. For example, parameters such as computing speed and memory occupancy rate.

[0046] It can be understood that by analyzing the operation performance parameters of each instruction in the target AI acceleration chip, the chip-friendly instruction set that matches the target AI acceleration chip can be screened out from all instruction sets. By running each instruction in the chip-friendly instruction set in the target AI acceleration chip, the target AI acceleration chip can exert better hardware computing performance.

[0047] The first operator graph can refer to the operator graph generated after reconstructing the operators in the original operator graph according to the computing unit features and / or instruction set features in the target AI acceleration chip.

[0048] In an optional implementation manner of this embodiment, the method for reconstructing the original operator graph once according to the computing unit features in the target AI acceleration chip can be: according to the efficient operation operator of each computing unit and the pre-constructed operator reconstruction strategy, reconstruct the original operator graph once, so that as many efficient operation operators as possible are included in the first operator graph.

[0049] Among them, the operator reconstruction strategy includes multiple operator reconstruction rules, and each operator reconstruction rule defines the implementation method of reconstructing one or more original operators into one or more reconstructed operators.

[0050] In an alternative implementation manner of this embodiment, the method for performing a first reconstruction on the original operator graph according to the instruction set characteristics in the target AI acceleration chip may be: performing a first reconstruction on the original operator graph according to the chip-friendly instruction set and the pre-constructed operator reconstruction strategy, so that as many operators for executing chip-friendly instructions as possible are included in the first operator graph.

[0051] S130. In each computing node in the first operator graph, identify the first quantization-sensitive node set and the second quantization-sensitive node set, and determine a set number of target nodes in the second quantization-sensitive node set; where the quantization sensitivity of the first quantization-sensitive nodes is higher than that of the second quantization-sensitive nodes.

[0052] Among them, the quantization-sensitive nodes may refer to the computing nodes that have a high impact on the loss function of the target neural network model, that is, the computing nodes that cause serious losses to the loss function of the target neural network model. The first quantization-sensitive nodes may refer to the quantization-sensitive nodes with relatively high sensitivity; the second quantization-sensitive nodes may refer to the quantization-sensitive nodes with general sensitivity. The quantization-sensitive node set may refer to a set composed of quantization-sensitive nodes. The first quantization-sensitive node set may refer to a set composed of each first quantization-sensitive node; the second quantization-sensitive node set may refer to a set composed of each second quantization-sensitive node. The target nodes may refer to the selected quantization-sensitive nodes for subsequent operations, and the number of target nodes may be selected according to actual applications. Exemplarily, if the second quantization-sensitive node set contains ten quantization-sensitive nodes, three quantization-sensitive nodes may be first selected as target nodes to avoid waste of computing resources caused by a large number of target nodes. If more target nodes are needed in subsequent operations, the number of target nodes may be increased.

[0053] Optionally, in each computing node in the first operator graph, identifying the first quantization-sensitive node set and the second quantization-sensitive node set may include: respectively calculating the influence degree of each computing node in the first operator graph on the loss function of the target neural network model; identifying the first quantization-sensitive nodes whose influence degree on the loss function exceeds the first influence degree threshold to form the first quantization-sensitive node set; identifying the second quantization-sensitive nodes whose influence degree on the loss function is greater than or equal to the second influence degree threshold and less than the first influence degree threshold to form the second quantization-sensitive node set, where the first influence degree threshold is greater than the second influence degree threshold. Among them, the first influence degree threshold may refer to the value for evaluating the first quantization-sensitive nodes. The second influence degree threshold may refer to the value for evaluating the second quantization-sensitive nodes. Exemplarily, calibration can be performed by introducing test input data, respectively calculating the influence degree of each computing node in the first operator graph on the loss function of the target neural network model, and screening out the first quantization-sensitive nodes whose influence degree on the loss function exceeds the first influence degree threshold to form the first quantization-sensitive node set, and screening out the second quantization-sensitive nodes whose influence degree on the loss function is greater than or equal to the second influence degree threshold and less than the first influence degree threshold to form the second quantization-sensitive node set. Thus, quantization-sensitive nodes with a high influence degree on the loss function of the target neural network model can be identified.

[0054] S140. In the first operator graph, replace each computing node in the first quantization-sensitive node set with a non-quantization node, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weightedly fit the calculation result of the replaced node.

[0055] Among them, the non-quantization node may refer to each node included in the operator graph corresponding to the target neural network model. The branch network may refer to a network including multiple branch nodes established according to the processing difference before and after quantization of the target node. The number of branch networks is the same as the number of target nodes selected in the second quantization-sensitive node set. Each branch network includes multiple branch nodes, and each branch node has the same structure as the target node replaced by the branch network where it is located.

[0056] That is, in this embodiment, it is considered not to quantize the operators in the operator graph that are abnormally sensitive to quantization, but to replace them with the corresponding operators in the target neural network model before quantization to optimize the calculation accuracy of the quantized neural network model.

[0057] Among them, each branch can be initialized with the same weight. Exemplarily, it can be initialized with the weight corresponding to the target node in the quantized neural network model.

[0058] Thus, by replacing each computing node in the first set of quantization-sensitive nodes with a non-quantized node and replacing the target node with a branch network corresponding to the target node, the loss of feature representation in the quantized neural network model can be suppressed, thereby effectively solving the problem of the decrease in the accuracy of the neural network model deployed by the low-bit quantization model.

[0059] It should be emphasized again that the essence of this application is: replacing the computing nodes sensitive to quantization, i.e., the target nodes, in the neural network model with multiple computing nodes of the same structure. Furthermore, the computing accuracy of a single computing node can be corrected by means of weighted fitting of the computing results of multiple nodes, so as to improve the model accuracy of the quantized neural network model.

[0060] S150. According to the storage hierarchy characteristics of the target AI acceleration chip, perform secondary reconstruction on the first operator graph to obtain a second operator graph.

[0061] Among them, the storage hierarchy characteristics may include: whether the target AI acceleration chip is a multi-level storage; the number of storage levels when the target AI acceleration chip is a multi-level storage, and the memory type of each storage level (for example, Flash, DRAM, or Cache), etc.

[0062] In this embodiment, secondary reconstruction strategies corresponding to different storage hierarchy characteristics can be preset respectively. For example, when the storage level is 1 and the memory type is Flash, the combination of quantization operators - non-quantization operators - quantization operators in the operator graph is uniformly reconstructed into a quantization operator. Furthermore, after obtaining the storage hierarchy characteristics of the target AI acceleration chip, the first operator graph can be secondarily reconstructed according to the secondary reconstruction strategy satisfied by the storage hierarchy characteristics to obtain a second operator graph.

[0063] Through the above settings, the operator graph of the quantized neural network model can be further optimized in combination with the storage hierarchy characteristics of the target AI acceleration chip, so as to maximize the computing performance of the target AI acceleration chip.

[0064] S160. Retrain the original quantized reconstruction model corresponding to the second operator graph to obtain a target quantized reconstruction model, and load the target quantized reconstruction model into the target AI acceleration chip for execution.

[0065] Among them, the original quantized reconstruction model may refer to the quantized reconstruction model corresponding to the second operator graph. The target quantized reconstruction model may refer to the original quantized reconstruction model that meets the set accuracy requirements after retraining. Retraining may refer to retraining the loss function of the original quantized reconstruction model using the target neural network model to obtain a target quantized reconstruction model that matches the quantized neural network model.

[0066] The technical solution of the embodiment of the present invention performs a primary reconstruction on the original operator graph matching the quantized neural network model to be loaded into the target AI acceleration chip according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph. Furthermore, in each computing node of the first operator graph, a first quantized sensitive node set and a second quantized sensitive node set are identified, and a set number of target nodes are determined in the second quantized sensitive node set. Further, in the first operator graph, each computing node in the first quantized sensitive node set is replaced with a non-quantized node, and each target node is replaced with a branch network. Finally, according to the storage hierarchy characteristics of the target AI acceleration chip, a secondary reconstruction is performed on the first operator graph to obtain a second operator graph. The original quantized reconstruction model corresponding to the second operator graph is retrained to obtain a target quantized reconstruction model, and the target quantized reconstruction model is loaded into the target AI acceleration chip for execution, which solves the problem of the accuracy degradation of the neural network model in the deployment of the low-bit quantization model, can suppress the loss of feature representation of the quantized neural network model, improves the accuracy of the quantized neural network model. At the same time, on the basis of improving the accuracy of the quantized neural network model, the optimal adaptation of the quantized neural network model to the hardware can be realized, and the computing performance of the AI acceleration chip can be exerted to the optimal.

[0067] Embodiment 2

[0068] Figure 2a The flowchart of an in-chip execution method for a quantized neural network model provided by the second embodiment of the present invention is based on the above embodiment and is appended. Specifically, in this embodiment, the operations before replacing each target node with a branch network are appended, which may specifically include: determining the branch number range corresponding to each target node according to the quantization pre- and post-processing differences of each target node; determining the branch number corresponding to each target node according to the branch number range of each target node and the total number of computing units in the target AI acceleration chip, and constructing a branch network corresponding to each target node according to each branch number.

[0069] As Figure 2a shown, the method includes:

[0070] S210. Identify the model calculation scenario matching the quantized neural network model.

[0071] Among them, the model calculation scenario may refer to the calculation scenario in which the quantized neural network model is actually applied. Exemplarily, according to the accuracy requirements of the calculation scenario, the model calculation scenario can be divided into a high-precision demand calculation scenario with relatively high requirements for calculation accuracy; and a low-precision demand calculation scenario with relatively low requirements for calculation accuracy.

[0072] S220. If it is determined that the model calculation scenario belongs to a high-precision demand calculation scenario, obtain the original operator graph that matches the quantization neural network model to be loaded into the target AI acceleration chip.

[0073] Among them, the high-precision demand calculation scenario may include: the machine vision image classification scenario during the automatic driving process, or the natural speech processing scenario during the intelligent question and answer process.

[0074] S230. According to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip, perform a first reconstruction on the original operator graph to obtain the first operator graph.

[0075] S240. Calculate the influence degree of each computing node in the first operator graph on the loss function of the target neural network model respectively.

[0076] In an optional implementation manner, calculating the influence degree of each computing node in the first operator graph on the loss function of the target neural network model may include: inputting the test input data into the target neural network model and the quantization neural network model respectively; obtaining the original output feature map and the quantization output feature map of each computing node in the target neural network model and the quantization neural network model; according to each of the original output feature maps and the quantization output feature maps, calculate the difference value of the feature maps before and after quantization of each computing node as the influence degree of the loss function.

[0077] Among them, the test input data may refer to the open-source standard data used to test the quantization neural network model. The original output feature map may refer to the output feature map corresponding to each computing node when the test input data is input into the target neural network model. The quantization output feature map may refer to the output feature map corresponding to each computing node when the test input data is input into the quantization neural network model. Thus, by evaluating the difference value between the feature maps of the computing node before and after quantization, the influence degree of each computing node on the loss function of the target neural network model can be calculated, providing an effective basis for subsequent operations.

[0078] In another optional embodiment, respectively calculating the influence of each computing node in the first operator graph on the loss function of the target neural network model may include: obtaining the current processing node in each of the computing nodes in turn; inputting the test input data into the target neural network model, and calculating the first loss function value of the target neural network model for the test input data; replacing the original weight matrix corresponding to the current processing node in the target neural network model with the quantized weight matrix corresponding to the current processing node in the quantized neural network model; re-inputting the test input data into the target neural network model, and calculating the second loss function value of the target neural network model for the test input data; calculating the loss difference value between the first loss function value and the second loss function value, and when determining that the loss difference value meets the loss difference threshold condition, determining the current processing node as a quantization sensitive node; returning to execute the operation of obtaining the current processing node in each of the computing nodes in turn until the processing of all computing nodes is completed.

[0079] Among them, the first loss function value may refer to the loss function value output by the target neural network model for the test input data. The second loss function value may refer to the loss function value output by the quantized neural network model for the test input data. The loss difference threshold condition may refer to a numerical value for evaluating the loss difference value, and illustratively, it may be a numerical value within a set range. Thus, by evaluating the loss difference value before and after the quantization processing of the current processing node, the current processing node whose loss difference value meets the loss difference threshold condition can be used as a quantization sensitive node.

[0080] It is worth noting that the Hessian trajectory calculation method can also be used to calculate the loss function growth of each computing node before and after quantization processing, and obtain quantization sensitive nodes to reduce the amount of calculation.

[0081] S250 : Identify a first quantization sensitive node whose loss function influence exceeds a first influence threshold, and form a first quantization sensitive node set.

[0082] S260 . Identify a second quantization sensitive node whose loss function influence is greater than or equal to a second influence threshold and less than the first influence threshold, to form a second quantization sensitive node set, wherein the first influence threshold is greater than the second influence threshold.

[0083] S270 . Determine a set number of target nodes in the second quantization sensitive node set; wherein the quantization sensitivity of the first quantization sensitive node is higher than that of the second quantization sensitive node.

[0084] S280. In the first operator graph, replace each computing node in the first quantization-sensitive node set with a non-quantized node, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weightedly fit the calculation result of the replaced node.

[0085] As Figure 2b shown in the schematic diagram of the branch network replacement process. Specifically, if four branches are selected according to the branch number range of the target node, each branch node can be branch A, branch B, branch C, and branch D; if five branches are selected according to the branch number range of the target node, each branch node can be branch A, branch B, branch C, branch D, and branch E. In a specific example, the network composed of branch A, branch B, branch C, branch D, and other branches is the branch network corresponding to the target node, and the activation function expression of this branch network is:

[0086]

[0087] where represents the eigenvalue output of each branch at the current L layer, where L represents the neural network layer where the target node replaced by the branch network is located; α b represents the weight of each branch. Usually, the weights of each branch are initialized to the same weight; b represents the number of branches.

[0088] In an optional implementation, before replacing each target node with a branch network, it may further include: determining the branch number range corresponding to each target node according to the quantization pre- and post-processing differences of each target node; determining the branch number corresponding to each target node according to the branch number range of each target node and the total number of computing units in the target AI acceleration chip, and constructing the branch network corresponding to each target node according to each branch number.

[0089] Among them, the quantization pre- and post-processing difference may refer to the feature map difference value of each target node before and after quantization. The branch number range may refer to the range of the branch number corresponding to each target node. Exemplarily, if the quantization pre- and post-processing difference of the target node is small, the branch number range corresponding to the target node is small; if the quantization pre- and post-processing difference of the target node is large, the branch number range corresponding to the target node is large. Thus, it can be ensured that the calculation result of the subsequent constructed branch network can be closer to the calculation result of the target node before quantization, improving the accuracy of the original quantization reconstruction model.

[0090] Among them, the number of branches may refer to the number of branches corresponding to the target node within the range of the number of branches, and the number of branches can be set according to actual application requirements. Exemplarily, if the range of the number of branches of the target node is 6-10, any number within 6-10 can be selected as the number of branches. The total number of computing units may refer to the number of computing units included in the target AI acceleration chip.

[0091] In an alternative embodiment, determining the range of the number of branches corresponding to each of the target nodes according to the processing differences before and after quantization of each target node may include: inputting test input data into the target neural network model and the quantized neural network model respectively; obtaining the original output feature maps and quantized output feature maps of each target node in the target neural network model and the quantized neural network model; calculating the feature map difference value of each target node before and after quantization according to each of the original output feature maps and quantized output feature maps; and determining the range of the number of branches corresponding to each of the target nodes according to the feature map difference values of each of the target nodes.

[0092] Thus, by calculating the feature map difference value between the original output feature map and the quantized output feature map of the target node in the target neural network model and the quantized neural network model, the range of the number of branches corresponding to each target node can be obtained, providing an effective basis for the subsequent construction of the branch network.

[0093] In another alternative embodiment, determining the number of branches corresponding to each of the target nodes according to the range of the number of branches of each of the target nodes and the total number of computing units in the target AI acceleration chip includes: selecting, from the range of the number of branches of each of the target nodes, the number of branches that is an integer multiple of the total number of computing units to determine the number of branches corresponding to each of the target nodes. Specifically, if the total number of computing units is 2 and the range of the number of branches is 5-10, any number that is an integer multiple of the total number of computing units, such as 2, 4, 6, 8, and 10, within the range of the number of branches 5-10 can be selected. Exemplarily, any one of 6 or 8 can be used as the number of branches corresponding to the target node respectively, and this embodiment does not limit this.

[0094] ]>Of course, in addition to selecting the number of branches that is an integer multiple of the total number of computing units as the number of branches corresponding to each of the target nodes, the number of branches that is a divisor of the total number of computing units can also be selected as the number of branches corresponding to each of the target nodes, etc.

[0095] Through the above settings, setting the integer ratio relationship between the computing units and the number of branches can enable each computing unit to exert the best computing performance when running the target quantization reconstruction model. "

[0096] S290. Reconstruct the first operator graph a second time according to the storage hierarchy characteristics of the target AI acceleration chip to obtain a second operator graph.

[0097] S2100. Retrain the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model, and load the target quantization reconstruction model into the target AI acceleration chip for execution.

[0098] It should be noted that in the embodiments of the present invention, retraining the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model may specifically be: obtaining the target neural network model as a teacher model, and sequentially obtaining a current training sample and inputting it into the original quantization reconstruction model and the teacher model. Obtaining the quantization model output results output by each branch network in the original quantization reconstruction model for the current training sample, and the teacher model output results output by each target node in the teacher model for the current training sample; according to the differences between the quantization model output results and the corresponding teacher model output results, after adjusting the parameters of the original quantization reconstruction model, return to execute the operation of sequentially obtaining a current training sample and inputting it into the original quantization reconstruction and the teacher model until the end iteration condition is satisfied to obtain the target quantization reconstruction model.

[0099] Among them, the teacher model may refer to a control model used to adjust the original quantization reconstruction model. In the embodiments of the present invention, the target neural network model may be used as the teacher model. The end iteration condition may refer to the condition for the end of the retraining of the target quantization reconstruction model. Exemplarily, it may be that the target quantization reconstruction model meets the set accuracy condition, or the number of iterations meets the set iteration threshold, etc.

[0100] In an optional implementation manner, according to the differences between the quantization model output results and the corresponding teacher model output results, adjusting the parameters of the original quantization reconstruction model may include: calculating a first type of loss function according to the differences between the quantization model output results and the corresponding teacher model output results; calculating a second type of loss function according to the differences between the final output result of the original quantization reconstruction model for the current training sample and the annotation result of the current training sample; calculating a target loss function according to the first type of loss function, the second type of loss function, and the loss function weight allocation method, and using the target loss function to adjust the parameters of the original quantization reconstruction model.

[0101] Such as Figure 2cThe figure shows a schematic flow chart of parameter adjustment of the original quantization reconstruction model. Among them, the first type of loss function can refer to the loss function calculated based on the difference between the quantization model output results output by each branch network in the original quantization reconstruction model for the current training sample and the teacher model output results output by each target node in the teacher model for the current training sample.

[0102] The second type of loss function can refer to the loss function calculated based on the difference between the final output result of the original quantization reconstruction model for the current training sample and the annotation result of the current training sample.

[0103] The specific calculation formulas of the first type of loss function and the second type of loss function can adopt corresponding calculation formulas according to different methods. Exemplarily, if the cross-entropy (CE) method is adopted, the expression of the loss function can be:

[0104]

[0105] If the mean squared error (MSE) method is adopted, the expression of the loss function can be:

[0106]

[0107] If the KL (Kullback-Leibler) method is adopted, the expression of the loss function can be:

[0108]

[0109] Among them, the target loss function can refer to the final loss function obtained according to the first type of loss function, the second type of loss function, and the loss function weight allocation method. Exemplarily, the calculation formula of the target loss function can be:

[0110]

[0111] Among them, l represents the number of the first type of loss functions, that is, the number of branch networks in the original quantization reconstruction model.

[0112] Loss org represents the second type of loss function, Let \(\alpha\) represent the first type of loss function, \(\beta\) represent the weight of the second type of loss function, and \(\gamma\) represent the weight of the first type of loss function. Among them, the loss function weight allocation method can refer to the weight allocation methods of the first type of loss function and the second type of loss function. Exemplarily, the loss function weight allocation method can be that the weight \(\gamma\) of the first type of loss function is 0.4 and the weight \(\beta\) of the second type of loss function is 0.6; it can also be that the weight \(\gamma\) of the first type of loss function is 0.5 and the weight \(\beta\) of the second type of loss function is 0.5. The embodiments of the present invention do not limit this.

[0113] In another alternative embodiment, after retraining the original quantization reconstruction model corresponding to the second operator graph to obtain the target quantization reconstruction model, it may further include: detecting whether the target quantization reconstruction model meets a preset calculation accuracy condition; if not, then after readjusting at least one hyperparameter of the target quantization reconstruction model, retraining to obtain a new target quantization reconstruction model; where the hyperparameters include at least one of the following: the number value of the target nodes determined in the quantization-sensitive node set, the number of branches corresponding to each target node, and the loss function weight allocation method.

[0114] Specifically, the more the number value of the target nodes and the closer the number of branches corresponding to each target node is to the maximum value of the branch number range, the more accurate the accuracy of the finally obtained target quantization reconstruction model will be. However, considering the computational burden on the hardware device, usually when initially determining the number value of the target nodes and the number of branches corresponding to each target node, larger numbers will not be selected. Therefore, when the target quantization reconstruction model does not meet the preset calculation accuracy condition, the number value of the target nodes and the number of branches corresponding to each target node can be appropriately increased again, and a new round of retraining is performed until the target quantization reconstruction model meets the preset calculation accuracy condition. In addition, the loss function weight allocation method will also affect the accuracy of the target quantization reconstruction model. Therefore, when the target quantization reconstruction model does not meet the preset calculation accuracy condition, the loss function weight allocation method can also be adaptively adjusted until the target quantization reconstruction model meets the preset calculation accuracy condition.

[0115] In the technical solution of the embodiment of the present invention, by discriminating the model calculation scenario matching the quantization neural network model, when it is determined that the model calculation scenario belongs to a high-precision demand calculation scenario, the original operator graph matching the quantization neural network model to be loaded into the target AI acceleration chip is obtained. Then, according to the calculation unit characteristics and / or instruction set characteristics in the target AI acceleration chip, the original operator graph is reconstructed once to obtain a first operator graph. Further, the influence degree of each calculation node in the first operator graph on the loss function of the target neural network model is calculated respectively, and the first quantization sensitive nodes with the loss function influence degree exceeding the first influence degree threshold are identified to form a first quantization sensitive node set. In addition, the second quantization sensitive nodes with the loss function influence degree greater than or equal to the second influence degree threshold and less than the first influence degree threshold are identified to form a second quantization sensitive node set. Then, a set number of target nodes are determined in the second quantization sensitive node set. In the first operator graph, each calculation node in the first quantization sensitive node set is replaced with a non-quantized node, and each target node is replaced with a branch network. Further, according to the storage hierarchy characteristics of the target AI acceleration chip, the first operator graph is reconstructed twice to obtain a second operator graph. Finally, the original quantization reconstruction model corresponding to the second operator graph is retrained to obtain a target quantization reconstruction model, and the target quantization reconstruction model is loaded into the target AI acceleration chip for execution, which solves the problem of the accuracy decline of the neural network model in the deployment of the low-bit quantization model, can suppress the loss of feature representation of the quantization neural network model, improves the accuracy of the quantization neural network model. At the same time, on the basis of improving the accuracy of the quantization neural network model, the optimal adaptation of the quantization neural network model on the hardware can be realized, and the computing performance of the AI acceleration chip can be exerted to the optimal.

[0116] As Figure 2d shown is a schematic flowchart of a preferred method for on-chip execution of a quantization neural network model provided by an embodiment of the present invention.

[0117] Specifically, first, perform post-training quantization on the target neural network model to obtain a quantized neural network model. Then, obtain the original operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip. And according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip, perform a first reconstruction on the original operator graph to obtain a first operator graph. Then, perform sensitivity analysis on each computing node in the first operator graph to identify a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes that have a high impact on the loss function of the target neural network model, and determine a set number of target nodes in the second set of quantization-sensitive nodes. Further, according to the processing differences before and after quantization of each target node, determine the branch number range corresponding to each target node, and according to the branch number range of each target node and the total number of computing units in the target AI acceleration chip, determine the branch number corresponding to each target node, so as to construct branch networks corresponding to each target node according to each branch number, and replace each target node with a matching branch network, and replace each computing node in the first set of quantization-sensitive nodes with a non-quantized node. After that, according to the storage hierarchy characteristics of the target AI acceleration chip, perform a second reconstruction on the first operator graph to obtain a second operator graph. Further, obtain the target neural network model as a teacher model, sequentially obtain a current training sample and input it into the original quantized reconstruction model corresponding to the second operator graph and the teacher model, and obtain the quantized model output results output by each branch network in the original quantized reconstruction model for the current training sample, and the teacher model output results output by each target node in the teacher model for the current training sample. According to the differences between the quantized model output results and the matching teacher model output results, adjust the parameters of the original quantized reconstruction model to obtain a target quantized reconstruction model. Finally, judge the computing accuracy of the target quantized reconstruction model. If the target quantized reconstruction model does not meet the preset computing accuracy condition, return to execute the construction of the branch networks corresponding to each target node. If the target quantized reconstruction model meets the preset computing accuracy condition, output the target quantized reconstruction model and perform corresponding model optimization and deployment. Thus, a target quantized reconstruction model that meets the preset computing accuracy condition can be obtained.

[0118] Embodiment III

[0119] Figure 3 FIG. is a schematic structural diagram of an on-chip execution device for a quantized neural network model provided in Embodiment III of the present invention. As Figure 3 shown, the device includes: an original operator graph acquisition module 310, a first operator graph reconstruction module 320, a sensitive node identification module 330, a node processing module 340, a second operator graph reconstruction module 350, and a reconstructed model acquisition module 360;

[0120] Among them, the original operator graph acquisition module 310 is used to acquire an original operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, where the quantized neural network model is obtained by performing post-training quantization on the target neural network model;

[0121] The first operator graph reconstruction module 320 is used to perform a first reconstruction on the original operator graph according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph;

[0122] The sensitive node identification module 330 is used to identify a first quantized sensitive node set and a second quantized sensitive node set among the computing nodes in the first operator graph, and determine a set number of target nodes in the second quantized sensitive node set; among them, the quantization sensitivity of the first quantized sensitive nodes is higher than that of the second quantized sensitive nodes;

[0123] The node processing module 340 is used to replace each computing node in the first quantized sensitive node set with a non-quantized node in the first operator graph, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weightedly fit the computing result of the replaced node;

[0124] The second operator graph reconstruction module 350 is used to perform a second reconstruction on the first operator graph according to the storage hierarchy characteristics of the target AI acceleration chip to obtain a second operator graph;

[0125] The reconstructed model acquisition module 360 is used to retrain the original quantized reconstructed model corresponding to the second operator graph to obtain a target quantized reconstructed model, and load the target quantized reconstructed model into the target AI acceleration chip for execution.

[0126] The technical solution of the embodiment of the present invention performs a primary reconstruction on the original operator graph matching the quantized neural network model to be loaded into the target AI acceleration chip according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph. Furthermore, in each computing node of the first operator graph, a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes are identified, and a set number of target nodes are determined in the second set of quantization-sensitive nodes. Further, in the first operator graph, each computing node in the first set of quantization-sensitive nodes is replaced with a non-quantized node, and each target node is replaced with a branch network. Finally, according to the storage hierarchy characteristics of the target AI acceleration chip, a secondary reconstruction is performed on the first operator graph to obtain a second operator graph. The original quantized reconstruction model corresponding to the second operator graph is retrained to obtain a target quantized reconstruction model, and the target quantized reconstruction model is loaded into the target AI acceleration chip for execution, which solves the problem of the accuracy decline of the neural network model in the deployment of the low-bit quantization model, can suppress the loss of feature representation of the quantized neural network model, improves the accuracy of the quantized neural network model. At the same time, on the basis of improving the accuracy of the quantized neural network model, the optimal adaptation of the quantized neural network model on the hardware can be realized, and the computing performance of the AI acceleration chip can be exerted to the optimal.

[0127] Optionally, the sensitive node recognition module 330 may specifically include: a loss function influence degree calculation unit, a first quantization-sensitive node set generation unit, and a second quantization-sensitive node set generation unit.

[0128] Among them, the loss function influence degree calculation unit is used to calculate the influence degree of each computing node in the first operator graph on the loss function of the target neural network model respectively.

[0129] The first quantization-sensitive node set generation unit is used to identify the first quantization-sensitive nodes whose loss function influence degree exceeds the first influence degree threshold and form a first quantization-sensitive node set.

[0130] The second quantization-sensitive node set generation unit is used to identify the second quantization-sensitive nodes whose loss function influence degree is greater than or equal to the second influence degree threshold and less than the first influence degree threshold, and form a second quantization-sensitive node set, where the first influence degree threshold is greater than the second influence degree threshold.

[0131] Optionally, the loss function influence degree calculation unit may specifically be used to: input the test input data into the target neural network model and the quantized neural network model respectively; obtain the original output feature map and the quantized output feature map of each computing node in the target neural network model and the quantized neural network model; and calculate the feature map difference value before and after quantization of each computing node as the loss function influence degree according to each original output feature map and quantized output feature map.

[0132] Optionally, the loss function influence degree calculation unit can be specifically used for: in each of the calculation nodes, sequentially obtaining the current processing node; inputting the test input data into the target neural network model, and calculating a first loss function value of the target neural network model for the test input data; replacing the original weight matrix corresponding to the current processing node in the target neural network model with the quantized weight matrix corresponding to the current processing node in the quantized neural network model; re-inputting the test input data into the target neural network model, and calculating a second loss function value of the target neural network model for the test input data; calculating a loss difference value between the first loss function value and the second loss function value, and when it is determined that the loss difference value satisfies the loss difference threshold condition, determining the current processing node as a quantization-sensitive node; returning to execute the operation of sequentially obtaining the current processing node in each of the calculation nodes until the processing of all the calculation nodes is completed.

[0133] Optionally, the on-chip execution device of the quantized neural network model may further include: a branch network construction module, configured to, before replacing each target node with a branch network, determine a branch number range corresponding to each target node according to the pre- and post-quantization processing differences of each target node; according to the branch number range of each target node and the total number of calculation units in the target AI acceleration chip, determine the branch number corresponding to each target node, and construct a branch network corresponding to each target node according to each branch number.

[0134] Optionally, the branch network construction module can be specifically used for: respectively inputting the test input data into the target neural network model and the quantized neural network model; obtaining the original output feature map and the quantized output feature map of each target node in the target neural network model and the quantized neural network model; calculating a feature map difference value of each target node before and after quantization according to each of the original output feature maps and the quantized output feature maps; and determining a branch number range corresponding to each target node according to the feature map difference value of each target node.

[0135] Optionally, the branch network construction module can be specifically used for: in the branch number range of each target node, selecting a branch number that is an integer multiple of the total number of calculation units, and determining the branch number corresponding to each target node.

[0136] Optionally, the original operator graph acquisition module 310 may be specifically configured to: identify a model calculation scenario that matches the quantization neural network model; if it is determined that the model calculation scenario belongs to a high-precision demand calculation scenario, obtain an original operator graph that matches the quantization neural network model to be loaded into the target AI acceleration chip; wherein, the high-precision demand calculation scenario includes: a machine vision image classification scenario during an autonomous driving process, or a natural speech processing scenario during an intelligent question and answer process.

[0137] The on-chip execution device of the quantization neural network model provided by the embodiments of the present invention can execute the on-chip execution method of the quantization neural network model provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0138] Embodiment 4

[0139] Figure 4 FIG. shows a schematic structural diagram of an electronic device 410 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0140] As Figure 4 shown, the electronic device 410 includes at least one processor 420, and a memory communicatively connected to at least one processor 420, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein, the memory stores a computer program executable by at least one processor, and the processor 420 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 into the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other through a bus 450. The input / output (I / O) interface 460 is also connected to the bus 450.

[0141] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disc, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0142] The processor 420 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 420 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 420 executes the various methods and processes described above, such as the in-chip execution method of the quantization neural network model.

[0143] The method includes:

[0144] Obtain an original operator graph that matches the quantization neural network model to be loaded into the target artificial intelligence (AI) acceleration chip, where the quantization neural network model is obtained by post-training quantization of the target neural network model;

[0145] According to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip, perform a first reconstruction on the original computing operator graph to obtain a first operator graph;

[0146] In each computing node of the first operator graph, identify a first quantization-sensitive node set and a second quantization-sensitive node set, and determine a set number of target nodes in the second quantization-sensitive node set; where the quantization sensitivity of the first quantization-sensitive nodes is higher than that of the second quantization-sensitive nodes;

[0147] In the first operator graph, replace each computing node in the first quantization-sensitive node set with a non-quantization node, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weight and fit the calculation result of the replaced node;

[0148] According to the storage hierarchy characteristics of the target AI acceleration chip, perform a second reconstruction on the first operator graph to obtain a second operator graph;

[0149] Retrain the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model, and load the target quantization reconstruction model into the target AI acceleration chip for execution.

[0150] In some embodiments, the method for on-chip execution of a quantized neural network model can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the method for on-chip execution of the quantized neural network model described above can be performed. Alternatively, in other embodiments, the processor 420 can be configured to execute the method for on-chip execution of the quantized neural network model by any other suitable means (e.g., by means of firmware).

[0151] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein can be implemented in a computing system that includes backend components (such as, for example, a data server), or a computing system that includes middleware components (such as, for example, an application server), or a computing system that includes frontend components (such as, for example, a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (such as, for example, a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0156] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0157] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0158] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An on-chip execution method for a quantized neural network model, characterized in that, Including: Obtain an original operator graph that matches the quantized neural network model to be loaded into the target artificial intelligence (AI) acceleration chip, where the quantized neural network model is obtained by performing post-training quantization on the target neural network model; According to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip, perform a first reconstruction on the original operator graph to obtain a first operator graph; Among the computing nodes in the first operator graph, identify a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes, and determine a set number of target nodes in the second set of quantization-sensitive nodes; wherein, the quantization sensitivity of the first quantization-sensitive nodes is higher than that of the second quantization-sensitive nodes; In the first operator graph, replace each computing node in the first set of quantization-sensitive nodes with a non-quantized node, and replace each target node with a branch network, where the branch network includes multiple branch nodes with the same structure as the replaced node, and is used to weighted fit the computing result of the replaced node; According to the storage hierarchy characteristics of the target AI acceleration chip, perform a second reconstruction on the first operator graph to obtain a second operator graph; Retrain the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model, and load the target quantization reconstruction model into the target AI acceleration chip for execution.

2. The method according to claim 1, wherein Among the computing nodes in the first operator graph, identifying a first set of quantization-sensitive nodes and a second set of quantization-sensitive nodes includes: Calculate the influence degree of each computing node in the first operator graph on the loss function of the target neural network model respectively; Identify the first quantization-sensitive nodes whose influence degree on the loss function exceeds the first influence degree threshold to form a first set of quantization-sensitive nodes; Identify the second quantization-sensitive nodes whose influence degree on the loss function is greater than or equal to the second influence degree threshold and less than the first influence degree threshold to form a second set of quantization-sensitive nodes, where the first influence degree threshold is greater than the second influence degree threshold.

3. The method according to claim 2, wherein Calculating the influence degree of each computing node in the first operator graph on the loss function of the target neural network model respectively includes: Input the test input data into the target neural network model and the quantized neural network model respectively; Obtain the original output feature map and the quantized output feature map of each computing node in the target neural network model and the quantized neural network model; According to each original output feature map and the quantized output feature map, calculate the difference value of the feature map before and after quantization of each computing node as the influence degree of the loss function.

4. The method according to claim 2, wherein Calculating the influence degree of each computing node in the first operator graph on the loss function of the target neural network model respectively includes: Among each computing node, sequentially obtain the current processing node; Input the test input data into the target neural network model, and calculate the first loss function value of the target neural network model for the test input data; In the target neural network model, replace the original weight matrix corresponding to the current processing node with the quantized weight matrix corresponding to the current processing node in the quantized neural network model; Re - input the test input data into the target neural network model, and calculate the second loss function value of the target neural network model for the test input data; Calculate the loss difference value between the first loss function value and the second loss function value, and when it is determined that the loss difference value meets the loss difference threshold condition, determine the current processing node as a quantization - sensitive node; Return to execute the operations of successively obtaining the current processing node among all the calculation nodes until the processing of all the calculation nodes is completed.

5. The method according to claim 1, characterized in that, Before replacing each target node with a branch network, it further includes: Determine the branch number range corresponding to each target node according to the processing differences before and after quantization of each target node; According to the branch number range of each target node and the total number of computing units in the target AI acceleration chip, determine the branch number corresponding to each target node, and construct a branch network corresponding to each target node according to each branch number.

6. The method according to claim 5, characterized in that, Determine the branch number range corresponding to each target node according to the processing differences before and after quantization of each target node, including: Input the test input data into the target neural network model and the quantized neural network model respectively; Obtain the original output feature map and the quantized output feature map of each target node in the target neural network model and the quantized neural network model; Calculate the feature map difference value of each target node before and after quantization according to each original output feature map and quantized output feature map; Determine the branch number range corresponding to each target node according to the feature map difference value of each target node.

7. The method according to claim 5, wherein Determine the branch number corresponding to each target node according to the branch number range of each target node and the total number of computing units in the target AI acceleration chip, including: Among the branch number ranges of each target node, select the branch number that is an integer multiple of the total number of computing units to determine the branch number corresponding to each target node.

8. The method according to any one of claims 1 to 7, characterized in that, Obtain the original operator graph matching the quantized neural network model to be loaded into the target AI acceleration chip, including: Identify the model calculation scenario matching the quantized neural network model; If it is determined that the model calculation scenario belongs to a high - precision demand calculation scenario, obtain the original operator graph matching the quantized neural network model to be loaded into the target artificial intelligence AI acceleration chip; Among them, the high - precision demand calculation scenario includes: the machine vision image classification scenario during the automatic driving process, or the natural speech processing scenario during the intelligent question - answering process.

9. An on-chip execution device for quantizing a neural network model, characterized in that, It includes: An original operator graph acquisition module, configured to obtain the original operator graph matching the quantized neural network model to be loaded into the target artificial intelligence AI acceleration chip, where the quantized neural network model is obtained by post - training quantization of the target neural network model; A first operator graph reconstruction module, configured to perform a first reconstruction on the original operator graph according to the computing unit characteristics and / or instruction set characteristics in the target AI acceleration chip to obtain a first operator graph; A sensitive node recognition module, configured to recognize a first quantization sensitive node set and a second quantization sensitive node set among the computing nodes in a first operator graph, and determine a set number of target nodes in the second quantization sensitive node set; wherein, the quantization sensitivity of the first quantization sensitive nodes is higher than that of the second quantization sensitive nodes; A node processing module, configured to replace each computing node in the first quantization sensitive node set with a non-quantized node in the first operator graph, and replace each target node with a branch network, where the branch network includes a plurality of branch nodes having the same structure as the replaced node, and is used to weighted fit the calculation result of the replaced node; A second operator graph reconstruction module, configured to perform secondary reconstruction on the first operator graph according to the storage hierarchy characteristics of the target AI acceleration chip to obtain a second operator graph; A reconstructed model acquisition module, configured to retrain the original quantization reconstruction model corresponding to the second operator graph to obtain a target quantization reconstruction model, and load the target quantization reconstruction model into the target AI acceleration chip for execution.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the in-chip execution method of the quantization neural network model according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the in-chip execution method of the quantization neural network model according to any one of claims 1-8 when executed by a processor.

Images