In-device quantization method and device for neural network model, equipment and medium

Abstract

The application discloses an on-chip adaptation method and device for quantized neural network models, and a medium. The method comprises: obtaining an operator graph matched with a quantized neural network model to be loaded into a target AI acceleration chip; identifying a set of quantization-sensitive nodes in each calculation node of the operator graph and determining a set number of target nodes; and replacing each target node with a shunt network, setting the parallelism of each shunt network according to the multi-core distributed architecture inside the target AI acceleration chip; reconstructing the operator graph according to at least one of the storage level characteristics, the calculation unit characteristics and the instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph; and loading a quantized reconstructed model corresponding to the reconstructed operator graph into the target AI acceleration chip for retraining and execution. Through the technical scheme of the application, the calculation performance of the AI acceleration chip can be optimized on the basis of improving the precision of the quantized neural network model.

Description

Technical Field

[0001] This invention relates to the field of on-chip execution of neural network models, and more particularly to on-chip adaptation methods, apparatuses, devices, and media for quantizing neural network models. Background Technology

[0002] Because the computational and spatial storage complexity of neural network models places high demands on their computation and storage, more and more ultra-large models will experience latency and low throughput issues when deployed and applied.

[0003] In existing technologies, to effectively reduce the amount of data stored in AI (Artificial Intelligence) accelerator chips and improve their performance, low-bit model quantization is commonly used for model compression, such as fixed-point 4-bit, 3-bit, 2-bit, or even 1-bit quantization, before deployment on the AI ​​accelerator chip. This addresses the latency and low throughput issues encountered during model deployment. However, low-bit quantization limits the differentiated representation of different inputs, increasing the error of the neural network model and reducing its accuracy. Furthermore, existing quantization deployment methods generally employ static quantization, failing to consider the actual hardware characteristics of the AI ​​accelerator chip. Therefore, effectively correcting the accuracy loss in neural network models caused by low-bit quantization and fully leveraging the computational performance of the AI ​​accelerator chip after quantization deployment are pressing issues that need to be addressed. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for on-chip adaptation of a quantized neural network model, so as to achieve accurate adaptation between the quantized neural network model and the deployed AI acceleration chip while improving the accuracy of the quantized neural network model.

[0005] According to one aspect of the present invention, an on-chip adaptation method for quantizing neural network models is provided, comprising:

[0006] Obtain an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, wherein the quantized neural network model is obtained by post-training quantization of the target neural network model;

[0007] In each computation node of the operator graph, a set of quantization-sensitive nodes is identified, and a set number of target nodes are determined in the set of quantization-sensitive nodes.

[0008] In the operator graph, each target node is replaced by a split network, and the parallelism of each split network is set according to the multi-core distributed architecture inside the target AI acceleration chip.

[0009] The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector.

[0010] The operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph.

[0011] The quantized reconstruction model corresponding to the reconstruction operator graph is loaded into the target AI acceleration chip for retraining and execution.

[0012] According to another aspect of the present invention, an on-chip adapter for quantizing a neural network model is provided, the adapter comprising:

[0013] The operator graph acquisition module is used to acquire the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip. The quantized neural network model is obtained by post-training quantization of the target neural network model.

[0014] The node determination module is used to identify a set of quantization-sensitive nodes in each computation node of the operator graph, and to determine a set number of target nodes in the set of quantization-sensitive nodes.

[0015] The post-processing module is used to replace each target node with a split network in the operator graph, and to set the parallelism of each split network according to the multi-core distributed architecture inside the target AI acceleration chip.

[0016] The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector.

[0017] The operator graph reconstruction module is used to reconstruct the operator graph based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph.

[0018] The model retraining module is used to load the quantized reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the on-chip adaptation method for the quantized neural network model according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the on-chip adaptation method of the quantized neural network model according to any embodiment of the present invention.

[0024] The technical solution of this invention analyzes the sensitivity of each node in the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, identifies the set of quantization-sensitive nodes, and determines a set number of target nodes in the set of quantization-sensitive nodes. Then, each target node in the operator graph is replaced using a splitting network, and the parallelism of each splitting network is set according to the multi-core distributed architecture inside the target AI acceleration chip. Next, the operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. Furthermore, the quantized reconstructed model corresponding to the reconstructed operator graph is loaded into the target AI acceleration chip for retraining and execution. This solves the problem of accuracy loss in neural network models caused by low-bit quantization, improves the accuracy of the quantized neural network model, and, while improving the accuracy of the quantized neural network model, achieves optimal hardware adaptation of the quantized neural network model, maximizing the computing performance of the AI ​​acceleration chip.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1This is a flowchart of an on-chip adaptation method for a quantized neural network model according to Embodiment 1 of the present invention;

[0028] Figure 2a This is a flowchart of an on-chip adaptation method for a quantized neural network model according to Embodiment 2 of the present invention;

[0029] Figure 2b This is a schematic diagram of a top-K mechanism-based traffic splitting network according to Embodiment 2 of the present invention;

[0030] Figure 2c This is a schematic diagram of the structure of an original quantization reconstruction model provided in Embodiment 2 of the present invention;

[0031] Figure 2d This is a schematic diagram of the process of retraining an original quantization reconstruction model according to Embodiment 2 of the present invention;

[0032] Figure 2e This is a flowchart illustrating an optional on-chip adaptation method for a quantized neural network model according to Embodiment 2 of the present invention.

[0033] Figure 3 This is a schematic diagram of the structure of an on-chip adapter for a quantized neural network model according to Embodiment 3 of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the on-chip adaptation method of the quantized neural network model according to an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," "target," "initial," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] Figure 1 This is a flowchart illustrating an on-chip adaptation method for quantizing a neural network model, as provided in Embodiment 1 of the present invention. This embodiment is applicable to low-bit quantization of neural network models. The method can be executed by an on-chip adaptation device for quantizing the neural network model. This on-chip adaptation device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0039] S110. Obtain an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, wherein the quantized neural network model is obtained by post-training quantization of the target neural network model.

[0040] Here, the target AI acceleration chip can refer to an AI acceleration chip that carries a quantized neural network model. An operator graph can refer to a graph containing the various operators in the quantized neural network model and the connections between them. Each operator is used to execute a defined computational logic, such as addition, subtraction, or convolution.

[0041] Quantization refers to the process of approximating original data with a changed precision. For example, if a quantization standard is to quantize all data between 0.3 and 0.4 to 0.3, then if the data is 0.35, the corresponding quantized data is 0.3. Post-training quantization refers to quantizing the training of a neural network model. The target neural network model can refer to the neural network model that needs to be post-training quantized, such as a dense neural network model.

[0042] S120. In each computation node of the operator graph, identify the set of quantization-sensitive nodes, and determine a set number of target nodes in the set of quantization-sensitive nodes.

[0043] In this context, a computation node refers to a processing unit in the operator graph used to implement a given data processing function. It can be understood as the computation operators included in the operator graph, such as addition or convolution operators. Each computation operator has a set weight coefficient, which can be obtained through iterative training during the model training process.

[0044] In this context, quantization-sensitive nodes refer to computational nodes that have a high impact on the loss function of the target neural network model, i.e., computational nodes that cause severe losses in the loss function of the target neural network model. The set of quantization-sensitive nodes refers to a collection of quantization-sensitive nodes. Target nodes refer to the quantization-sensitive nodes selected for subsequent operations. The number of target nodes can be selected based on the actual application. For example, if the set of quantization-sensitive nodes contains ten quantization-sensitive nodes, three quantization-sensitive nodes can be initially selected as target nodes to avoid wasting computational resources due to too many target nodes. If more target nodes are needed in subsequent operations, the number of target nodes can be increased.

[0045] The process of post-training quantization of the target neural network model to obtain a quantized neural network model involves quantizing the weight coefficients of each computation node in the target neural network model to reduce data storage. For example, if the data precision of the weight coefficients of each computation node in the target neural network model is float32, after post-training quantization, the data precision of the quantized neural network model can be int8, int4, int2, or int1. It is understood that through the above post-training quantization process, compared to the target neural network model, the computational precision of each computation node in the quantized neural network model will suffer a computational loss. The purpose of this embodiment is to find the computational nodes with greater computational loss as quantization-sensitive nodes among all computational nodes, and to optimize the quantized neural network model based on these quantization-sensitive nodes.

[0046] S130. In the operator graph, each target node is replaced by a splitting network, and the parallelism of each splitting network is set according to the multi-core distributed architecture inside the target AI acceleration chip. The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to guide the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and sum and output the output results of each target splitting unit according to the weight of the splitting selector.

[0047] In this context, a splitting network can refer to splitting different input feature maps into one or more splitting units for different types of splitting-specific feature quantization processing, and then matching the processed input feature maps according to the weight coefficients of the splitting units, and finally summing the output results of each target splitting unit according to the weights of the splitting selector.

[0048] The input quantization calculation methods in different splitting units can differ. The splitting selector can perform splitting selection based on different input feature maps. A splitting unit can quantize the input feature map and then process it using its own weight coefficients, following the same calculation logic as the node being replaced. A summing unit can sum the outputs of each target splitting unit according to the weights of the splitting selector.

[0049] It is understandable that after quantizing the weight coefficient of a certain computation node in the operator graph corresponding to the target neural network model, that computation node may only have a good computational effect on the input feature map of a specific data range. For example, it may only have a good computational effect on the input feature map that takes a wide range of values ​​of 0 or 1, or it may only have a good computational effect on the input feature map that takes a wide range of values ​​greater than 0.5.

[0050] Based on this, the inventors creatively proposed to construct multiple split units for the same computing node A. The computing logic of each split unit is the same as that of computing node A, for example, they are all convolution computing units. However, the multiple split units are obtained by using different quantization methods during post-training quantization (that is, different split units have different weight coefficients). Thus, different split units can have different quantization effects on input feature maps with different numerical ranges. Correspondingly, by designing a flow selector, input feature maps with different numerical ranges are directed to one or more different flow units. Simultaneously, the summation weights of the directed flow units are set, and the directed flow units then perform a matching quantization process on the input feature maps to make them closer to the numerical range that the flow unit can optimize. Then, the flow unit processes the input feature maps according to the computational logic matched with the computational node A, using its own weight coefficients, and obtains the output results. Finally, the summation unit sums the output results of each flow unit according to the weights assigned by the flow selector, resulting in a more accurate processing result.

[0051] It should be noted that in an extreme case, if the split selector selects only one split unit to process the input feature map, the summation unit can directly output the output of that split unit.

[0052] In this context, "multi-core distributed architecture" refers to the distribution architecture of computing cores within the target AI acceleration chip. "Parallelism" refers to the number of computations performed in parallel by the offloading network. For example, if the number of computing cores in the multi-core distributed architecture exceeds the number of offloading networks, a number of cores matching the number of offloading networks can be randomly selected as the parallelism of the offloading network. If the number of computing cores in the multi-core distributed architecture is less than the number of offloading networks, the offloading networks can be regrouped according to the number of computing cores, and the parallelism of the offloading networks can be set accordingly. Specifically, if the number of computing cores is 5 and the number of offloading networks is 8, an even number of computing cores, such as 4, can be randomly selected, and the parallelism of the offloading network can be set to 4.

[0053] Therefore, by replacing the target node with a split network, different quantization methods can be selected for computation based on different input feature maps, thus effectively solving the problem of accuracy loss in neural network models caused by low-bit quantization.

[0054] It needs to be emphasized again that the essence of this application is to replace the quantization-sensitive computation nodes in the operator graph corresponding to the neural network model with a splitting network. Then, by selecting the output results of one or more splitting units that are applicable to different numerical ranges of the input feature map for weighted calculation, the computational accuracy of the output results of a single computation node can be corrected, thereby improving the model accuracy of the quantized neural network model.

[0055] S140. Based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip, the operator graph is reconstructed to obtain a reconstructed operator graph.

[0056] The storage level characteristics may include: whether the target AI acceleration chip is a multi-level storage; the number of storage levels when the target AI acceleration chip is a multi-level storage and the memory type of each storage level (e.g., Flash, DRAM or Cache).

[0057] In this embodiment, a storage level reconstruction strategy corresponding to different storage level characteristics can be pre-defined. For example, when the storage level is 1 and the memory type is Flash, the combination of quantization operators, non-quantization operators, and quantization operators in the operator graph is uniformly reconstructed into a single quantization operator. Furthermore, after obtaining the storage level characteristics of the target AI acceleration chip, the operator graph can be reconstructed a second time according to the storage level reconstruction strategy satisfied by the storage level characteristics to obtain a reconstructed operator graph.

[0058] The computing unit characteristics refer to the features of each computing unit in the target AI acceleration chip. These computing units can be understood as Arithmetic and Logic Units (ALUs) within the target AI acceleration chip, which are the smallest units in the chip used to perform logical operations. The target AI acceleration chip may include one or more computing cores, and each computing core contains one or more ALUs.

[0059] In this embodiment, the characteristics of the computing unit may include the number of computing units, the data bit width that each computing unit can compute (e.g., float32 or int16), the computing core to which each computing unit belongs, the parallelism between computing units, and the efficient operation operator corresponding to each computing unit. Here, an efficient operation operator can be understood as an operator that can be efficiently executed by a computing unit after being configured there.

[0060] An instruction set refers to a set of commands used to instruct the hardware in a target AI acceleration chip to perform specified operations or processing functions. The instruction set can be a RISC instruction set, a RISC-V instruction set, or an ARM instruction set, etc., and this embodiment does not impose any limitations on this.

[0061] Instruction set characteristics can be understood as the computational performance parameters of each instruction in the target AI acceleration chip, determined through testing after loading the instruction set into the chip. Examples include parameters such as computation speed and memory usage.

[0062] Understandably, by analyzing the computational performance parameters of each instruction in the target AI accelerator chip, a chip-friendly instruction set matching the target AI accelerator chip can be selected from the entire instruction set. By running the instructions from the chip-friendly instruction set in the target AI accelerator chip, the target AI accelerator chip can achieve better hardware computing performance.

[0063] Reconstructing an operator graph can refer to the operator graph obtained by reconstructing the operators and their connections in the operator graph based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip.

[0064] S150. Load the quantized reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution.

[0065] Here, the quantization reconstruction model can refer to a quantization neural network model that matches the reconstruction operator graph. Retraining can refer to retraining the loss function of the quantization reconstruction model using the target neural network model to obtain a target quantization reconstruction model that matches the quantization neural network model.

[0066] The technical solution of this invention analyzes the sensitivity of each node in the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, identifies the set of quantization-sensitive nodes, and determines a set number of target nodes in the set of quantization-sensitive nodes. Then, each target node in the operator graph is replaced using a splitting network, and the parallelism of each splitting network is set according to the multi-core distributed architecture inside the target AI acceleration chip. Next, the operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. Furthermore, the quantized reconstructed model corresponding to the reconstructed operator graph is loaded into the target AI acceleration chip for retraining and execution. This solves the problem of accuracy loss in neural network models caused by low-bit quantization, improves the accuracy of the quantized neural network model, and, while improving the accuracy of the quantized neural network model, achieves optimal hardware adaptation of the quantized neural network model, maximizing the computing performance of the AI ​​acceleration chip.

[0067] Example 2

[0068] Figure 2a This is a flowchart of an on-chip adaptation method for a quantized neural network model provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Specifically, this embodiment refines the operation of identifying a set of quantization-sensitive nodes in each computation node of the operator graph. Specifically, it may include: calculating the influence of each computation node in the operator graph on the loss function of the target neural network model; identifying quantization-sensitive nodes whose loss function influence exceeds the target influence threshold, and forming a set of quantization-sensitive nodes.

[0069] like Figure 2a As shown, the method includes:

[0070] S210. Identify the model computation scenario that matches the quantized neural network model.

[0071] Here, "model computation scenario" can refer to the computational scenario in which a quantifiable neural network model is actually applied. For example, based on the accuracy requirements of the computation scenario, model computation scenarios can be divided into high-precision computation scenarios with high accuracy requirements and low-precision computation scenarios with low accuracy requirements.

[0072] S220. If it is determined that the model calculation scenario belongs to a high-precision demand calculation scenario, then obtain the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip.

[0073] The high-precision computing scenarios include: machine vision image classification in autonomous driving, or natural speech processing in intelligent question answering.

[0074] S230. Calculate the influence of each computation node in the operator graph on the loss function of the target neural network model.

[0075] Among them, the impact of the loss function can refer to the degree of influence of each computation node in the operator graph on the loss function of the target neural network model.

[0076] In an optional implementation, calculating the influence of each computation node in the operator graph on the loss function of the target neural network model may include: inputting test input data into the target neural network model and the quantized neural network model respectively; obtaining the original output feature map and the quantized output feature map of each computation node in the target neural network model and the quantized neural network model; and calculating the feature map difference value of each computation node before and after quantization as the influence of the loss function based on the original output feature map and the quantized output feature map.

[0077] In this context, test input data can refer to open-source standard data used to test the quantized neural network model. The original output feature map can refer to the output feature map corresponding to each computation node when the test input data is input into the target neural network model. The quantized output feature map can refer to the output feature map corresponding to each computation node when the test input data is input into the quantized neural network model. Therefore, by evaluating the difference in feature maps of computation nodes before and after quantization, this difference can be used as the influence of the loss function, providing a valid basis for subsequent operations.

[0078] In another optional implementation, calculating the influence of each computation node in the operator graph on the loss function of the target neural network model may include: sequentially obtaining the current processing node in each computation node; inputting test input data into the target neural network model and calculating the first loss function value of the target neural network model for the test input data; replacing the original weight matrix in the target neural network model corresponding to the current processing node with the quantized weight matrix in the quantized neural network model corresponding to the current processing node; re-inputting the test input data into the target neural network model and calculating the second loss function value of the target neural network model for the test input data; calculating the loss difference value between the first loss function value and the second loss function value, and determining the current processing node as a quantization sensitive node when the loss difference value satisfies the loss difference threshold condition; returning to execute the operation of sequentially obtaining the current processing node in each computation node until the processing of all computation nodes is completed.

[0079] Here, the first loss function value can refer to the loss function value output by the target neural network model for the test input data. The second loss function value can refer to the loss function value output by the quantized neural network model for the test input data. The loss difference threshold condition can refer to the numerical value used to evaluate the loss difference value; for example, it can be a value within a set range. Therefore, by evaluating the loss difference value before and after quantization processing at the current processing node, the current processing node whose loss difference value meets the loss difference threshold condition can be designated as a quantization-sensitive node.

[0080] S240. Identify quantization-sensitive nodes whose influence of the loss function exceeds the target influence threshold, and form a set of quantization-sensitive nodes.

[0081] The target impact threshold can refer to a pre-set value used to evaluate the impact of the loss function.

[0082] S250. Determine a set number of target nodes in the set of quantized sensitive nodes.

[0083] S260. In the operator graph, each target node is replaced by a split network, and the parallelism of each split network is set according to the multi-core distributed architecture inside the target AI acceleration chip.

[0084] The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weights of the splitting selector.

[0085] Optionally, the splitting selector includes: a fully connected layer based on a set attention mechanism; the attention mechanism includes: a Softmax mechanism or a TopK mechanism; each splitting unit is used, after being selected by the splitting selector, to quantize the current input features by matching the splitting weights and matching quantization methods, and then to obtain the calculation result by using the calculation logic matching the replaced node.

[0086] If the attention mechanism is TopK and K is 1, then no summation is performed, and the output of the current splitting unit is used as the final output. The Softmax mechanism or TopK mechanism can be selected based on actual needs.

[0087] The matching and splitting weights for each splitting unit need to be calculated based on different attention mechanisms. For example, if the attention mechanism is the Softmax mechanism, then the matching and splitting weights for each splitting unit... If the attention mechanism is a TopK mechanism, then the weights of the matching and splitting of each splitting unit are... The activation function expression for this splitting network is: in, Input the feature values ​​of the current L layer. denoted by b, where b represents the number of splitting units, L represents the neural network layer containing the target node being replaced by the splitting network, and W represents the weight of the splitting selector.

[0088] like Figure 2b The diagram shows the top-K mechanism's splitting network structure when K is 2. Specifically, when K is 2, the number of splitting units selected is 2. The two selected splitting units quantize the current input features using their own weight coefficients and the matched quantization method. Then, the summing unit sums the outputs of each target splitting unit according to the weights of the splitting selector.

[0089] like Figure 2c The diagram shows a simplified structural representation of the original quantization reconstruction model with three nodes. Specifically, if the first node in the quantization neural network model is identified as the target node, a splitting network based on the Softmax mechanism is used to replace the first node, thereby completing the construction of the original quantization reconstruction model.

[0090] In an optional implementation, before replacing each target node with a splitting network in the operator graph, the method may further include: determining the range of the number of splitting units corresponding to each target node based on the difference in processing before and after quantization; determining the number of splitting units corresponding to each target node based on the range of the number of splitting units for each target node and the total number of computing units in the target AI acceleration chip; and constructing a splitting network corresponding to each target node according to the number of splitting units.

[0091] The difference in processing before and after quantization refers to the difference in feature maps of each target node before and after quantization. The range of the number of splitting units refers to the range of the number of splitting units corresponding to each target node. For example, if the difference in processing before and after quantization of a target node is small, the range of the number of splitting units corresponding to the target node is small; if the difference in processing before and after quantization of a target node is large, the range of the number of splitting units corresponding to the target node is large. This ensures that the calculation results of the subsequently constructed branch network are closer to the calculation results of the target node before quantization, thus improving the accuracy of the original quantization reconstruction model.

[0092] The number of offloading units can refer to the number of offloading units corresponding to the target node within the range of offloading unit quantity. The number of offloading units can be set according to actual application requirements. For example, if the number of offloading units of the target node is in the range of 6-10, then the number of offloading units can be selected as any number within the range of 6-10. The total number of computing units can refer to the number of computing units contained in the target AI acceleration chip.

[0093] In an optional implementation, determining the range of the number of splitting units corresponding to each target node based on the processing differences before and after quantization can include: inputting test input data into the target neural network model and the quantization neural network model respectively; obtaining the original output feature map and the quantized output feature map of each target node in the target neural network model and the quantization neural network model; calculating the feature map difference value of each target node before and after quantization based on the original output feature map and the quantized output feature map; and determining the range of the number of splitting units corresponding to each target node based on the feature map difference value of each target node.

[0094] Therefore, by calculating the feature map difference between the original output feature map and the quantized output feature map of the target node in the target neural network model and the quantized neural network model, we can determine the range of the number of branching units corresponding to each target node, providing an effective basis for the subsequent construction of branch networks.

[0095] In another optional implementation, determining the number of splitting units corresponding to each target node based on the range of the number of splitting units for each target node and the total number of computing units in the target AI acceleration chip can include: selecting a number of splitting units that is an integer multiple of the total number of computing units from the range of the number of splitting units for each target node, and determining this as the number of splitting units corresponding to each target node. Specifically, if the total number of computing units is 2 and the range of the number of splitting units is 5-10, then any number from the integer multiples of the total number of computing units of 2, 4, 6, 8, and 10 that falls within the range of 5-10 can be selected. For example, either 6 or 8 can be used as the number of splitting units corresponding to each target node. This embodiment does not impose any limitation on this.

[0096] Of course, in addition to selecting a number of splitting units that is an integer multiple of the total number of computing units as the number of splitting units corresponding to each of the target nodes, a number of splitting units that is a divisor of the total number of computing units can also be selected as the number of splitting units corresponding to each of the target nodes, etc.

[0097] By setting the above parameters to establish an integer ratio between the number of computing units and the number of shunt units, each computing unit can achieve optimal computing performance when running the target quantization reconstruction model.

[0098] S270. Based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip, the operator graph is reconstructed to obtain a reconstructed operator graph.

[0099] S280. Load the quantized reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution.

[0100] It is worth noting that, in this embodiment of the invention, the quantization reconstruction model corresponding to the reconstruction operator graph is loaded into the target AI acceleration chip for retraining and execution. Specifically, this can be achieved by: obtaining the target neural network model as the teacher model, and sequentially obtaining a current training sample and inputting it into the original quantization reconstruction model and the teacher model; obtaining the quantization model output results of each branch network in the original quantization reconstruction model for the current training sample, and the teacher model output results of each target node in the teacher model for the current training sample; adjusting the parameters of the original quantization reconstruction model based on the differences between the output results of each quantization model and the matched output results of each teacher model. The process of sequentially obtaining a current training sample and inputting it into the original quantization reconstruction model and the teacher model is repeated until the end-of-iteration condition is met, thus obtaining the target quantization reconstruction model.

[0101] The teacher model can refer to a control model used to adjust the original quantization reconstruction model. In this embodiment of the invention, the target neural network model is used as the teacher model. The end-of-iteration condition can refer to the condition under which the retraining of the target quantization reconstruction model ends. For example, it can be that the target quantization reconstruction model meets a set accuracy condition, or that the number of iterations meets a set iteration threshold, etc.

[0102] In an optional implementation, adjusting the parameters of the original quantization reconstruction model based on the differences between the outputs of each of the quantization models and the outputs of the matched teacher models may include: calculating a first type of loss function based on the differences between the outputs of each of the quantization models and the outputs of the matched teacher models; calculating a second type of loss function based on the differences between the final output of the original quantization reconstruction model for the current training sample and the labeled output of the current training sample; calculating a target loss function based on the first type of loss function, the second type of loss function, and the loss function weight allocation method; and using the target loss function to adjust the parameters of the original quantization reconstruction model.

[0103] The first type of loss function can refer to the loss function calculated based on the difference between the output results of each branch network in the original quantization reconstruction model for the current training sample and the output results of each target node in the teacher model for the current training sample. The second type of loss function can refer to the loss function calculated based on the difference between the final output result of the original quantization reconstruction model for the current training sample and the labeled result of the current training sample.

[0104] The specific calculation formulas for the first and second type of loss functions can be adopted according to different methods. For example, if the cross-entropy (CE) method is used, the expression for the loss function can be:

[0105] If the Mean Squared Error (MSE) method is used, the expression for the loss function can be:

[0106] If the KL (Kullback-Leibler) method is used, the expression for the loss function can be:

[0107] The target loss function can refer to the final loss function obtained based on the first type of loss function, the second type of loss function, and the weighting method of the loss functions. For example, the formula for calculating the target loss function can be: Where l represents the number of first-type loss functions, which is also the number of split networks in the original quantization reconstruction model. org This represents the second type of loss function. Let β represent the weight of the first type of loss function, β represent the weight of the second type of loss function, and γ represent the weight of the first type of loss function. The weight allocation method can refer to the weight allocation method of both the first and second type of loss functions. For example, the weight allocation method can be that the weight γ of the first type of loss function is 0.4, and the weight β of the second type of loss function is 0.6; or it can be that the weight γ of the first type of loss function is 0.5, and the weight β of the second type of loss function is 0.5. This embodiment of the invention does not impose any limitations on this.

[0108] like Figure 2d The diagram illustrates the retraining process of the original quantization reconstruction model. Specifically, the splitting network updates the parameter values ​​(typically weight parameters) of its splitting units, and the previously trained samples are input into the updated splitting network and the original quantization reconstruction model. Based on the differences between the outputs of each quantization model and the outputs of the matched teacher models, the target loss function is calculated, and the parameters of the original quantization reconstruction model are adjusted using the target loss function. If the retrained original quantization reconstruction model still does not meet the termination iteration condition, the process returns to updating the number of splitting units in the splitting network until the termination iteration condition is met and the target quantization reconstruction model is obtained.

[0109] In an optional implementation, after loading the quantization reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution, the method may further include: detecting whether the trained target quantization reconstruction model meets the preset computational accuracy conditions; if not, then retraining to obtain a new target quantization reconstruction model after readjusting at least one hyperparameter of the target quantization reconstruction model; wherein the hyperparameter includes at least one of the following: the number of target nodes determined in the quantization sensitive node set, the number of splitting units included in the splitting network, and the loss function weight allocation method.

[0110] Specifically, a higher number of target nodes and splitting units will result in a more accurate target quantization reconstruction model. However, considering the computational burden on hardware, a large number of target nodes and splitting units is typically not chosen initially. Therefore, if the target quantization reconstruction model does not meet the preset accuracy requirements, the number of target nodes and splitting units can be appropriately increased, and a new round of retraining can be performed until the target quantization reconstruction model meets the preset accuracy requirements. Furthermore, the weight allocation method of the loss function also affects the accuracy of the target quantization reconstruction model. Therefore, if the target quantization reconstruction model does not meet the preset accuracy requirements, the weight allocation method of the loss function can be adaptively adjusted until the target quantization reconstruction model meets the preset accuracy requirements.

[0111] The technical solution of this invention involves obtaining an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip when the model computation scenario is a high-precision computation scenario. Then, the influence of each computation node in the operator graph on the loss function of the target neural network model is calculated, and quantized sensitive nodes whose loss function influence exceeds the target influence threshold are identified, forming a set of quantized sensitive nodes. Next, a set number of target nodes are determined from the set of quantized sensitive nodes. In the operator graph, each target node is replaced using a splitting network, and the multi-core distributed processing of the target AI acceleration chip is considered. The architecture is designed to set the parallelism of each branch network. Further, based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip, the operator graph is reconstructed to obtain a reconstructed operator graph. Finally, the quantized reconstructed model corresponding to the reconstructed operator graph is loaded into the target AI acceleration chip for retraining and execution. This solves the problem of accuracy loss in neural network models caused by low-bit quantization, improves the accuracy of quantized neural network models, and achieves optimal hardware adaptation of the quantized neural network model while improving its accuracy, thus maximizing the computing performance of the AI ​​acceleration chip.

[0112] like Figure 2e The diagram shown is a flowchart illustrating an optional on-chip adaptation method for a quantized neural network model provided in an embodiment of the present invention.

[0113] Specifically, firstly, the target neural network model is post-trained and quantized to obtain a quantized neural network model. Then, sensitivity analysis is performed on each computation node in the operator graph matched with the quantized neural network model to identify quantization-sensitive nodes with high impact on the loss function of the target neural network model, and a set number of target nodes are determined. Further, the splitting network is constructed by updating the number of splitting units, updating the splitting selector, and inserting dummy nodes. Each target node in the operator graph matched with the quantized neural network model is replaced using the splitting network. Based on the multi-core distributed architecture inside the target AI acceleration chip, the parallelism of each splitting network is set. The operator graph is reconstructed based on at least one of the storage level characteristics, computation unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. The quantized reconstruction model corresponding to the reconstructed operator graph is used as the original quantized reconstruction model. Further, the target neural network model is obtained as the teacher model, and sequentially... A current training sample is input into the original quantization reconstruction model and the teacher model. The outputs of each branch network in the original quantization reconstruction model for the current training sample, and the outputs of each target node in the teacher model for the current training sample, are obtained. Based on the differences between the quantization model outputs and the matched teacher model outputs, the parameters of the original quantization reconstruction model are adjusted to obtain the target quantization reconstruction model. Finally, the computational accuracy of the target quantization reconstruction model is judged. If the target quantization reconstruction model does not meet the preset computational accuracy conditions, the branch networks are reconstructed, and the target nodes in the quantization neural network model are replaced with the reconstructed branch networks to obtain the original quantization reconstruction model. After readjusting at least one hyperparameter of the target quantization reconstruction model, it is retrained to obtain a new target quantization reconstruction model. If the target quantization reconstruction model meets the preset computational accuracy conditions, the target quantization reconstruction model is output and corresponding model optimization deployment is performed. Thus, a target quantization reconstruction model that meets the preset computational accuracy conditions can be obtained.

[0114] Example 3

[0115] Figure 3 This is a schematic diagram of the structure of an on-chip adapter for a quantized neural network model provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an operator graph acquisition module 310, a node determination module 320, a post-processing module 330, an operator graph reconstruction module 340, and a model retraining module 350;

[0116] The operator graph acquisition module 310 is used to acquire an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip. The quantized neural network model is obtained by post-training quantization of the target neural network model.

[0117] The node determination module 320 is used to identify a set of quantization-sensitive nodes in each computation node of the operator graph, and determine a set number of target nodes in the set of quantization-sensitive nodes.

[0118] The post-processing module 330 is used to replace each target node with a split network in the operator graph, and set the parallelism of each split network according to the multi-core distributed architecture inside the target AI acceleration chip.

[0119] The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector.

[0120] The operator graph reconstruction module 340 is used to reconstruct the operator graph based on at least one of the storage level characteristics, computing unit characteristics and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph.

[0121] The model retraining module 350 is used to load the quantized reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution.

[0122] The technical solution of this invention analyzes the sensitivity of each node in the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, identifies the set of quantization-sensitive nodes, and determines a set number of target nodes in the set of quantization-sensitive nodes. Then, each target node in the operator graph is replaced using a splitting network, and the parallelism of each splitting network is set according to the multi-core distributed architecture inside the target AI acceleration chip. Next, the operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. Furthermore, the quantized reconstructed model corresponding to the reconstructed operator graph is loaded into the target AI acceleration chip for retraining and execution. This solves the problem of accuracy loss in neural network models caused by low-bit quantization, improves the accuracy of the quantized neural network model, and, while improving the accuracy of the quantized neural network model, achieves optimal hardware adaptation of the quantized neural network model, maximizing the computing performance of the AI ​​acceleration chip.

[0123] Optionally, the node determination module 320 may specifically include: a loss function influence calculation unit and a quantization sensitive node set generation unit;

[0124] The loss function influence calculation unit is used to calculate the influence of each calculation node in the operator graph on the loss function of the target neural network model.

[0125] The quantization sensitive node set generation unit is used to identify quantization sensitive nodes whose influence of the loss function exceeds the target influence threshold, and form a quantization sensitive node set.

[0126] Optionally, the loss function influence calculation unit can be used to: input test input data into the target neural network model and the quantized neural network model respectively; obtain the original output feature map and quantized output feature map of each computing node in the target neural network model and the quantized neural network model; and calculate the feature map difference value of each computing node before and after quantization as the loss function influence value based on the original output feature map and the quantized output feature map.

[0127] Optionally, the loss function impact calculation unit can be specifically used for: sequentially obtaining the current processing node in each of the computing nodes; inputting test input data into the target neural network model and calculating the first loss function value of the target neural network model for the test input data; replacing the original weight matrix in the target neural network model corresponding to the current processing node with the quantized weight matrix in the quantized neural network model corresponding to the current processing node; re-inputting the test input data into the target neural network model and calculating the second loss function value of the target neural network model for the test input data; calculating the loss difference value between the first loss function value and the second loss function value, and determining the current processing node as a quantization sensitive node when the loss difference value satisfies the loss difference threshold condition; returning to execute the operation of sequentially obtaining the current processing node in each of the computing nodes until the processing of all computing nodes is completed.

[0128] Optionally, the on-chip adaptation device for quantizing the neural network model may further include: a split network construction module, used to determine the range of the number of split units corresponding to each target node based on the difference in processing before and after quantization of each target node in the operator graph before replacing each target node with a split network; determine the number of split units corresponding to each target node based on the range of the number of split units of each target node and the total number of computing units in the target AI acceleration chip; and construct a split network corresponding to each target node according to the number of split units.

[0129] Optionally, the splitting network construction module can be used to: input test input data into the target neural network model and the quantized neural network model respectively; obtain the original output feature map and quantized output feature map of each target node in the target neural network model and the quantized neural network model; calculate the feature map difference value of each target node before and after quantization based on the original output feature map and the quantized output feature map; and determine the range of the number of splitting units corresponding to each target node based on the feature map difference value of each target node.

[0130] Optionally, the traffic splitting network construction module can be used to: select a number of traffic splitting units that is an integer multiple of the total number of calculation units from the range of the number of traffic splitting units for each of the target nodes, and determine the number of traffic splitting units corresponding to each of the target nodes respectively.

[0131] Optionally, the operator graph acquisition module 310 can be used to: identify the model calculation scenario that matches the quantized neural network model; if it is determined that the model calculation scenario belongs to a high-precision demand calculation scenario, then acquire the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip; wherein, the high-precision demand calculation scenario includes: machine vision image classification scenario in the process of autonomous driving, or natural speech processing scenario in the process of intelligent question answering.

[0132] The on-chip adaptation device for the quantized neural network model provided in this embodiment of the invention can execute the on-chip adaptation method for the quantized neural network model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0133] Example 4

[0134] Figure 4 A schematic diagram of an electronic device 410 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0135] like Figure 4As shown, the electronic device 410 includes at least one processor 420 and a memory, such as a read-only memory (ROM) 430 or a random access memory (RAM) 440, communicatively connected to the at least one processor 420. The memory stores computer programs executable by the at least one processor. The processor 420 can perform various appropriate actions and processes based on the computer program stored in the ROM 430 or loaded into the RAM 440 from storage unit 490. The RAM 440 may also store various programs and data required for the operation of the electronic device 410. The processor 420, ROM 430, and RAM 440 are interconnected via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0136] Multiple components in electronic device 410 are connected to I / O interface 460, including: input unit 470, such as keyboard, mouse, etc.; output unit 480, such as various types of monitors, speakers, etc.; storage unit 490, such as disk, optical disk, etc.; and communication unit 4100, such as network card, modem, wireless transceiver, etc. Communication unit 4100 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] Processor 420 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 420 performs the various methods and processes described above, such as the on-chip adaptation method for quantizing neural network models.

[0138] The method includes:

[0139] Obtain an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, wherein the quantized neural network model is obtained by post-training quantization of the target neural network model;

[0140] In each computation node of the operator graph, a set of quantization-sensitive nodes is identified, and a set number of target nodes are determined in the set of quantization-sensitive nodes.

[0141] In the operator graph, each target node is replaced by a split network, and the parallelism of each split network is set according to the multi-core distributed architecture inside the target AI acceleration chip.

[0142] The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector.

[0143] The operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph.

[0144] The quantized reconstruction model corresponding to the reconstruction operator graph is loaded into the target AI acceleration chip for retraining and execution.

[0145] In some embodiments, the on-chip adaptation method for quantizing a neural network model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 410 via ROM 430 and / or communication unit 4100. When the computer program is loaded into RAM 440 and executed by processor 420, one or more steps of the on-chip adaptation method for quantizing a neural network model described above can be performed. Alternatively, in other embodiments, processor 420 can be configured to perform the on-chip adaptation method for quantizing a neural network model by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0151] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0152] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for on-chip adaptation of a quantized neural network model, characterized in that, include: Obtain an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip, wherein the quantized neural network model is obtained by post-training quantization of the target neural network model; In each computation node of the operator graph, a set of quantization-sensitive nodes is identified, and a set number of target nodes are determined in the set of quantization-sensitive nodes. In the operator graph, each target node is replaced by a split network, and the parallelism of each split network is set according to the multi-core distributed architecture inside the target AI acceleration chip. The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector. The operator graph is reconstructed based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. The quantized reconstruction model corresponding to the reconstruction operator graph is loaded into the target AI acceleration chip for retraining and execution; In this process, identifying a set of quantization-sensitive nodes among the computational nodes in the operator graph includes: calculating the influence of each computational node in the operator graph on the loss function of the target neural network model; the influence of the loss function refers to the degree of influence of each computational node in the operator graph on the loss function of the target neural network model; identifying computational nodes whose influence of the loss function exceeds the target influence threshold as quantization-sensitive nodes, thus forming a set of quantization-sensitive nodes. The process of obtaining an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip includes: identifying the model computation scenario that matches the quantized neural network model; if it is determined that the model computation scenario belongs to a high-precision computation scenario, then obtaining an operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip; wherein, the high-precision computation scenario includes: machine vision image classification scenario in autonomous driving process, or natural speech processing scenario in intelligent question answering process.

2. The method according to claim 1, characterized in that, Calculate the influence of each computation node in the operator graph on the loss function of the target neural network model, including: The test input data is input into the target neural network model and the quantization neural network model, respectively; Obtain the original output feature map and the quantized output feature map of each computing node in the target neural network model and the quantized neural network model; Based on the original output feature maps and the quantized output feature maps, the difference in feature maps before and after quantization for each computing node is calculated as the influence of the loss function.

3. The method according to claim 1, characterized in that, Calculate the influence of each computation node in the operator graph on the loss function of the target neural network model, including: In each of the aforementioned computing nodes, the current processing node is obtained sequentially; The test input data is input into the target neural network model, and the first loss function value of the target neural network model for the test input data is calculated. Replace the original weight matrix corresponding to the current processing node in the target neural network model with the quantized weight matrix corresponding to the current processing node in the quantized neural network model; The test input data is re-inputted into the target neural network model, and the second loss function value of the target neural network model for the test input data is calculated. Calculate the loss difference between the first loss function value and the second loss function value, and when it is determined that the loss difference value meets the loss difference threshold condition, determine the current processing node as a quantization sensitive node; The process returns to the operation performed on each computing node, sequentially retrieving the operation of the current processing node until the processing of all computing nodes is completed.

4. The method according to claim 1, characterized in that, Before replacing each target node using a splitting network in the operator graph, the following steps are also included: Based on the differences in processing before and after quantization at each target node, determine the range of the number of splitting units corresponding to each target node. Based on the range of the number of distribution units for each target node and the total number of computing units in the target AI acceleration chip, the number of distribution units corresponding to each target node is determined, and a distribution network corresponding to each target node is constructed according to the number of distribution units.

5. The method according to claim 4, characterized in that, Based on the differences in processing before and after quantization at each target node, the range of the number of splitting units corresponding to each target node is determined, including: The test input data is input into the target neural network model and the quantization neural network model, respectively; Obtain the original output feature map and the quantized output feature map of each target node in the target neural network model and the quantized neural network model; Based on the original output feature maps and the quantized output feature maps, calculate the feature map difference value of each target node before and after quantization. Based on the feature map difference values ​​of each target node, determine the range of the number of splitting units corresponding to each target node.

6. The method according to claim 4, characterized in that, Based on the range of the number of offloading units for each target node and the total number of computing units in the target AI acceleration chip, determine the number of offloading units corresponding to each target node, including: Within the range of the number of splitting units for each target node, a number of splitting units whose value is an integer multiple of the total number of calculation units is selected and determined as the number of splitting units corresponding to each target node.

7. An on-chip adapter for quantizing a neural network model, characterized in that, include: The operator graph acquisition module is used to acquire the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip. The quantized neural network model is obtained by post-training quantization of the target neural network model. The node determination module is used to identify a set of quantization-sensitive nodes in each computation node of the operator graph, and to determine a set number of target nodes in the set of quantization-sensitive nodes. The post-processing module is used to replace each target node with a split network in the operator graph, and to set the parallelism of each split network according to the multi-core distributed architecture inside the target AI acceleration chip. The splitting network includes a splitting selector, multiple optional splitting units connected to the splitting selector, and summing units connected to each splitting unit. The splitting network is used to divert the feature map input to the splitting selector to at least one matching target splitting unit for quantization calculation, and to sum and output the output results of each target splitting unit according to the weight of the splitting selector. The operator graph reconstruction module is used to reconstruct the operator graph based on at least one of the storage level characteristics, computing unit characteristics, and instruction set characteristics of the target AI acceleration chip to obtain a reconstructed operator graph. The model retraining module is used to load the quantized reconstruction model corresponding to the reconstruction operator graph into the target AI acceleration chip for retraining and execution. The node determination module specifically includes: a loss function influence calculation unit and a quantization sensitive node set generation unit; the loss function influence calculation unit is used to calculate the influence of each computation node in the operator graph on the loss function of the target neural network model; the loss function influence refers to the degree of influence of each computation node in the operator graph on the loss function of the target neural network model; the quantization sensitive node set generation unit is used to identify computation nodes whose loss function influence exceeds the target influence threshold as quantization sensitive nodes, forming a quantization sensitive node set; Specifically, the operator graph acquisition module is used to: identify the model computation scenario that matches the quantized neural network model; if it is determined that the model computation scenario belongs to a high-precision computation scenario, then acquire the operator graph that matches the quantized neural network model to be loaded into the target AI acceleration chip; wherein, the high-precision computation scenario includes: machine vision image classification scenario in the process of autonomous driving, or natural speech processing scenario in the process of intelligent question answering.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the on-chip adaptation method for the quantized neural network model according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the on-chip adaptation method of the quantized neural network model according to any one of claims 1-6.

Images