Method, device, electronic device, and medium for accelerating neural network model inference

Abstract

To provide an acceleration method, a device, an electronic apparatus, and a medium for neural network model inference according to the present disclosure.SOLUTION: The method comprises: acquiring image training data, text training data or voice training data; determining a first neural network model as an object of acceleration; converting preset arithmetic of the first neural network model into first arithmetic for arithmetic logic simulation of target arithmetic to obtain a second neural network; performing quantization sense training on the second neural network model according to a preset bit width to obtain a third neural network model for quantization; and converting first arithmetic of the third neural network into target arithmetic to obtain an accelerated target neural network model corresponding to the first neural network model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to artificial intelligence technology, and in particular to a method, apparatus, electronic device, and medium for accelerating neural network model inference. [Background technology]

[0002] In the field of deep learning, functions such as softmax, sigmoid, and swish are very important parts of neural network models, but the calculation of these functions is complicated and the calculation cost is high, which slows down the inference speed of the neural network model. To solve this problem, related art generally simplifies these complicated calculations, for example, by combining multiple table lookups and performing arithmetic operations on the table lookup results to realize the calculation of softmax, thereby reducing the calculation cost. However, such a conventional simplification method reduces the model accuracy. Summary of the Invention [Problem to be solved by the invention]

[0003] In order to solve the technical problems such as the low model accuracy caused by the above simplification method, the present disclosure is presented. The embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a medium for accelerating neural network model inference. [Means for solving the problem]

[0004] A method for accelerating neural network model inference according to one aspect of an embodiment of the present disclosure includes the steps of: acquiring image training data, text training data, or voice training data; determining a first neural network model to be accelerated; converting a preset operation of a preset network layer in the first neural network model into a first operation for simulating an operation logic of a target operation to obtain a second neural network model; performing quantization sensing training on the second neural network model based on the image training data, the text training data, or the voice training data according to a preset bit width, which is a bit width that supports a neural network accelerator to process data, to obtain a quantized third neural network model; and converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0005] A method for accelerating neural network model inference according to another aspect of an embodiment of the present disclosure includes a step of acquiring processing target data, which is image data, text data, or audio data, and a step of determining output data corresponding to the processing target data based on an accelerated target neural network model corresponding to a first neural network model acquired in advance, wherein the target neural network model is acquired based on the method for accelerating neural network model inference according to any of the above embodiments.

[0006] A neural network model inference acceleration device according to a further aspect of an embodiment of the present disclosure includes a first acquisition module for acquiring image training data, text training data, or voice training data; a first determination module for determining a first neural network model to be accelerated; a first processing module for converting a preset operation of a preset network layer in the first neural network model into a first operation for simulating the operation logic of a target operation to obtain a second neural network model; a second processing module for performing quantization sensing training on the second neural network model based on the image training data, the text training data, or the voice training data according to a preset bit width, which is a bit width that supports a neural network accelerator to process data, to obtain a quantized third neural network model; and a third processing module for converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0007] A neural network model inference acceleration device according to a further aspect of an embodiment of the present disclosure includes a second acquisition module for acquiring processing target data, which is image data, text data, or audio data, and a fourth processing module for determining output data corresponding to the processing target data based on an accelerated target neural network model corresponding to a first neural network model acquired in advance, wherein the target neural network model is acquired based on a method for accelerating neural network model inference according to any of the above embodiments.

[0008] A computer-readable storage medium according to a further aspect of an embodiment of the present disclosure stores a computer program, the computer program executing a method for accelerating neural network model inference according to any of the above embodiments of the present disclosure.

[0009] An electronic device according to a further aspect of an embodiment of the present disclosure includes a processor and a memory for storing processor-executable commands, and the processor realizes a method for accelerating neural network model inference according to any of the above embodiments of the present disclosure by reading the executable commands from the memory and executing the commands.

[0010] A further aspect of an embodiment of the present disclosure is a computer program product, the instructions of which, when executed by a processor, implements the method for accelerating neural network model inference according to any of the above embodiments. Effect of the Invention

[0011] According to the above-mentioned embodiments of the present disclosure, the method, device, electronic device and medium for accelerating neural network model inference convert the complex preset operation in the neural network into a simple target operation, and simulate the operation logic of the target operation in the process of quantization sensing training, so that the error caused by the simplified target operation in the training process can be simulated, thereby ensuring that the accuracy of the model is not reduced under the condition of simplified operation of the target neural network model, and thus effectively solving the problem of low model accuracy caused by the simplification method of the prior art.

[0012] The following further describes in detail the technical solutions of the present disclosure with reference to figures and examples. [Brief description of the drawings]

[0013] [Figure 1] 1 is an exemplary application scenario of the method for accelerating neural network model inference according to the present disclosure. [Diagram 2] 1 is a flowchart of a method for accelerating neural network model inference according to one exemplary embodiment of the present disclosure. [Diagram 3]4 is a flowchart of a first operation according to an exemplary embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method for accelerating neural network model inference according to another exemplary embodiment of the present disclosure. [Diagram 5] 10 is a flowchart of a method for accelerating neural network model inference according to a further exemplary embodiment of the present disclosure. [Figure 6] 4 is a flowchart of a target calculation according to one exemplary embodiment of the present disclosure. [Figure 7] FIG. 2 is a structural schematic diagram of an accelerator for neural network model inference according to one exemplary embodiment of the present disclosure; [Figure 8] FIG. 7 is a structural schematic diagram of a second processing module 704 according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 13 is a structural schematic diagram of an accelerator for neural network model inference according to another exemplary embodiment of the present disclosure; [Figure 10] FIG. 13 is a structural schematic diagram of an accelerator for neural network model inference according to a further exemplary embodiment of the present disclosure; [Figure 11] FIG. 8 is a structural schematic diagram of a fourth processing module 802 according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 2 is a structural schematic diagram of an application embodiment of the electronic device of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. The described embodiments are only a part of the embodiments of the present disclosure, and are not all of the embodiments of the present disclosure, and the present disclosure is not limited by the exemplary embodiments described herein.

[0015] Unless otherwise specified, the relative arrangement of components and steps, formulas, and numerical values ​​described in these examples do not limit the scope of the present disclosure. Summary of the Disclosure

[0016] In the process of realizing the present disclosure, the inventors have found that in the field of deep learning, functions such as softmax, sigmoid, and swish are very important parts of neural network models, but the calculation of these functions is complicated and the calculation cost is high, which slows down the inference speed of the neural network model. To solve this problem, related art generally simplifies these complicated calculations, for example, by combining multiple table lookups and performing arithmetic operations on the table lookup results to realize the calculation of softmax, thereby reducing the calculation cost. However, such a conventional simplification method results in low model accuracy. Illustrative Description

[0017] FIG. 1 is an exemplary application scenario of the method for accelerating neural network model inference according to the present disclosure.

[0018] In target detection application scenarios of Transformer series neural networks, detection results are generally obtained by performing target detection on a detection target image using a target detection model based on Transformer.The encoder and decoder of the Transformer network architecture of the target detection model involve a multi-headed self-attention (MSA) network block, and the self-attention operation of the multi-headed self-attention network block involves a softmax function, and the operation of the softmax function includes an exponential operation. By executing the acceleration method of neural network model inference of the present disclosure using the acceleration device of neural network model inference of the present disclosure, the target detection model can be a first neural network model to be accelerated, and the exponential operation of the softmax layer in the first neural network model is converted into a first operation to obtain a second neural network model, and the first operation is used to simulate the operation logic of the target operation, and the target operation is a simplified operation, for example a table lookup operation, and quantization aware training is performed for the second neural network model according to a preset bit width based on the image training data. The quantization sensing training is performed to obtain a quantized third neural network model, and the preset bit width can be a bit width that supports the neural network accelerator to process data. After the quantization sensing training is completed, the first operation in the third neural network model is converted into a target operation to obtain an accelerated target neural network model corresponding to the first neural network model. The target neural network model realizes an exponential operation of a softmax layer based on the simplified target operation, thereby reducing the amount of calculation. The quantization sensing training process simulates the operation logic of the simplified target operation, thereby simulating the error caused by the simplified operation. The obtained target neural network model has a relatively high accuracy even with a relatively low amount of calculation, and can solve the problem of the relatively low accuracy of the traditional simplified operation model. Exemplary Methods

[0019] 2 is a flowchart of a method for accelerating neural network model inference according to an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices such as servers and terminals, and as shown in FIG. 2, the method includes steps 201 to 205.

[0020] In step 201, image training data, text training data or audio training data is obtained.

[0021] Here, the image training data may be image data of any scene, for example road environment image data collected in an automatic driving scene, medical image data, etc., and may be specifically set according to actual needs. The text training data may be text data of any scene, for example text data requiring any text semantic identification scene, and the voice training data may be voice data of any scene, for example voice data for voice control collected in a smart home, voice data for voice control in a vehicle, and may be specifically set according to actual needs. The specific data to be acquired may be determined based on the neural network model that is actually intended to accelerate. For example, if the neural network model that needs to accelerate is an automatic driving scene road element target detection model, the road environment image data acquired in advance is acquired as the image training data.

[0022] In step 202, a first neural network model to be accelerated is determined.

[0023] Here, the first neural network model may be a model of any application scene, and the first neural network model is a full-precision model obtained by training in advance. For example, when the target detection model of the road element in the automatic driving scene needs to be accelerated, the target detection model is the first neural network model that is the acceleration target. The first neural network model needs to match the training data obtained in step 201, and in actual application, the training data and the first neural network model can be stored in a certain storage space in advance, for example, stored correspondingly, or associated by an address, or the address of the training data and the first neural network model that is the acceleration target are obtained according to the user's specification, and the specific can be set according to actual needs, and is not limited in the present disclosure.

[0024] In step 203, the preset operation of the preset network layer in the first neural network model is converted into a first operation for simulating the operation logic of the target operation to obtain a second neural network model.

[0025] Here, the preset network layer and the preset operation of the preset network layer can be set according to actual needs, for example, when the first neural network model is a target detection model based on Transformer, the preset network layer can be a softmax layer, and the preset operation can be an exponential operation in softmax, or the preset network layer can be a self-attention network layer, and the preset operation is an exponential operation in the self-attention network layer. Also, for example, for a first neural network model with a sigmoid activation function, the preset network layer can be a sigmoid activation layer, and the preset operation can be an exponential operation in the sigmoid activation layer, and there is no specific limitation. The target calculation is a simplified calculation corresponding to the preset calculation, such as a table lookup operation, a linear fitting operation, etc., and the first calculation is used to simulate the calculation logic of the target calculation. For example, in the table lookup operation, the target subrange to which the input data of the lookup target table belongs is first determined, the table is looked up based on the target subrange to obtain the calculation results corresponding to both ends of the target subrange, and the target calculation result corresponding to the input data of the lookup target table is obtained using a linear interpolation method based on the calculation results corresponding to both ends; in the first calculation, the first subrange to which the first input data of the first calculation belongs is first determined, and the calculation results corresponding to both ends of the first subrange are respectively determined, and the first calculation result corresponding to the first input data is obtained using a linear interpolation method based on the calculation results corresponding to both ends, thereby simulating the calculation logic of the target calculation. Here, the determination of the calculation results corresponding to both ends of the first subrange may be obtained by real-time calculation or by table lookup, and can be specifically set according to actual needs.

[0026] In practical application, the preset network layer can include one or more network layers, and the preset algorithm can include one or more algorithms. For example, when a model includes not only a softmax function but also a sigmoid activation function, the preset network layer can include a normalization network layer in which the softmax function is located and an activation layer in which the sigmoid activation function is located, and the preset operation can include an exponential operation in softmax and an exponential operation in sigmoid, which can be specifically set according to actual needs, and is not limited in the present disclosure. Any preset operation with complex calculations of any preset network layer can be used to realize the acceleration of the model by using the method of the present disclosure.

[0027] Step 201 and steps 202-203 may be performed in any order.

[0028] In step 204, based on the image training data, the text training data, or the audio training data, quantization-sensitive training is performed on the second neural network model according to a preset bit-width, which is a bit-width that supports the neural network accelerator to process data, to obtain a quantized third neural network model.

[0029] Here, the preset bit width can be set according to actual needs, for example, 8 bits, 16 bits, 4 bits, 2 bits, 1 bit, etc., specifically, 8-bit integer type (int8). The quantization-aware training inserts a pseudo-quantization node into the model to simulate the rounding and clamping operations performed by the quantization model in the inference process, thereby improving the model's adaptability to quantized data in the training process and obtaining higher quantization model accuracy. The quantization objects can include weights, activation function outputs, gradients, etc., and can be specifically set according to actual needs. The quantization weights and activation function outputs can effectively reduce the memory occupation space of the model, and the quantization gradients can reduce the overhead of inverse calculation. Specifically, whether image training data, text training data, or voice training data is adopted as the training data can be determined based on the specific application scenario of the second neural network model or the first neural network model. For the determined model, the type of training data required for its quantization-sensitive training is determined to be one of the above image training data, text training data and voice training data. The bit width supported by the neural network accelerator for data processing is the bit width of the network parameters (e.g., weights), input data, output data, intermediate data (e.g., activation function output) supported by the neural network accelerator when implementing the inference calculation of the neural network based on the neural network accelerator. For example, if the neural network accelerator supports a bit width of int8, the input of each data processing module in the neural network accelerator needs to be int8 data, and the weights used in the calculation are also int8 data. Specifically, it can be set according to actual needs.The bit width supported by the full-precision neural network model is generally 64-bit floating-point number (float64) or 32-bit floating-point number (float32). In order to accelerate the inference speed of the neural network model based on the neural network accelerator, various data of the full-precision neural network model need to be quantized to the bit width supported by the neural network accelerator, so that the full-precision neural network model is quantized according to the bit width supported by the neural network accelerator to process data, and the quantized neural network model can be placed on the neural network accelerator to perform inference calculation, so as to achieve the purpose of accelerating the inference of the neural network model, and the memory space occupied by the low-bit quantized neural network model is smaller, so that the memory occupation of the inference process can be effectively reduced.

[0030] In practical applications, the quantization of the model weights, activation function outputs and gradients can be set to the same or different preset bit widths, specifically according to practical needs, and is not limited in this disclosure.

[0031] Optionally, in the process of quantization-sensitive training, the model forward / backward propagation calculation and the pseudo-quantization node calculation can be realized in floating-point calculation, and only need to follow the fixed-point correlation rules (e.g., rounding and clamping) of the preset bit width, and after the training is completed, it is quantized again to a true fixed-point model, which becomes the third neural network model.

[0032] For example, in the process of quantization sensing training, the floating-point number "0.1" represents the fixed-point number "1", and the floating-point number "0.2" represents the fixed-point number "2". By this analogy, the floating-point number and the fixed-point number have a certain multiple relationship, and in the training process, the fixed-point number rules are followed. For example, if the fixed-point number is "3.5", it is rounded up or down to an integer of "4" or "3", and the floating-point number "0.35" is converted to "0.4" or "0.3". Thus, the floating-point number that satisfies the fixed-point number rules is adopted to determine the network loss, and the loss caused by the simplification of the operation is simulated. After the training is completed, the floating-point model is converted into a fixed-point model by model numerical conversion, so that the model accuracy is not reduced.

[0033] In step 205, the first operation of the third neural network model is transformed into a target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0034] Here, the first operation is a simulation of the operation logic of the target operation, and after the quantization sensing training is completed, the first operation of the third neural network model can be directly converted into the target operation, and the converted target neural network model has the same accuracy as the third neural network model, thereby realizing that the accuracy of the model is not reduced even when the model calculation amount is reduced.

[0035] Illustratively, the softmax function is expressed as follows:

number

[0036] The method for accelerating neural network model inference according to this embodiment converts the complex preset operation in the neural network into a simple target operation, and simulates the operation logic of the target operation in the process of quantization sensing training, thereby simulating the error caused by the simplified target operation in the training process, thereby ensuring that the accuracy of the model is not reduced under the condition of simplified operation of the target neural network model, and thereby effectively solving the problem of low model accuracy caused by the simplification method of the prior art.

[0037] FIG. 3 is a flow chart of a first operation according to one exemplary embodiment of the present disclosure.

[0038] In one selectable example, the target operation is a table lookup operation, and the first operation may specifically include steps 301 to 303.

[0039] In step 301, a first subrange to which a first input data of a first operation belongs is determined.

[0040] Here, in the training process of the first neural network model, a first input range is determined according to the input data of the preset operation, the first input range is divided into a number of subranges, the operation results corresponding to the end points of each subrange are calculated and obtained, and the operation results corresponding to each subrange and the end points of each subrange are stored, or each subrange is stored, which can be specifically set according to actual needs. Thus, when the first operation needs to be performed, the first subrange to which the first input data belongs can be determined by comparing the first input data with the end points of each subrange.

[0041] In step 302, the calculation results respectively corresponding to both ends of the first sub-range are determined based on a preset scheme.

[0042] Here, the preset method may include two methods, namely, calculation and table lookup, and since the second neural network model is used for quantization sensing training, the calculation results corresponding to both ends of the first subrange may be calculated in real time during actual calculation during training, or the calculation results corresponding to the end points of each subrange may be calculated and stored in advance, and the calculation results corresponding to both ends of the first subrange may be directly obtained from the storage area during calculation. Regardless of which method is adopted, the accuracy of the first calculation simulating the calculation logic of the target calculation is not affected. In order to further reduce the amount of calculation in the quantization sensing training, the calculation results corresponding to both ends of the first subrange may be determined using the table lookup method.

[0043] In step 303, according to the first input data, the first sub-range, and the calculation results corresponding to both end points of the first sub-range, a first calculation result corresponding to the first input data is obtained by using a linear interpolation method.

[0044] Here, the linear interpolation method is an interpolation method in which the interpolation function is a first-order polynomial, and the specific interpolation principle is not described. In order to simulate the calculation logic of the table lookup operation, after determining the calculation results corresponding to both ends of the first sub-range, linear interpolation is performed on the first input data according to the calculation results corresponding to both ends of the first sub-range to obtain a first calculation result corresponding to the first input data.

[0045] The present disclosure provides a method for simulating a table lookup operation using a first operation in a second neural network model, thereby simulating errors due to table lookup operations in the process of quantization-sensitive training, and once training is completed, the model can achieve a relatively high accuracy on quantized data that is the same or similar to that of the first neural network model, and the model accuracy due to the table lookup operation is not significantly degraded.

[0046] FIG. 4 is a flowchart of a method for accelerating neural network model inference according to another exemplary embodiment of the present disclosure.

[0047] In one optional example, the target operation is a table lookup operation, and the method of the present disclosure may further include steps 401 to 404 after step 202 of determining a first neural network model to be accelerated.

[0048] In step 401, a first input range is determined based on the input data of the preset calculation in the training process of the first neural network model.

[0049] Here, in the process of obtaining the first neural network model through training, the input data of the preset operation can be recorded, and the range of the input data of the preset operation recorded in the training process can be combined to determine the first input range.

[0050] In actual application, since the first input range is a table lookup operation applied to the quantization sensing training process, the input data of the preset operation of the first neural network model is a floating-point number, and the first input range suitable for the quantization sensing training can be determined based on the floating-point input data. For example, in the second neural network model, the input of the first operation is a floating-point number "0.2" of a fixed-point number "2" simulating the preset bit width, and the range of the input data of the preset operation is "0.001-0.999", the first input range can be determined as "0.0-1.0", and can be specifically set according to actual needs.

[0051] In step 402, a first input range is segmented based on the preset segment number to obtain sub-ranges corresponding to the preset segment number.

[0052] Here, the number of preset segments can be set according to actual needs, for example, 100 segments, 50 segments, 20 segments, etc., and is not limited in the embodiments of the present disclosure.

[0053] For example, "0.0-1.0" can be divided into five sub-ranges, namely, "0.0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8, 0.8-1.0". The specific division manner of each sub-range can be set according to actual needs, and may be equal pitch division or non-equal pitch division, and is not limited to the embodiments of the present disclosure.

[0054] In step 403, for each subrange, a calculation result corresponding to each of the two end points of the subrange is determined based on a preset calculation.

[0055] Here, the calculation results of both ends of each subrange can be obtained based on a preset calculation, for example, an exponential calculation, and accurate calculation results of both ends of each subrange can be calculated based on the exponential calculation.

[0056] Step 404 stores the results of the calculations corresponding to each subrange and each endpoint of each respective subrange for use in a table lookup operation.

[0057] Here, after storing the calculation results corresponding to each end point of each subrange in advance, in the process of quantization sensing training, when performing the first operation, the first subrange to which the first input data of the first operation belongs is determined, and then the calculation results corresponding to both end points of the first subrange are determined based on table lookup, and linear interpolation is performed based on the calculation results of the both end points to obtain the first operation result corresponding to the first input data, thereby realizing the simulation of the calculation logic of the table lookup operation during the quantization sensing training, and ensuring that the model accuracy will not decrease after being converted into a subsequent target operation.

[0058] The present disclosure segments the input data range of the first operation, thereby determining and storing each sub-range used in the table lookup operation and the operation result corresponding to each sub-range, so that in the process of quantization sensing training, for the first operation, the operation results of both end points of the first sub-range to which the first input data belongs are obtained through the table lookup operation, and then the first operation result corresponding to the first input data is obtained based on linear interpolation, thereby reducing the calculation amount of the quantization sensing training process and improving the training efficiency.

[0059] In one alternative example, determining 202 the first neural network model to be accelerated includes step 2021.

[0060] In step 2021, a pre-constructed neural network is trained based on preset training data to obtain a first neural network model.

[0061] Here, the preset training data can be image training data, text training data or voice training data, specifically determined according to the application scenario of the first neural network model. The preset training data can be training data used in the quantization sensing training, and can be more training data than the training data of the quantization sensing training, specifically set according to actual needs, and not limited in the embodiments of the present disclosure. The pre-constructed neural network can be set according to actual needs, for example, a Transformer network or other feasible network used for target detection. The training process of the first neural network model is as follows: The preset training data is used as the input of the neural network, the output of the neural network is obtained, a network loss is determined according to the tag data corresponding to the preset training data and the output of the neural network, and the network parameters are adjusted according to the network loss, and this process is performed until the network loss meets the preset condition, and when the network loss meets the preset condition, the training is terminated, and the first neural network model is obtained. The specific training process can be set according to actual needs, and not limited in the embodiments of the present disclosure.

[0062] In one alternative example, the preset operations include exponential and / or logarithmic operations.

[0063] In one selectable example, the preset operations may also include other complex operations besides exponential operations and logarithmic operations, and may be specifically set according to actual needs.

[0064] Each embodiment and each optional example of the present disclosure may be implemented alone or, in a non-conflicting situation, may be implemented in combination in any combination manner.

[0065] The present disclosure performs quantization-sensitive training by converting a preset operation in a neural network model into a first operation that simulates a table lookup operation, thereby simulating the operation logic of a simplified target operation in the training process, thereby ensuring that no new errors are generated after the first operation is converted into a quantized target operation, and therefore the model accuracy is not reduced, and the efficient calculation of a complex function can be realized by a simple table lookup operation under the condition of ensuring the model accuracy.When the target neural network model is arranged in the neural network accelerator for processing, the quantization-sensitive training is performed according to the bit width supported by the neural network accelerator, so that no new errors are generated in the arrangement stage, and further the model accuracy in hardware acceleration can be guaranteed, and the consistency between the accuracy of neural network training and arrangement can be realized.

[0066] 5 is a flowchart of a method for accelerating neural network model inference according to a further exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices such as servers and terminals. As shown in FIG. 5, the method of this embodiment includes steps 501 and 502.

[0067] In step 501, data to be processed, which may be image data, text data or audio data, is obtained.

[0068] Here, the specific type of the processing target data can be determined according to actual needs. For example, when the first neural network model is a road element detection model of an automatic driving scene, the processing target data is collected road environment image data.

[0069] In step 502, output data corresponding to the data to be processed is determined based on an accelerated target neural network model corresponding to the previously obtained first neural network model.

[0070] Here, the target neural network model is obtained according to any of the above embodiments or the method for accelerating neural network model inference of the selectable example, and the specific description can be referred to the above content, and the specific description of the inference process of the target neural network model is omitted.

[0071] The present disclosure can effectively improve data processing efficiency when data processing is performed by an accelerated target neural network model, and highly efficient processing of complex preset calculations can be realized based on simplified target calculations, and model accuracy can be guaranteed.

[0072] 6 is a flowchart of a target calculation according to an exemplary embodiment of the present disclosure. The target calculation is a table lookup operation, and the calculation process of the target calculation includes steps 601 to 603.

[0073] In step 601, the target subrange to which the input data of the lookup target table belongs is determined.

[0074] Here, the target subrange may be a subrange in the above-mentioned first input range, or may be a subrange in a subrange obtained by dividing a second input range determined based on the range of input data of the first operation of the quantization sensing training process, and can be specifically set according to actual needs.

[0075] In actual applications, since the target neural network model is a quantized model, each subrange can be a subrange of an input data type suitable for the target operation of the target neural network model. For example, if the target neural network model is int8, the endpoints of each subrange can be of int8 type, and can be specifically set according to actual needs.

[0076] In step 602, based on the target subrange, a table is looked up to obtain calculation results corresponding to both end points of the target subrange, respectively.

[0077] The principle of table lookup in this step can be understood by referring to the above content, and therefore the explanation thereof will be omitted.

[0078] In step 603, according to the input data of the lookup target table, the target sub-range, and the calculation results corresponding to both end points of the target sub-range, a linear interpolation method is used to obtain a target calculation result corresponding to the input data of the lookup target table.

[0079] The specific operation of this step can be seen from the above, so the explanation will be omitted.

[0080] The present disclosure can effectively improve data processing efficiency when the table lookup can realize highly efficient processing for complex preset calculations and ensure model accuracy.

[0081] The method for accelerating neural network model inference according to the embodiments of the present disclosure may be executed by any suitable device having data processing capability, including but not limited to a terminal device and a server. Alternatively, any method for accelerating neural network model inference according to the embodiments of the present disclosure may be executed by a processor, for example, the processor may execute any method for accelerating neural network model inference according to the embodiments of the present disclosure by calling a corresponding command stored in a memory. Hereinafter, the description thereof will be omitted. Exemplary Apparatus

[0082] 7 is a structural schematic diagram of an acceleration device for neural network model inference according to an exemplary embodiment of the present disclosure, which may be used to implement an embodiment of the method according to the present disclosure, and the device shown in FIG. 7 includes a first acquisition module 701, a first determination module 702, a first processing module 703, a second processing module 704 and a third processing module 705.

[0083] The first acquisition module 701 is for acquiring image training data, text training data or voice training data, the first determination module 702 is for determining a first neural network model to be accelerated, the first processing module 703 is for converting a preset operation of a preset network layer in the first neural network model determined by the first determination module 702 into a first operation for simulating an operation logic of a target operation to obtain a second neural network model, the second processing module 704 is for performing quantization sensing training on the second neural network model acquired by the first processing module 703 according to a preset bit width, which is a bit width that supports a neural network accelerator to perform data processing, based on the image training data, the text training data or the voice training data, to obtain a quantized third neural network model, and the third processing module 705 is for converting the first operation of the third neural network model acquired by the second processing module 704 into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0084] In one optional example, Figure 8 is a structural schematic diagram of the second processing module 704 according to one exemplary embodiment of the present disclosure. In this example, the target operation is a table lookup operation, and the second processing module 704 includes a first determining unit 7041, a second determining unit 7042 and a first processing unit 7043.

[0085] The first determination unit 7041 is for determining a first sub-range to which the first input data of the first operation belongs, the second determination unit 7042 is for determining the operation results corresponding to both end points of the first sub-range respectively based on a preset manner, and the first processing unit 7043 is for obtaining a first operation result corresponding to the first input data using a linear interpolation manner based on the first input data, the first sub-range and the operation results corresponding to both end points of the first sub-range respectively.

[0086] FIG. 9 is a structural schematic diagram of an accelerator for neural network model inference according to another exemplary embodiment of the present disclosure.

[0087] In one optional example, the target operation is a table lookup operation, and the apparatus of the present disclosure further includes a second determination module 706, a third determination module 707, a fourth determination module 708 and a fifth processing module 709.

[0088] The second determination module 706 is for determining a first input range based on input data of the preset operation in the training process of the first neural network model; the third determination module 707 is for segmenting the first input range based on a preset segment number to obtain sub-ranges corresponding to the preset segment number; the fourth determination module 708 is for determining, for each of the sub-ranges, operation results corresponding to both end points of the sub-range based on the preset operation; and the fifth processing module 709 is for storing each of the sub-ranges and the operation results corresponding to each end point of each of the sub-ranges, for use in the table lookup operation.

[0089] In one optional example, the first determination module 702 includes a second processing unit 7021 for training a pre-constructed neural network based on preset training data to obtain the first neural network model.

[0090] In one alternative example, the preset operations include exponential and / or logarithmic operations.

[0091] 10 is a structural schematic diagram of an acceleration device for neural network model inference according to a further exemplary embodiment of the present disclosure, which is for implementing the corresponding method embodiment of the present disclosure, and the device shown in FIG.

[0092] The second acquisition module 801 is for acquiring processing target data, which is image data, text data or audio data, and the fourth processing module 802 is for determining output data corresponding to the processing target data based on an accelerated target neural network model corresponding to a previously acquired first neural network model, where the target neural network model is acquired based on the acceleration method for neural network model inference according to any of the above embodiments or selectable examples.

[0093] FIG. 11 is a structural schematic diagram of the fourth processing module 802 according to an exemplary embodiment of the present disclosure.

[0094] In one alternative example, the target operation is a table lookup operation, and the fourth processing module 802 includes a third determination unit 8021, a third processing unit 8022 and a fourth processing unit 8023.

[0095] The third determination unit 8021 is for determining a target sub-range to which the input data of the lookup target table belongs, the third processing unit 8022 is for looking up a table based on the target sub-range to obtain calculation results corresponding to both end points of the target sub-range, and the fourth processing unit 8023 is for obtaining a target calculation result corresponding to the input data of the lookup target table using a linear interpolation method based on the input data of the lookup target table, the target sub-range and the calculation results corresponding to both end points of the target sub-range. Exemplary Electronic Devices

[0096] An electronic device according to an embodiment of the present disclosure includes a memory for storing a computer program; and a processor for executing a computer program stored in the memory, the processor implementing, when the computer program is executed, the method for accelerating neural network model inference described in any of the above embodiments of the present disclosure.

[0097] 12 is a structural schematic diagram of an application embodiment of the electronic device of the present disclosure. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0098] The processor 11 may be a central processing unit (CPU) or other type of processing unit having data processing and / or command execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0099] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored in the computer-readable storage media, and the processor 11 may execute the program instructions to realize the methods and / or other expected functions in each embodiment of the present disclosure described above.

[0100] In one example, electronic device 10 may further include input device(s) 13 and output device(s) 14 connected to each other via a bus system and / or other form of connection mechanism (not shown).

[0101] For example, the input device 13, eg a microphone or a microphone array as mentioned above, can be used to capture an input signal of a sound source.

[0102] The input device 13 may include, for example, a keyboard and a mouse.

[0103] The output device 14 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0104] Of course, for convenience of explanation, Fig. 12 shows only some of the components related to the present disclosure in the electronic device 10, and omits components such as a bus and an input / output interface. In addition, the electronic device 10 may further include any other appropriate components according to a specific application situation. Exemplary Computer Program Products and Computer-Readable Storage Media

[0105] In addition to the methods and apparatus described above, embodiments of the present disclosure may be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform method steps according to various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0106] The computer program product may be written in any combination of one or more programming languages ​​to create program code for carrying out operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and general procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed as follows: entirely on the user computing device, partially on the user computing device, as a separate software package, partly on the user computing device and partly on a remote computing device, or entirely on a remote computing device or server.

[0107] An embodiment of the present disclosure may also be a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in methods according to various embodiments of the present disclosure described herein above in the "Exemplary Methods" section.

[0108] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0109] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, the advantages, merits, effects, etc. mentioned in the present disclosure are merely illustrative and not limiting. Various embodiments of the present disclosure do not necessarily have those advantages, merits, effects, etc. In addition, the specific details of the above disclosure are merely for illustrative and easy-to-understand functions and are not limiting, and the above details do not necessarily limit the present disclosure to be realized by the above specific details.

[0110] Various embodiments in this specification are described in a sequential manner, and each embodiment is described mainly focusing on the differences from other embodiments, and the same or similar parts between the embodiments may be mutually referred to. The system embodiments are basically corresponding to the method embodiments, and therefore the description is relatively brief, and the relevant parts may be referred to in part of the description of the method embodiments.

[0111] The methods and apparatus of the present disclosure can be implemented in a number of ways. For example, the methods and apparatus of the present disclosure can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps in the methods is merely illustrative, and the steps of the methods of the present disclosure are not limited to the order specifically described above unless otherwise stated. In some embodiments, the present disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods of the present disclosure. Thus, the present disclosure also includes a recording medium storing a program for performing the methods of the present disclosure.

[0112] The above description has been presented for purposes of illustration and description. Moreover, the description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although a number of exemplary aspects and embodiments have been described above, those skilled in the art may recognize certain variations, modifications, variations, additions, and subcombinations thereof.

Claims

1. A method for accelerating neural network model inference, each step of which is performed by an accelerator for neural network model inference, comprising: obtaining image training data, text training data or audio training data; determining a first neural network model to be accelerated; Transforming a preset operation of a preset network layer in the first neural network model into a first operation for simulating an operation logic of a target operation to obtain a second neural network model; performing quantization-sensitive training on the second neural network model according to a preset bit width, which is a bit width that supports a neural network accelerator to process data, based on the image training data, the text training data, or the audio training data, to obtain a quantized third neural network model; converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model; A method for accelerating neural network model inference, comprising:

2. the target operation is a table lookup operation; The first operation includes: determining a first subrange to which first input data of the first operation belongs; determining a calculation result corresponding to each of the two end points of the first sub-range based on a preset scheme; and obtaining a first calculation result corresponding to the first input data by using a linear interpolation method based on the first input data, the first sub-range, and the calculation results corresponding to both end points of the first sub-range.

2. The method of claim 1, further comprising:

3. the target operation is a table lookup operation; The method includes, after the step of determining a first neural network model to be accelerated, determining a first input range based on input data of the preset operation in a training process of the first neural network model; Segmenting the first input range based on a preset number of segments to obtain a sub-range corresponding to the preset number of segments; determining, for each of the subranges, a calculation result corresponding to each of the two end points of the subrange based on the preset calculation; storing the results of the calculation corresponding to each of the sub-ranges and each of the endpoints of each of the sub-ranges for use in the table look-up operation.

2. The method of claim 1, further comprising:

4. The step of determining a first neural network model to be accelerated includes: training a pre-constructed neural network based on preset training data to obtain the first neural network model; 2. The method of claim 1, further comprising:

5. The preset calculation includes an exponential calculation and / or a logarithmic calculation.

2. The method of claim 1, further comprising:

6. obtaining data to be processed, the data being image data, text data or audio data; determining output data corresponding to the target data based on an accelerated target neural network model corresponding to a previously obtained first neural network model; Including, The target neural network model is obtained based on the method for accelerating neural network model inference according to any one of claims 1 to 5. A method for accelerating neural network model inference.

7. the target operation is a table lookup operation; The calculation process of the target calculation is as follows: determining a target subrange to which the input data of the lookup target table belongs; According to the target subrange, look up a table to obtain calculation results corresponding to both ends of the target subrange respectively; and obtaining a target calculation result corresponding to the input data of the lookup target table by using a linear interpolation method based on the input data of the lookup target table, the target sub-range, and the calculation results corresponding to both end points of the target sub-range.

7. The method of claim 6, wherein the method comprises:

8. a first acquisition module for acquiring image training data, text training data, or audio training data; a first determination module for determining a first neural network model to be accelerated; A first processing module converts a preset operation of a preset network layer in the first neural network model into a first operation for simulating an operation logic of a target operation to obtain a second neural network model; a second processing module for performing quantization-sensitive training on the second neural network model according to the image training data, the text training data, or the audio training data, according to a preset bit width, which is a bit width that supports a neural network accelerator to perform data processing, to obtain a quantized third neural network model; and a third processing module for converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model. A neural network model inference acceleration device comprising:

9. a second acquisition module for acquiring data to be processed, the data being image data, text data, or audio data; a fourth processing module for determining output data corresponding to the processing target data based on an accelerated target neural network model corresponding to the previously obtained first neural network model; Including, The target neural network model is obtained by a method for accelerating neural network model inference according to any one of claims 1 to 5. A neural network model inference acceleration device comprising:

10. A computer-readable storage medium for storing a computer program, The computer program executes the method for accelerating neural network model inference according to any one of claims 1 to 5. A computer-readable storage medium comprising:

11. A processor; a memory for storing instructions executable by said processor; Including, The processor reads the executable command from the memory and executes the executable command, thereby implementing the method for accelerating neural network model inference according to any one of claims 1 to 5.

1. An electronic device comprising:

Images