Method and apparatus for accelerating neural network model inference, electronic device and medium

Abstract

Embodiments of the present disclosure disclose a method, device, electronic equipment and storage medium for accelerating inference of a neural network model, wherein the method comprises: obtaining image training data, text training data or speech training data; determining a first neural network model to be accelerated; converting a preset operation of a preset network layer in the first neural network model into a first operation to obtain a second neural network model, the first operation being used to simulate operation logic of a target operation; performing quantization-aware training on the second neural network model according to a preset bit width based on the image training data, the text training data or the speech training data to obtain a third neural network model after quantization; and converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model. Embodiments of the present disclosure simulate errors caused by simplified operations in the training process to ensure high model accuracy under the condition of reducing model calculation amount.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a method and apparatus for accelerating neural network model inference, an electronic device, and a medium. BACKGROUND

[0002] In the field of deep learning, functions such as softmax, sigmoid, swish, etc. are very important parts of neural network models. However, the operation of these functions is complex and has high computational cost, which leads to low inference speed of neural network models. To solve this problem, related technologies usually simplify these complex operations, such as implementing the calculation of softmax through the combination of multiple lookup tables and the four arithmetic operations of the lookup table results, to reduce the computational cost. However, the existing simplification method is prone to low model accuracy. SUMMARY

[0003] To solve the technical problem of low model accuracy caused by the above-mentioned simplification method, the present disclosure is proposed. Embodiments of the present disclosure provide a method and apparatus for accelerating neural network model inference, an electronic device, and a medium.

[0004] According to one aspect of an embodiment of the present disclosure, a method for accelerating neural network model inference is provided, including: obtaining image training data, text training data, or speech training data; determining a first neural network model to be accelerated; converting a preset operation of a preset network layer in the first neural network model into a first operation to obtain a second neural network model, the first operation being used to simulate the operation logic of a target operation; performing quantization-aware training on the second neural network model according to a preset bit width based on the image training data, the text training data, or the speech training data to obtain a third neural network model after quantization, the preset bit width being a bit width supported by a neural network accelerator for data processing; and converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0005] According to another aspect of an embodiment of the present disclosure, a method for accelerating neural network model inference is provided, including: obtaining to-be-processed data, the to-be-processed data being image data, text data, or speech data;

[0006] Based on a pre-obtained accelerated target neural network model corresponding to a first neural network model, determining output data corresponding to the to-be-processed data; the target neural network model is obtained based on the method for accelerating neural network model inference provided in any of the above embodiments.

[0007] According to still another aspect of the embodiments of the present disclosure, an acceleration device for neural network model inference is provided, comprising: a first acquisition module configured to acquire image training data, text training data or voice training data; a first determination module configured to determine a first neural network model to be accelerated; a first processing module configured to convert a preset operation of a preset network layer in the first neural network model into a first operation, to obtain a second neural network model, the first operation being configured to simulate operation logic of a target operation; a second processing module configured to perform quantization-aware training on the second neural network model according to a preset bit width based on the image training data, the text training data or the voice training data, to obtain a third neural network model after quantization, the preset bit width being a bit width supported by a neural grid accelerator for data processing; and a third processing module configured to convert the first operation of the third neural network model into the target operation, to obtain an accelerated target neural network model corresponding to the first neural network model.

[0008] According to still another aspect of the embodiments of the present disclosure, an acceleration device for neural network model inference is provided, comprising: a second acquisition module configured to acquire to-be-processed data, the to-be-processed data being image data, text data or voice data; and a fourth processing module configured to determine output data corresponding to the to-be-processed data based on a pre-obtained accelerated target neural network model corresponding to a first neural network model, the target neural network model being obtained based on the acceleration method for neural network model inference according to any one of the above embodiments.

[0009] According to still another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute the acceleration method for neural network model inference according to any one of the above embodiments of the present disclosure.

[0010] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory configured to store executable instructions for the processor; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the acceleration method for neural network model inference according to any one of the above embodiments of the present disclosure.

[0011] Based on the acceleration method, device, electronic device and medium for neural network model inference provided in the above embodiments of the present disclosure, the complex preset operation in the neural network is converted into a simple target operation, and the operation logic of the target operation is simulated in the quantization-aware training process, so that the error caused by the simplified target operation can be simulated in the training process. The model precision is ensured not to be reduced under the condition of simplified operation, so that the problem of low model precision caused by the simplification method in the prior art is effectively solved.

[0012] The technical solutions of the present disclosure are described in further detail below with reference to the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like elements throughout. The accompanying drawings are intended to provide a further understanding of embodiments of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the present disclosure, and do not constitute a limitation of the present disclosure. In the drawings, like reference numerals refer to like elements throughout.

[0014] Figure 1 is an exemplary application scenario of the method for accelerating the inference of the neural network model provided by the present disclosure;

[0015] Figure 2 is a flowchart of the method for accelerating the inference of the neural network model provided by an exemplary embodiment of the present disclosure;

[0016] Figure 3 is a flowchart of the first operation provided by an exemplary embodiment of the present disclosure;

[0017] Figure 4 is a flowchart of the method for accelerating the inference of the neural network model provided by another exemplary embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of the method for accelerating the inference of the neural network model provided by yet another exemplary embodiment of the present disclosure;

[0019] Figure 6 is a flowchart of the target operation provided by an exemplary embodiment of the present disclosure;

[0020] Figure 7 is a structural diagram of the device for accelerating the inference of the neural network model provided by an exemplary embodiment of the present disclosure;

[0021] Figure 8 is a structural diagram of the second processing module 704 provided by an exemplary embodiment of the present disclosure;

[0022] Figure 9 is a structural diagram of the device for accelerating the inference of the neural network model provided by another exemplary embodiment of the present disclosure;

[0023] Figure 10 is a structural diagram of the device for accelerating the inference of the neural network model provided by yet another exemplary embodiment of the present disclosure;

[0024] Figure 11 is a structural diagram of the fourth processing module 802 provided by an exemplary embodiment of the present disclosure;

[0025] Figure 12 FIG. 1 is a structural schematic diagram of an electronic device according to an application embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0027] It should be noted that: unless otherwise specified, the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0028] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor do they represent the inevitable logical sequence between them.

[0029] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two or more.

[0030] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, unless specifically limited or given the opposite implication by the context, it can be understood as one or more in general.

[0031] In addition, the term "and / or" in the present disclosure is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0032] It should also be understood that the description of various embodiments of the present disclosure emphasizes the differences between various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0033] At the same time, it should be understood that in order to facilitate description, the size of each part shown in the drawings is not drawn in accordance with the actual proportional relationship.

[0034] The following description of at least one example embodiment is merely illustrative in nature and does not in any way limit the present disclosure and its applications or uses.

[0035] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can have not been discussed in any detail because such techniques, methods, and apparatus are deemed to be part of the state of the art

[0036] It should be borne in mind, that, like in any business endeavor, all such similar terms are not referred to a specific magnitudes per se, the actual values will depend on the specific application for which the terms represent.

[0037] Embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and the like electronic devices, which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations that can be suitable for use with terminal devices, computer systems, servers, and the like electronic devices include, but are not limited to: personal computers, servers, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers, mainframe computers, and distributed cloud computing technology environments that include any of the above systems, and the like.

[0038] Terminal devices, computer systems, servers, and the like electronic devices can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, that perform particular tasks or implement particular abstract data types. Computer systems / servers can be practiced in distributed cloud-computing environments with remote processing devices that are linked through a communications network. In a distributed cloud-computing environment, program modules can be located in local and / or remote computer system storage media including memory storage devices.

[0039] SUMMARY

[0040] In the process of implementing the present disclosure, the inventors found that in the field of deep learning, functions such as softmax, sigmoid, swish, and the like are very important parts of neural network models, but the operation of these functions is complex and the calculation cost is high, which leads to a low inference speed of the neural network model. In order to solve this problem, the related art usually simplifies these complex operations, such as implementing the calculation of softmax by combining multiple lookup tables and four arithmetic operations of the lookup table results, to reduce the calculation cost, but the existing simplification method is prone to cause low model accuracy.

[0041] EXEMPLARY SUMMARY

[0042] Figure 1is an exemplary application scenario of the method for accelerating inference of a neural network model provided by the present disclosure.

[0043] In the target detection application scenario of the neural network of the Transformer series, a Transformer-based target detection model is usually used to perform target detection on a to-be-detected image to obtain a detection result. The Multi-headed Self-attention (MSA) network block is involved in the encoder and the decoder of the Transformer network architecture of the target detection model, and the self-attention operation of the Multi-headed Self-attention network block involves a softmax function. The operation of the softmax function includes an exponential operation. By using the device for accelerating inference of a neural network model and the method for accelerating inference of a neural network model provided by the present disclosure, the target detection model can be taken as a first neural network model to be accelerated, the exponential operation of the softmax layer in the first neural network model is converted into a first operation, a second neural network model is obtained, the first operation is used to simulate the operation logic of a target operation, the target operation is a simplified operation, such as a lookup table operation, the second neural network model is trained according to a preset bit width based on image training data, a third neural network model after quantization is obtained, the preset bit width can be a bit width supported by a neural network accelerator for data processing, after the training of the quantization perception is completed, the first operation in the third neural network model is converted into the target operation, and an accelerated target neural network model corresponding to the first neural network model is obtained. The target neural network model implements the exponential operation of the softmax layer based on the simplified target operation, so as to reduce the calculation amount, and the operation logic of the simplified target operation is simulated in the training process of the quantization perception, so as to simulate the error caused by the simplified operation. Therefore, the obtained target neural network model still has high precision on the basis of low calculation amount, and the problem of low precision of the existing simplified operation model is solved.

[0044] EXEMPLARY METHOD

[0045] Figure 2 is a flowchart of the method for accelerating inference of a neural network model provided by an exemplary embodiment of the present disclosure. The present embodiment can be applied on electronic devices such as servers, terminals, and the like, as shown in Figure 2 , including the following steps:

[0046] Step 201: Obtain image training data, text training data, or voice training data.

[0047] The image training data can be image data from any scene, such as road environment image data collected in autonomous driving scenarios, medical image data, etc., and can be specifically set according to actual needs. The text training data can be text data from any scene, such as text data from any scenario requiring text semantic recognition; the voice training data can be voice data from any scene, such as voice data collected for voice control in smart homes, voice data collected for voice control in vehicles, and can be specifically set according to actual needs. The specific type of data acquired depends on the neural network model to be accelerated. For example, if the neural network model to be accelerated is a road element target detection model for autonomous driving scenarios, then pre-obtained road environment image data will be used as the image training data.

[0048] Step 202: Determine the first neural network model to be accelerated.

[0049] The first neural network model can be any model for any application scenario, and it is a pre-trained, fully accurate model. For example, when it is necessary to accelerate the target detection model of road elements in an autonomous driving scenario, this target detection model is used as the first neural network model to be accelerated. The first neural network model needs to be adapted to the training data obtained in step 201. In practical applications, the training data and the first neural network model can be pre-stored in a certain storage space, such as corresponding storage, or associated through address, or the user can specify the address for obtaining the training data and the first neural network model to be accelerated, etc. The specific settings can be configured according to actual needs, and this disclosure does not limit them.

[0050] Step 203: Convert the preset operation of the preset network layer in the first neural network model into the first operation to obtain the second neural network model. The first operation is used to simulate the operation logic of the target operation.

[0051] The preset network layer and the preset operation of the preset network layer can be set according to actual needs. For example, when the first neural network model is a target detection model based on a Transformer, the preset network layer can be a softmax layer, and the preset operation can be an exponential operation in the softmax, or the preset network layer can be a self-attention network layer, and the preset operation can be an exponential operation in the self-attention network layer. For another example, for the first neural network model with a sigmoid activation function, the preset network layer can be a sigmoid activation layer, and the preset operation can be an exponential operation in the sigmoid activation layer, and the specific implementation is not limited. The target operation is a simplified operation corresponding to the preset operation, such as a lookup table operation or a linear fitting operation. In the lookup table operation, the target sub-range to which the input data to be looked up belongs is determined first, the operation results corresponding to the two endpoints of the target sub-range are obtained by looking up the table based on the target sub-range, and the target operation result corresponding to the input data to be looked up is obtained by using linear interpolation based on the operation results corresponding to the two endpoints. In the first operation, the first sub-range to which the first input data belongs is determined first, and then the operation results corresponding to the two endpoints of the first sub-range are determined, and then the first operation result corresponding to the first input data is obtained by using linear interpolation based on the operation results corresponding to the two endpoints, so as to simulate the operation logic of the target operation. The operation results corresponding to the two endpoints of the first sub-range can be obtained by real-time calculation or by looking up a table, and the specific implementation can be set according to actual needs.

[0052] In actual applications, the preset network layer can include one or more types of network layers, and the preset algorithm can include one or more algorithms. For example, when a model includes both a softmax function and a sigmoid activation function, the preset network layer can include a normalization network layer in which the softmax function is located and an activation layer in which the sigmoid activation function is located, and the preset operation can include an exponential operation in the softmax and an exponential operation in the sigmoid. The specific implementation can be set according to actual needs, and the present disclosure is not limited in this regard. The method of the present disclosure can be used to accelerate the model for any preset network layer with a complex preset operation.

[0053] Steps 201 and steps 202-203 are not in a specific order.

[0054] In step 204, the second neural network model is trained according to the preset bit width based on image training data, text training data, or voice training data, to obtain a quantized third neural network model. The preset bit width is a bit width supported by a neural network accelerator for data processing.

[0055] The preset bit width can be set according to actual needs, such as 8 bits, 16 bits, 4 bits, 2 bits, 1 bit, and the like, and specifically, for example, 8-bit integer (int8). The quantization-aware training is to insert a pseudo-quantization node in the model to simulate the rounding and clamping operations of the quantized model in the inference process, thereby improving the adaptability of the model to the quantized data in the training process and obtaining higher quantized model precision. The quantization objects can include weights, activation outputs, gradients, and the like, and can be set according to actual needs. Quantizing the weights and activation outputs can effectively reduce the memory occupation space of the model, and quantizing the gradients can reduce the overhead of the reverse calculation. Whether the training data is image training data, text training data, or speech training data can be determined according to the specific application scenario of the second neural network model or the foregoing first neural network model. For a certain model, the type of training data required for quantization-aware training is determined, which can be one of the foregoing image training data, text training data, and speech training data. The bit width supported by the neural network accelerator for data processing refers to the bit width of the network parameters (such as weights), input data, output data, intermediate data (such as activation outputs), and the like supported by the neural network accelerator when implementing inference calculation of the neural network based on the neural network accelerator. For example, if the neural network accelerator supports int8 bit width, the input of each data processing module in the neural network accelerator needs to be int8 data, and the weights used for calculation are also int8 data, and the like. The bit width can be set according to actual needs. The bit width supported by the full-precision neural network model is usually 64-bit floating point (float64) or 32-bit floating point (float32). In order to accelerate the inference speed of the neural network model based on the neural network accelerator, it is necessary to quantize various data of the full-precision neural network model to the bit width supported by the neural network accelerator, so that the quantized neural network model can be deployed on the neural network accelerator for inference calculation, so as to accelerate the inference of the neural network model, and the low-bit quantized neural network model occupies less storage space, thereby effectively reducing the memory occupation in the inference process.

[0056] In actual applications, the quantization of the weights, activation outputs, and gradients of the model can be set to the same or different preset bit widths, which can be set according to actual needs, and the present disclosure is not limited thereto.

[0057] Optionally, in the quantization-aware training process, the model forward and backward propagation calculation and pseudo-quantization node calculation can be implemented in floating-point calculation, and the preset bit width fixed-point related rules (such as rounding and clamping) can be followed, and then quantized into a real fixed-point model after training is completed, as the third neural network model.

[0058] For example, in the quantization-aware training process, the floating-point number 0.1 can represent the fixed-point number 1, the floating-point number 0.2 can represent the fixed-point number 2, and so on, so that the floating-point number and the fixed-point number have a fixed multiple relationship, and the rules of the fixed-point number are followed in the training process, such as the fixed-point number encountering 3.5 will be rounded up or down to 4 or 3, and the floating-point number 0.35 will be converted to 0.4 or 0.3, so that the floating-point number satisfying the fixed-point number rule can be used to determine the network loss to simulate the loss caused by simplified operation, and after the training is completed, the floating-point model is converted into a fixed-point model through model value conversion, and the model precision is not reduced.

[0059] Step 205, converting the first operation of the third neural network model into the target operation to obtain an accelerated target neural network model corresponding to the first neural network model.

[0060] Since the first operation is an operation logic simulating the target operation, after the quantization-aware training is completed, the first operation of the third neural network model can be directly converted into the target operation, and the converted target neural network model has the same precision as the third neural network model, so that the model precision is not reduced in the case of reducing the model calculation amount.

[0061] For example, the softmax function is as follows:

[0062]

[0063] The exponential operation F(x) = e x After converting into a lookup table operation, the calculation is as follows:

[0064] F(x) = LUT exp (x)

[0065] Wherein, LUT (Look Up Table) represents a lookup table operation, LUT exp(x) represents a look-up table operation on the input x to obtain its corresponding exponential operation result, specifically, first determine the interval to which x belongs in the table, obtain the exponential operation results corresponding to the two endpoints of the interval by look-up table, and based on the exponential operation results corresponding to the two endpoints, perform linear interpolation to obtain the exponential operation result corresponding to x. Since the computer has a very large amount of calculation when calculating the value of the exponential function (such as calculating by Taylor series expansion or other possible ways), converting into look-up table operation, only simple comparison, look-up and linear interpolation are needed to quickly obtain the exponential operation result, thereby effectively reducing the model calculation amount and improving the reasoning speed of the model.

[0066] The acceleration method for neural network model inference provided by the embodiment converts the complex preset operation in the neural network into a simple target operation, and simulates the operation logic of the target operation in the quantization perception training process, so as to simulate the error caused by the simplified target operation in the training process, so that the target neural network model ensures that the accuracy of the model will not be reduced under the condition of simplified operation, thereby effectively solving the problem of low model accuracy caused by the simplification method of the prior art.

[0067] Figure 3 is a flowchart of the first operation provided by an example embodiment of the disclosure.

[0068] In an optional example, the target operation is a look-up table operation, and the first operation specifically can include the following steps:

[0069] Step 301, determine the first sub-range to which the first input data of the first operation belongs.

[0070] Among them, the first input range can be determined based on the input data of the preset operation in the training process of the first neural network model, the first input range is divided into multiple sub-ranges, the operation results corresponding to the endpoints of each sub-range are calculated and obtained, and each sub-range and the operation results corresponding to the endpoints of each sub-range are stored, or each sub-range is stored. Specifically, it can be set according to actual needs. Thus, when the first operation is needed, the first input data can be compared with the endpoints of each sub-range to determine the first sub-range to which the first input data belongs.

[0071] Step 302, determine the operation results corresponding to the two endpoints of the first sub-range based on a preset manner.

[0072] The preset manner can include calculation and table lookup. Since the second neural network model is used for quantization-aware training, the operation results corresponding to the two endpoints of the first sub-range can be calculated in real time during actual operation in the training process, or the operation results corresponding to the endpoints of each sub-range can be calculated in advance and stored, and then the operation results corresponding to the two endpoints of the first sub-range can be directly obtained from the storage area during operation. Regardless of which manner is used, the accuracy of the operation logic of the first operation simulation target operation is not affected. In order to further reduce the calculation amount of quantization-aware training, the table lookup manner can be used to determine the operation results corresponding to the two endpoints of the first sub-range.

[0073] In step 303, based on the first input data, the first sub-range, and the operation results corresponding to the two endpoints of the first sub-range, a linear interpolation method is used to obtain the first operation result corresponding to the first input data.

[0074] The linear interpolation method refers to an interpolation method in which the interpolation function is a first-degree polynomial, and the specific interpolation principle is not repeated. In order to simulate the operation logic of the lookup table operation, after the operation results corresponding to the two endpoints of the first sub-range are determined, linear interpolation is performed based on the operation results corresponding to the two endpoints of the first sub-range at the first input data to obtain the first operation result corresponding to the first input data.

[0075] The second neural network model is used to simulate the lookup table operation in the first operation, so that during the quantization-aware training process, the error caused by the lookup table operation can be simulated, and after the training is completed, the model can also achieve the same or similar high accuracy as the first neural network model under quantized data, and the model accuracy will not be significantly reduced due to the lookup table operation.

[0076] Figure 4 FIG. 4 is a flowchart of an acceleration method for neural network model inference according to another example embodiment of the present disclosure.

[0077] In one optional example, the target operation is a lookup table operation; after the first neural network model to be accelerated is determined in step 202, the method of the present disclosure further includes:

[0078] In step 401, based on the input data of the preset operation in the training process of the first neural network model, the first input range is determined.

[0079] During the training process to obtain the first neural network model, the input data of the preset operation can be recorded, and the first input range can be determined by comprehensively recording the range of the input data of the preset operation in the training process.

[0080] In actual application, the first input range is a lookup table operation applied to the quantization-aware training process, and the input data of the preset operation of the first neural network model is a floating-point number. The first input range suitable for quantization-aware training can be determined according to the floating-point input data. For example, in the second neural network model, the input of the first operation is a floating-point number 0.2 simulating a preset bit width of fixed-point number 2, and the range of the input data of the preset operation is 0.001-0.999. Therefore, the first input range is determined as 0.0-1.0, which can be set according to actual requirements.

[0081] In step 402, the first input range is segmented based on the preset number of segments to obtain a preset number of sub-ranges.

[0082] The preset number of segments can be set according to actual requirements, such as 100 segments, 50 segments, 20 segments, etc. The present embodiment does not limit.

[0083] For example, 0.0-1.0 is divided into 5 sub-ranges, which are 0.0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8, and 0.8-1.0. The division method of each sub-range can be set according to actual requirements, which can be equal interval division or unequal interval division. The present embodiment does not limit.

[0084] In step 403, for each sub-range, the operation results corresponding to the two endpoints of the sub-range are determined based on the preset operation.

[0085] The operation results of the two endpoints of each sub-range can be calculated based on the preset operation, such as exponential operation, to obtain the accurate operation results of the two endpoints of each sub-range.

[0086] In step 404, each sub-range and the operation results corresponding to each endpoint of each sub-range are stored for lookup table operation.

[0087] After storing the operation results corresponding to each endpoint of each sub-range in advance, in the quantization-aware training process, when the first operation is performed, the operation results corresponding to the two endpoints of the first sub-range can be determined based on the lookup table after determining the first sub-range to which the first input data of the first operation belongs. Then, the first operation result corresponding to the first input data is obtained based on the linear interpolation of the operation results of the two endpoints, so as to simulate the operation logic of the lookup table operation in the quantization-aware training process, and ensure that the model accuracy will not be reduced after being converted into the target operation.

[0088] The disclosure determines each sub-range for the table lookup operation and the operation result corresponding to each sub-range by segmenting the input data range of the first operation and storing the operation result, so that in the quantization-aware training process, for the first operation, the operation result of the two endpoints of the first sub-range to which the first input data belongs can be obtained through the table lookup operation, and then the first operation result corresponding to the first input data is obtained based on linear interpolation, thereby reducing the calculation amount of the quantization-aware training process and improving the training efficiency.

[0089] In an optional example, the determining of the first neural network model to be accelerated in step 202 comprises:

[0090] In step 2021, the pre-established neural network is trained based on the preset training data to obtain the first neural network model.

[0091] The preset training data can be image training data, text training data or voice training data, which is specifically determined according to the application scenario of the first neural network model. The preset training data can be the training data used for quantization-aware training as described above, or more training data than the training data used for quantization-aware training. The actual demand can be set, and the embodiments of the disclosure are not limited. The pre-established neural network can be set according to the actual demand, such as a Transformer network or other implementable network used for target detection. The training process of the first neural network model is as follows: the preset training data is used as the input of the neural network, the output of the neural network is obtained, the network loss is determined based on the label data corresponding to the preset training data and the output of the neural network, the network parameters are adjusted based on the network loss, and the training is ended until the network loss meets the preset condition, and the first neural network model is obtained. The specific training process can be set according to the actual demand, and the embodiments of the disclosure are not limited.

[0092] In an optional example, the preset operation includes exponential operation and / or logarithmic operation.

[0093] In an optional example, the preset operation can also include other complex operations in addition to exponential operation and logarithmic operation, which can be set according to the actual demand.

[0094] The embodiments of the disclosure and the optional examples can be implemented alone or in any combination without conflict.

[0095] The disclosure converts preset operations in a neural network model into first operations simulating lookup table operations, performs quantization-aware training, thereby simulating the operation logic of a simplified target operation in the training stage, so that after the first operation is converted into a quantized target operation, no new error is generated, thereby ensuring that the model accuracy will not be reduced, and realizing efficient calculation of a complex function through simple table lookup operation under the condition of ensuring model accuracy. When the target neural network model is deployed to a neural network accelerator for processing, since the quantization-aware training is performed according to the bit width supported by the neural network accelerator, no new error is generated in the deployment stage, further ensuring the model accuracy under hardware acceleration, and realizing consistent accuracy in neural network training and deployment.

[0096] Figure 5 is a flowchart of an acceleration method of a neural network model inference provided by another exemplary embodiment of the disclosure. The present embodiment can be applied on electronic devices such as servers, terminals and the like. As shown in Figure 5 The method of the present embodiment comprises:

[0097] Step 501, obtaining the to-be-processed data, the to-be-processed data being image data, text data or voice data.

[0098] The specific type of the to-be-processed data can be determined according to actual needs. For example, when the first neural network model is a road element detection model for an autonomous driving scenario, the to-be-processed data is collected road environment image data.

[0099] Step 502, determining the output data corresponding to the to-be-processed data based on the pre-obtained accelerated target neural network model corresponding to the first neural network model.

[0100] The target neural network model is obtained based on the acceleration method of the neural network model inference of any one of the above embodiments or optional examples, and the details are described above and will not be repeated here. The inference process of the specific target neural network model will not be repeated.

[0101] The disclosure accelerates the data processing through the accelerated target neural network model, realizes efficient processing of the complex preset operation based on the simplified target operation, and effectively improves the data processing efficiency under the condition of ensuring the model accuracy.

[0102] In one optional example, Figure 6 is a flowchart of a target operation provided by an exemplary embodiment of the disclosure. The target operation is a lookup table operation, and the operation process of the target operation comprises:

[0103] Step 601, determining the target sub-range to which the lookup table input data belongs.

[0104] The target sub-range can be a sub-range in the first input range, or a sub-range obtained by dividing the second input range determined according to the range of the input data of the first operation of the quantization perception training process. The sub-range can be set according to actual needs.

[0105] In actual application, since the target neural network model is a quantized model, each sub-range can be a sub-range suitable for the input data type of the target operation of the target neural network model. For example, when the target neural network model is int8, the endpoints of each sub-range can be of the int8 type, which can be set according to actual needs.

[0106] At step 602, the two endpoints of the target sub-range correspond to the operation results, respectively, are obtained by table lookup based on the target sub-range.

[0107] The table lookup principle of this step is described above and will not be repeated here.

[0108] At step 603, the target operation result corresponding to the input data to be looked up is obtained by linear interpolation based on the input data to be looked up, the target sub-range, and the operation results corresponding to the two endpoints of the target sub-range, respectively.

[0109] The specific operation of this step is described above and will not be repeated here.

[0110] The present disclosure realizes efficient processing of complex preset operations through table lookup operation, effectively improves the data processing efficiency while ensuring the model accuracy.

[0111] Any neural network model inference acceleration method provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to terminal devices and servers, etc. Alternatively, any neural network model inference acceleration method provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing any neural network model inference acceleration method mentioned by the embodiments of the present disclosure by calling corresponding instructions stored in a memory. The following will not be repeated.

[0112] EXEMPLARY DEVICE

[0113] Figure 7 FIG. 1 is a structural schematic diagram of an acceleration device for neural network model inference provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as the device shown in FIG. 1. Figure 7 The device shown in FIG. 1 includes a first acquisition module 701, a first determination module 702, a first processing module 703, a second processing module 704, and a third processing module 705.

[0114] The first obtaining module 701 is configured to obtain image training data, text training data, or voice training data; the first determining module 702 is configured to determine a first neural network model to be accelerated; the first processing module 703 is configured to convert a preset operation of a preset network layer in the first neural network model determined by the first determining module 702 into a first operation, to obtain a second neural network model, wherein the first operation is used to simulate the operation logic of a target operation; the second processing module 704 is configured to perform quantization perception training on the second neural network model obtained by the first processing module 703 according to a preset bit width based on the image training data, the text training data, or the voice training data, to obtain a third neural network model after quantization, wherein the preset bit width is a bit width supported by a neural network accelerator for data processing; and the third processing module 705 is configured to convert the first operation of the third neural network model obtained by the second processing module 704 into the target operation, to obtain an accelerated target neural network model corresponding to the first neural network model.

[0115] In an optional example, Figure 8 FIG. 7B is a structural schematic diagram of the second processing module 704 provided by an example embodiment of the present disclosure. In the example, the target operation is a lookup table operation, and the second processing module 704 includes a first determining unit 7041, a second determining unit 7042, and a first processing unit 7043.

[0116] The first determining unit 7041 is configured to determine a first sub-range to which first input data of the first operation belongs; the second determining unit 7042 is configured to determine, based on a preset manner, operation results respectively corresponding to two endpoints of the first sub-range; and the first processing unit 7043 is configured to obtain, based on the first input data, the first sub-range, and the operation results respectively corresponding to the two endpoints of the first sub-range, a first operation result corresponding to the first input data by using a linear interpolation manner.

[0117] Figure 9 FIG. 7C is a structural schematic diagram of an acceleration device for neural network model inference provided by another example embodiment of the present disclosure.

[0118] In an optional example, the target operation is a lookup table operation; and the device of the present disclosure further includes a second determining module 706, a third determining module 707, a fourth determining module 708, and a fifth processing module 709.

[0119] The second determining module 706 is configured to determine a first input range based on input data of the preset operation in a training process of the first neural network model.

[0120] In an optional example, the first determining module 702 includes a second processing unit 7021 configured to train a pre-established neural network based on preset training data to obtain the first neural network model.

[0121] In an optional example, the preset operation includes an exponential operation and / or a logarithmic operation.

[0122] Figure 10 FIG. 8 is a structural schematic diagram of an acceleration device for neural network model inference according to another example embodiment of the present disclosure. The device of this example embodiment can be used to implement the corresponding method embodiments of the present disclosure. For example, the device shown in FIG. 8 can be used to implement the method embodiments shown in FIG. 7. Figure 10 The device shown in FIG. 8 includes a second obtaining module 801 and a fourth processing module 802.

[0123] The second obtaining module 801 is configured to obtain to-be-processed data, which is image data, text data, or voice data. The fourth processing module 802 is configured to determine output data corresponding to the to-be-processed data based on a target neural network model obtained by accelerating a first neural network model obtained in advance. The target neural network model is obtained based on the neural network model inference acceleration method provided in any of the above example embodiments or optional examples.

[0124] Figure 11 FIG. 8 is a structural schematic diagram of an acceleration device for neural network model inference according to another example embodiment of the present disclosure. The device of this example embodiment can be used to implement the corresponding method embodiments of the present disclosure. For example, the device shown in FIG. 8 can be used to implement the method embodiments shown in FIG. 7.

[0125] In an optional example, the target operation is a lookup table operation, and the fourth processing module 802 includes a third determining unit 8021, a third processing unit 8022, and a fourth processing unit 8023.

[0126] The third determining unit 8021 is configured to determine a target sub-range to which the table lookup input data belongs; the third processing unit 8022 is configured to obtain, based on the target sub-range, operation results corresponding to two endpoints of the target sub-range respectively by table lookup; and the fourth processing unit 8023 is configured to obtain, based on the table lookup input data, the target sub-range, and the operation results corresponding to the two endpoints of the target sub-range respectively, a target operation result corresponding to the table lookup input data by using a linear interpolation method.

[0127] EXEMPLARY ELECTRONIC DEVICE

[0128] The embodiments of the present disclosure further provide an electronic device, comprising: a memory configured to store a computer program;

[0129] A processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for accelerating the inference of the neural network model in any of the embodiments of the present disclosure is implemented.

[0130] Figure 12 Fig. 1 is a structural schematic diagram of an electronic device according to an application embodiment of the present disclosure. In this embodiment, the electronic device 10 comprises one or more processors 11 and a memory 12.

[0131] The processor 11 can be a central processing unit (CPU) or other forms of processing units having data processing capability and / or instruction execution capability, and can control other components in the electronic device 10 to perform desired functions.

[0132] The memory 12 can comprise one or more computer program products, which can comprise various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can comprise, for example, random access memory (RAM), cache memory and / or the like. The non-volatile memory can comprise, for example, read-only memory (ROM), hard disk, flash memory and / or the like. One or more computer program instructions can be stored on the computer readable storage media, and the processor 11 can run the program instructions to implement the methods of the embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components and the like can also be stored in the computer readable storage media.

[0133] In one example, the electronic device 10 can further comprise an input device 13 and an output device 14, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0134] For example, the input device 13 can be the microphone or microphone array described above, which is configured to capture the input signal of the sound source.

[0135] In addition, the input device 13 can include, for example, a keyboard, a mouse, and the like.

[0136] The output device 14 can output various information, including the determined distance information, direction information, and the like, to the outside. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0137] Of course, in order to simplify, Figure 12 Only some of the components of the electronic device 10 related to the present disclosure are shown in FIG. 1, and components such as a bus, an input / output interface, and the like are omitted. In addition, the electronic device 10 can include any other appropriate components according to a specific application.

[0138] EXEMPLARY COMPUTER PROGRAM PRODUCT AND COMPUTER-READABLE STORAGE MEDIUM

[0139] In addition to the above-mentioned methods and devices, embodiments of the present disclosure can also be a computer program product including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the specification.

[0140] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0141] In addition, embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the specification.

[0142] The computer readable storage medium can be any combination of one or more computer readable medium(s). The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] The above generally describes the basic principles of the disclosure in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the disclosure are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as the various embodiments of the disclosure must have. In addition, the specific details of the above disclosure are only for the purpose of example and for the purpose of understanding, and the above details do not limit the disclosure to the above specific details.

[0144] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0145] The block diagrams of the devices, apparatuses, equipment, systems involved in the disclosure are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0146] The methods and apparatus of the present disclosure can be implemented in a number of ways. For example, the methods and apparatus of the present disclosure can be implemented using software, hardware, firmware, or any combination of software, hardware, and firmware. The order of any steps described above is merely exemplary and the steps of the methods of the present disclosure need not be performed in the order described unless otherwise specified. Furthermore, in some embodiments, the present disclosure can also be implemented as a program for running on a computer or a processor to implement the methods according to the present disclosure. Thus, the present disclosure also covers a record medium storing the program in a non-transitory manner. The program can be realized in any of the following forms: an object code, a code composed of a program language that can be interpreted by a computer, or a code composed of a language that can be converted into a machine language by an interpreter.

[0147] It is also noted that the methods of the present disclosure can be implemented by a computer or a processor of a computer. Furthermore, the present disclosure also covers a computer program for running on a computer or a processor to implement the methods according to the present disclosure.

[0148] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0149] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A method for accelerating inference in a neural network model, comprising: Acquire image training data, text training data, or speech training data; Determine the first neural network model to be accelerated; The preset operation of the preset network layer in the first neural network model is converted into the first operation to obtain the second neural network model. The first operation is used to simulate the operation logic of the target operation. Based on the image training data, the text training data, or the speech training data, the second neural network model is subjected to quantization perception training according to a preset bit width to obtain a quantized third neural network model. The preset bit width is the bit width supported by the neural network accelerator for data processing. In the quantization perception training process, the forward and backward propagation calculations and pseudo-quantization node calculations of the model are implemented using floating-point numbers that satisfy the fixed-point number rules of the preset bit width. The first operation of the third neural network model is converted into the target operation to obtain the accelerated target neural network model corresponding to the first neural network model.

2. The method according to claim 1, wherein, The target operation is a lookup table operation; the first operation includes: Determine the first sub-range to which the first input data of the first operation belongs; Based on a preset method, determine the calculation results corresponding to the two endpoints of the first sub-range respectively; Based on the first input data, the first sub-range, and the calculation results corresponding to the two endpoints of the first sub-range, the first calculation result corresponding to the first input data is obtained by linear interpolation.

3. The method according to claim 1, wherein, The target operation is a lookup table operation; after determining the first neural network model to be accelerated, the method further includes: Based on the input data of the preset operation during the training process of the first neural network model, the first input range is determined; Based on a preset number of segments, the first input range is divided into segments to obtain the preset number of sub-ranges; For each subrange, based on the preset operation, determine the operation results corresponding to the two endpoints of the subrange respectively; The calculation results corresponding to each of the sub-ranges and each endpoint of each sub-range are stored for use in the lookup table operation.

4. The method according to claim 1, wherein, The first neural network model to be accelerated includes: The first neural network model is obtained by training a pre-established neural network based on preset training data.

5. The method according to claim 1, wherein, The preset operations include exponential operations and / or logarithmic operations.

6. A method for accelerating inference in a neural network model, comprising: Acquire data to be processed, which may be image data, text data, or voice data; Based on the accelerated target neural network model corresponding to the pre-obtained first neural network model, the output data corresponding to the data to be processed is determined; the target neural network model is obtained based on the accelerated inference method of neural network model according to any one of claims 1-5.

7. The method according to claim 6, wherein, The target operation is a lookup table operation, and the operation process includes: Determine the target sub-range to which the input data of the table to be searched belongs; Based on the target sub-range, the calculation results corresponding to the two endpoints of the target sub-range are obtained by looking up a table; Based on the input data of the lookup table, the target sub-range, and the calculation results corresponding to the two endpoints of the target sub-range, the target calculation result corresponding to the input data of the lookup table is obtained by linear interpolation.

8. An acceleration device for neural network model inference, comprising: The first acquisition module is used to acquire image training data, text training data, or voice training data. The first determining module is used to determine the first neural network model to be accelerated; The first processing module is used to convert the preset operation of the preset network layer in the first neural network model into the first operation to obtain the second neural network model. The first operation is used to simulate the operation logic of the target operation. The second processing module is used to perform quantization perception training on the second neural network model according to a preset bit width based on the image training data, the text training data, or the speech training data, to obtain a quantized third neural network model. The preset bit width is the bit width supported by the neural grid accelerator for data processing. In the quantization perception training process, the forward and backward propagation calculations and pseudo-quantized node calculations of the model are implemented using floating-point numbers that satisfy the fixed-point number rules of the preset bit width. The third processing module is used to convert the first operation of the third neural network model into the target operation to obtain the accelerated target neural network model corresponding to the first neural network model.

9. An acceleration device for neural network model inference, comprising: The second acquisition module is used to acquire data to be processed, which is image data, text data, or voice data. The fourth processing module is used to determine the output data corresponding to the data to be processed based on the accelerated target neural network model corresponding to the pre-obtained first neural network model; the target neural network model is obtained based on the accelerated method of neural network model inference according to any one of claims 1-5.

10. A computer-readable storage medium storing a computer program for performing the accelerated inference method for a neural network model according to any one of claims 1-7.

11. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the accelerated neural network model inference method according to any one of claims 1-5, or to execute the instructions to implement the accelerated neural network model inference method according to claim 6 or 7.

Images