Chip adaptation method and device for deep learning model, chip and medium

Abstract

The application discloses a chip adaptation method and device of a deep learning model, a chip and a medium. The method comprises the following steps: loading a target model to be adapted; verifying whether a chip adaptation condition is met according to a numerical relationship between a target accuracy of a target computing unit and a parameter accuracy of the target model; if yes, loading each input vector set and a hidden state vector data range, and transferring them to a target memory; calling each built-in computing instruction, and performing the following steps: obtaining an input vector quantization scale and a hidden state vector quantization scale according to each input vector set and the hidden state vector data range, and calculating corresponding input weight quantization matrices, cycle weight quantization matrices and quantization scales according to the target accuracy and an input weight matrix and a cycle weight matrix of each gate structure, and outputting the matrices and the quantization scales to an engine file to generate an adapted target model. Through the technical scheme of the embodiment of the application, the complexity of quantization calculation can be reduced, and the efficiency of parameter quantization can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to neural network technology, in particular to a chip adaptation method and device of a deep learning model, a chip and a medium, and more particularly to a chip adaptation scenario of a recurrent neural network. BACKGROUND

[0002] At present, recurrent neural networks have been widely used in text recognition, speech recognition, natural language processing and other scenarios. While the model accuracy is improved, a huge amount of parameters and calculations are also introduced. Model quantization can reduce memory bandwidth and storage occupation, reduce power consumption, improve throughput and reduce latency.

[0003] The existing parameter quantization method of the recurrent neural network usually needs to determine the iteration interval according to the amplitude of the quantization data variation, so as to adjust the quantization parameters in the neural network according to the iteration interval.

[0004] When the above method is applied to the training or fine-tuning process of the recurrent neural network on the chip, repeated iterations are required, which makes the process time-consuming and complex, and brings huge computing power cost and time cost to the machine learning chip. Therefore, how to improve the efficiency of parameter quantization and reduce the consumption of storage space with less accuracy loss is a problem to be solved at present. SUMMARY

[0005] Embodiments of the present application provide a chip adaptation method, device, chip and medium of a deep learning model to reduce the amount of calculation in the model quantization process and improve the efficiency of parameter quantization.

[0006] In a first aspect, embodiments of the present application provide a chip adaptation method of a deep learning model, executed by a machine learning chip, comprising:

[0007] loading a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, the gate structure having a matching input weight matrix and a recurrent weight matrix;

[0008] verifying whether a chip adaptation condition for the target model is met according to a numerical relationship between a target accuracy of a target computing unit in the machine learning chip and a parameter accuracy of the target model to be adapted;

[0009] if yes, loading an input vector set and a hidden state vector data range respectively matched with each recurrent neural network;

[0010] carrying each of the recurrent neural networks, the input vector set and the hidden state vector data range to a target memory close to the computing unit in the machine learning chip by at least one level of memory carrying;

[0011] The computing units call each computing instruction built in the machine learning chip according to the data in the target memory, and perform the following operations:

[0012] According to the input vector set and the hidden state vector data range matched with the recurrent neural network, the input vector quantization scale and the hidden state vector quantization scale are calculated;

[0013] According to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale, and the input weight matrix and the recurrent weight matrix of each gate structure, the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated respectively.

[0014] The input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the adapted target model.

[0015] In a second aspect, the embodiment of the present application also provides a chip adaptation device for a deep learning model, which comprises:

[0016] A target model loading module is configured to load a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, and the gate structure having a matched input weight matrix and a recurrent weight matrix;

[0017] An adaptation condition verification module is configured to verify whether the chip adaptation condition for the target model is met according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted;

[0018] A data loading module is configured to load an input vector set and a hidden state vector data range matched with each recurrent neural network if the numerical relationship is met.

[0019] A data carrying module is configured to carry each recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip through at least one level of memory carrying.

[0020] An instruction calling module is configured to call each computing instruction built in the machine learning chip according to the data in the target memory through the computing units, and perform the following operations:

[0021] A quantization scale calculation module is configured to calculate the input vector quantization scale and the hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network.

[0022] a quantization parameter calculation module, configured to calculate input weight quantization matrices, cycle weight quantization matrices and quantization scales corresponding to each gate structure respectively according to the target precision, input vector quantization scales, hidden state vector quantization scales and input weight matrices and cycle weight matrices of each gate structure;

[0023] a target model adaptation module, configured to output the input weight quantization matrices, cycle weight quantization matrices and quantization scales corresponding to each gate structure respectively to an engine file to generate an adapted target model.

[0024] In a third aspect, an embodiment of the present application further provides a machine learning chip, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the chip adaptation method of the deep learning model according to any embodiment of the present application when executing the program.

[0025] In a fourth aspect, an embodiment of the present application further provides a storage medium of computer executable instructions, which stores a computer program, wherein the program is executed by a processor to implement the chip adaptation method of the deep learning model according to any embodiment of the present application.

[0026] The application loads a target model including at least one recurrent neural network, verifies whether the chip adaptation condition of the target model is met according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted, if yes, loads the input vector set and the hidden state vector data range respectively matched with each recurrent neural network, carries the recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip through at least one memory carrying mode, then, according to the data in the target memory, each computing instruction in the machine learning chip is called to execute the calculation of the input vector quantization scale and the hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network, then the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated according to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the recurrent weight matrix of each gate structure, and the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the operation of the adapted target model, when the recurrent neural network loaded in the machine learning chip does not match the computing power of the chip, a new on-chip quantization method of model parameters is proposed, the problems that the parameter quantization method of the recurrent neural network in the prior art is time-consuming and has high computational complexity, which brings huge power cost and time cost to the machine learning chip are solved, the on-chip quantization complexity and the calculation time are greatly reduced under the condition of small accuracy loss, and the efficiency of parameter quantization in the recurrent neural network is improved. In the model inference scene with high real-time requirement, the calculation time can be greatly reduced under the premise of ensuring the accuracy.

[0027] In particular, the entire parameter quantization and inference method can be implemented by hardware, the matched quantized accuracy 2D computing capability and 1D computing capability are configured, special 1D computing instructions such as round, clip and the like are supported, the inference capability is greatly improved, and the on-chip storage space is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1a is a flowchart of a chip adaptation method of a deep learning model in embodiment one of the application;

[0029] Figure 1b is a structure schematic diagram of a network unit in a long short-term memory network in embodiment one of the application;

[0030] Figure 2ais a flow chart of a chip adaptation method of a deep learning model in Embodiment Two of the present application;

[0031] Figure 2b is a parameter quantization flowchart of a recurrent neural network in Embodiment Two of the present application;

[0032] Figure 3a is a flow chart of a chip adaptation method of a deep learning model in Embodiment Three of the present application;

[0033] Figure 3b is a schematic diagram of a language translation model in Embodiment Three of the present application;

[0034] Figure 3c is a schematic diagram of an LSTM network unit iterative calculation in Embodiment Three of the present application;

[0035] Figure 3d is an inference flowchart in an adapted language translation model in Embodiment Three of the present application;

[0036] Figure 3e is a structure schematic diagram of a quantization calculation of an adapted language translation model in Embodiment Three of the present application;

[0037] Figure 3f is a flow chart of an inference method in an adapted language translation model in Embodiment Three of the present application;

[0038] Figure 4 is a structure schematic diagram of a chip adaptation device of a deep learning model in Embodiment Four of the present application;

[0039] Figure 5 is a structure schematic diagram of a machine learning chip in Embodiment Five of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0041] Embodiment One

[0042] Figure 1aA flowchart of a chip adaptation method for a deep learning model provided by Embodiment One of the present application. This embodiment can be applied to the quantization of model parameters in a recurrent neural network. The method can be performed by a chip adaptation device for a deep learning model. The device can be implemented in software and / or hardware (firmware) and can be integrated into a tool for providing recurrent neural network parameter quantization services. For example, it can be integrated into a machine learning chip. The method specifically includes the following steps:

[0043] S110, load the target model to be adapted. The target model includes at least one recurrent neural network, which includes at least one network unit, and the network unit includes at least one gate structure, which has a matching input weight matrix and a recurrent weight matrix.

[0044] The target model can be a model that has completed the model training process and needs to be loaded and calculated in a machine learning chip. The recurrent neural network can be a type of recurrent neural network that takes sequence data as input, performs recursion in the evolution direction of the sequence, and connects all nodes (network units) in a chain. Common recurrent neural networks include bidirectional recurrent neural networks (Bi-RNN), gate recurrent units (GRU), and long short-term memory networks (LSTM).

[0045] In a specific example, a recurrent neural network can include multiple chain-connected network units, each network unit including multiple gate structures, and each gate structure having a matching input weight matrix and a recurrent weight matrix.

[0046] For example, an LSTM network generally includes one or more LSTM network units. As shown in Figure 1b The structure of an LSTM network unit is shown in the diagram. It mainly includes four gate structures: input gate, output gate, forget gate, and cell gate.

[0047] The input weight matrix can be a matrix composed of values that adjust the input to the gate structure during the generation of the output result by the gate structure. The recurrent weight matrix can be a matrix composed of values that modify the corresponding components of the recurrent vector.

[0048] In this embodiment, the recurrent neural network to be quantized specifically refers to a network used to meet a set data processing function after model training by a training sample set. Each model parameter in the recurrent neural network has met the accuracy requirement of model training. Correspondingly, the data accuracy of each matrix element in the input weight matrix and the recurrent weight matrix of each gate structure of each network unit in the recurrent neural network meets the accuracy requirement of model training. For example, the data accuracy of each matrix element can be fp64 (double precision), fp32 (single precision), or fp16 (half precision), etc.

[0049] In S120, whether the chip adaptation condition for the target model is met is verified according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted.

[0050] The target computing unit can refer to a computing unit with a set computing power in the machine learning chip. The target accuracy can refer to the operation accuracy corresponding to the target computing unit. For example, it can be double-precision computing power (64 bits, FP64), single-precision computing power (32 bits, FP32), half-precision computing power (16 bits, FP16), and integer computing power (INT8 or INT4), etc. Generally, the higher the number of digits, the higher the accuracy, the higher the complexity of operations that can be supported, and the wider the application scenarios.

[0051] The parameter accuracy can refer to the numerical accuracy of each model parameter in the target model that completes the model training. Similarly, the parameter accuracy can be double-precision computing power (64 bits, FP64), single-precision computing power (32 bits, FP32), half-precision computing power (16 bits, FP16), and integer computing power (INT8 or INT4), etc.

[0052] Generally, the machine learning chip includes multiple computing units with different accuracies, such as double-precision computing power (64 bits, FP64), single-precision computing power (32 bits, FP32), half-precision computing power (16 bits, FP16), and integer computing power (INT8 or INT4), etc. The computing units with different accuracies have different computing powers. Generally, the larger the computing power, the lower the computing accuracy, but the computing time will be correspondingly reduced.

[0053] In actual use, it is necessary to determine which precision computing unit in the machine learning chip to use for the target model according to the actual application scenario of the target model. In a specific example, when the target model is applied in a language translation scenario, it has a high real-time requirement, and therefore the highest computing unit (i.e., the target computing unit) in the machine learning chip can be used to adapt to the target model to maximize the real-time operation. Correspondingly, in this embodiment, the target computing unit can be the highest computing unit, and the target precision can be the precision of the highest computing unit.

[0054] Further, it is necessary to first verify whether parameter quantization is required for the target model. When it is determined that parameter quantization is required, the corresponding quantization scale is calculated synchronously. For example, quantization from FP32 to INT8.

[0055] The chip adaptation condition can refer to the precision adaptation requirement between the target model and the machine learning chip. For example, when the parameter precision of the target model to be adapted is higher than the target precision, it is determined that the chip adaptation condition for the target model is met. At this time, the target model needs to be quantized in the machine learning chip.

[0056] It needs to be noted again that when the chip adaptation condition for the target model is met, the application does not quantize all model parameters of the target model according to the target precision, but the inventors, by fully considering the accuracy and timeliness requirements, select to quantize only the input weight matrix and the recurrent weight matrix of each gate structure in the recurrent neural network, and the remaining model parameters still maintain the model precision obtained by training the target model.

[0057] S130, if yes, load the input vector set and the hidden state vector data range respectively matched with each recurrent neural network.

[0058] The input vector set can refer to a calibration data set matched with the recurrent neural network. Specifically, the input vector set can be a subset of the training sample set used to train each recurrent neural network in the target model.

[0059] The hidden state vector data range can be determined according to the type of the recurrent neural network. For example, if the recurrent neural network is LSTM, since the value range of the activation function is (-1, 1), the hidden state vector data range should be (-1, 1). The quantization scale can refer to the standard scale required when mapping values from a larger value set to a smaller set. The input vector quantization scale can refer to the quantization scale required when quantizing the input vector to a pre-set set. The hidden state vector quantization scale can refer to the quantization scale required when quantizing the hidden state vector to a pre-set set.

[0060] S140, the recurrent neural network, the input vector set and the hidden state vector data range are carried to the target memory close to the computing unit in the machine learning chip by at least one level of memory carrying.

[0061] The target memory can refer to a memory block closest to the computing unit.

[0062] Memory carrying can refer to carrying data in the lower memory to the upper memory. Generally, the farther the distance from the target memory, the larger the storage capacity, but the slower the carrying speed.

[0063] S150, according to the data in the target memory, each computing instruction in the machine learning chip is called to perform subsequent operation steps (steps S160 to S180) by each computing unit.

[0064] The computing instruction can refer to a pre-set instruction for quantitative calculation of the target model. For example, it can include instructions for performing steps S160 to S180.

[0065] S160, according to the input vector set and the hidden state vector data range matched with the recurrent neural network, the input vector quantization scale and the hidden state vector quantization scale are calculated.

[0066] In an optional embodiment, according to the input vector set and the hidden state vector data range matched with the recurrent neural network, the input vector quantization scale and the hidden state vector quantization scale are calculated, which can include:

[0067] According to the value domain distribution of the input vector set, the input vector numerical range corresponding to the input vector set is calculated, and the input vector quantization threshold is determined according to the input vector numerical range; the input vector quantization scale is calculated according to the input vector quantization threshold, the pre-set quantization range and the quantization mode; the hidden state vector quantization threshold is determined according to the hidden state vector data range, and the hidden state vector quantization scale is calculated according to the hidden state vector quantization threshold, the pre-set quantization range and the quantization mode.

[0068] The value domain distribution of the input vector set can refer to the value distribution information of each tensor in the input vector set. For example, the quantization threshold of the input vector set can be calculated by maximum-minimum, relative entropy (Kullback-Leibler divergence, KL) divergence and percentile.

[0069] The preset quantization range can refer to a preset quantization range matching the target precision of the target computing unit in the machine learning chip. For example, when the target model is applied in a language translation scenario, the real-time requirement is high, and therefore the highest computing unit in the machine learning chip can be fully utilized to adapt to the target model to maximize the real-time operation. If the machine learning chip can include the computing power of fp32, int8 and int32, the target precision of the target computing unit in the machine learning chip can be set to the highest computing power int8. At this time, the quantization range can be (0, 255) or (-127, 127), which is not limited in the embodiment. The quantization manner can refer to the manner used when the input weight matrix and the recurrent weight matrix are quantized, such as a symmetric quantization method or an asymmetric quantization method. In the embodiment, the symmetric quantization method is preferred as the quantization manner.

[0070] In a specific example, the value range of the input vector set can be calculated by the value range distribution of the input vector set, so that the input vector quantization threshold is determined as the absolute value of the maximum positive value or the minimum negative value. If the preset quantization range is selected as (-127, 127), the input vector quantization scale S Similarly, if the hidden state vector data range is (-1, 1), the hidden state vector quantization threshold is 1, and if the preset quantization range is selected as (-127, 127), the hidden state vector quantization scale S Therefore, the input vector quantization scale S x is calculated by the input vector quantization threshold, the preset quantization range and the quantization manner. The hidden state vector quantization scale S h is calculated by the hidden state vector quantization threshold, the preset quantization range and the quantization manner. The above provides an effective data basis for subsequent gate quantization.

[0071] S170, according to the target precision, the input vector quantization scale, the hidden state vector quantization scale, and the input weight matrix and the recurrent weight matrix of each gate structure, the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated.

[0072] The input weight quantization matrix can refer to the quantized input weight matrix, and the recurrent weight quantization matrix can refer to the quantized recurrent weight matrix. That is, the data precision of each matrix element in the input weight quantization matrix and the recurrent weight quantization matrix is the data precision after the matrix elements in the input weight matrix and the recurrent weight matrix are quantized as a whole. For example, the data precision of each matrix element in the input weight quantization matrix and the recurrent weight quantization matrix can be int16, int8 or int4, and int is an integer type.

[0073] Wherein, since multiple gate structures are included in the recurrent neural network, the input weight matrix and the recurrent weight matrix of each gate structure need to be quantized into matching input weight quantization matrix and recurrent weight quantization matrix respectively. In the embodiment, the specific description of the quantization process can be made only with one gate structure as an example, and the quantization process of other gate structures is completely consistent.

[0074] Correspondingly, in an optional embodiment, according to the target precision, the input vector quantization scale, the hidden state vector quantization scale, and the input weight matrix and the recurrent weight matrix of each gate structure, the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated, which can include:

[0075] obtaining the target input weight matrix and the target recurrent weight matrix corresponding to the target gate structure currently processed; splicing the first result obtained by multiplying the target input weight matrix with the input vector quantization scale and the second result obtained by multiplying the target recurrent weight matrix with the hidden state vector quantization scale to obtain a target splicing matrix; calculating the target quantization scale corresponding to the target gate structure according to the target splicing matrix; and quantizing the target splicing matrix according to the target precision and the target quantization scale corresponding to the target gate structure to obtain the target input weight quantization matrix and the target recurrent weight quantization matrix corresponding to the target gate structure.

[0076] Wherein, the target gate structure can refer to the gate structure currently processed in the recurrent neural network, and exemplarily, can be any one of the input gate i, the output gate o, the forget gate f or the cell gate c in a network unit. The target input weight matrix can refer to the input weight matrix W corresponding to the target gate structure; and the target recurrent weight matrix can refer to the recurrent weight matrix R corresponding to the target gate structure.

[0077] The first result can refer to the result obtained by multiplying the target input weight matrix with the input vector quantization scale, and the second result can refer to the result obtained by multiplying the target recurrent weight matrix with the hidden state vector quantization scale. Exemplarily, taking the input gate i as an example, the input weight matrix corresponding to the input gate i is W i , and the recurrent weight matrix corresponding to the input gate i is R i . The first result A1 = W i *S x , and the second result A2 = R i *S h .

[0078] The target splicing matrix can refer to the new matrix obtained by splicing the first result and the second result, and the target splicing matrix Further, according to the quantization threshold of the target splicing matrix, the preset quantization range and the quantization mode, a target quantization scale corresponding to the target gate structure is obtained. Then, according to the target quantization scale and the target accuracy of the target computing power calculation unit in the machine learning chip, the target splicing matrix is quantized to obtain a target input weight quantization matrix and a target cycle weight quantization matrix corresponding to the target gate structure.

[0079] On the basis of the above-mentioned embodiments, the embodiments of the present application can also use special computing instructions to calculate the input vector quantization scale, the hidden state vector quantization scale, the input weight quantization matrix and the cycle weight quantization matrix corresponding to each gate structure; wherein the special computing instructions are encapsulated by using the computing logic constructed by any one of the maximum and minimum value, the relative entropy divergence and the percentile number algorithm.

[0080] S180, the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the adapted target model.

[0081] Specifically, the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale of the input gate i, the output gate o, the forget gate f and the cell gate c in each network unit are obtained by the above-mentioned method, and then the input weight quantization matrix and the cycle weight quantization matrix are stored in the corresponding parameters of the machine learning chip (typically, Tensor processor), and the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale are output to the Engine (engine) file to generate the adapted target model.

[0082] The application loads a target model including at least one recurrent neural network, verifies whether the chip adaptation condition of the target model is met according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted, loads the input vector set and the hidden state vector data range respectively matched with each recurrent neural network if yes, carries the recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip through at least one memory carrying mode, then calls each computing instruction built-in in the machine learning chip according to the data in the target memory through each computing unit, calculates the input vector quantization scale and the hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network, calculates the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure respectively according to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the recurrent weight matrix of each gate structure, and outputs the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure to the engine file to generate the operation of the adapted target model. When the recurrent neural network loaded in the machine learning chip does not match the computing power of the chip, a new on-chip quantization method of model parameters is proposed, which solves the problems of long process time and high computational complexity of the parameter quantization method of the recurrent neural network in the prior art, brings huge power cost and time cost to the machine learning chip, and realizes the on-chip quantization calculation with less precision loss, greatly reduces the complexity and calculation time of on-chip quantization calculation, and improves the efficiency of parameter quantization in the recurrent neural network. In particular, the whole parameter quantization and inference method can be realized by hardware, the matched quantized accuracy 2D computing capability and 1D computing capability are configured, special 1D computing instructions such as round, clip, etc. are supported, the inference capability is greatly improved, and the occupation of on-chip storage space is greatly reduced.

[0083] Embodiment two

[0084] Figure 2a A flowchart of a chip adaptation method of a deep learning model is provided for embodiment two of the application. Based on the above-mentioned embodiment, the target quantization scale corresponding to the target gate structure calculated according to the target splicing matrix is refined, that is, the maximum value of the matrix elements in the target splicing matrix is obtained after the absolute value processing of the matrix elements in the target splicing matrix, and the target quantization scale corresponding to the target gate structure is calculated according to the maximum value of the matrix elements, the preset quantization range and the quantization method.

[0085] Correspondingly, the target splicing matrix is quantized according to a target precision in the machine learning chip, a target quantization scale corresponding to the target gate structure, to obtain a target input weight quantization matrix and a target recurrent weight quantization matrix corresponding to the target gate structure, refined as: dividing the target splicing matrix by the target quantization scale to obtain an intermediate result matrix; according to the target precision, performing rounding and truncation processing on the intermediate result matrix to obtain a target quantization splicing matrix; and performing unsplicing processing on the target quantization splicing matrix to obtain the target input weight quantization matrix and the target recurrent weight quantization matrix.

[0086] As shown in the method, the method comprises the following specific steps: Figure 2a

[0087] S210, loading a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, the gate structure having a matching input weight matrix and a recurrent weight matrix.

[0088] Specifically, the recurrent neural network to be quantized is obtained, and accordingly the type of each gate structure in each network unit in the recurrent neural network can be known, and in the embodiment of the application, the four types of gate structures include an input gate i, an output gate o, a forget gate f, and a cell gate c. Correspondingly, each gate structure contains a corresponding input weight matrix and a recurrent weight matrix.

[0089] S220, verifying whether a chip adaptation condition of the target model is met according to a numerical relationship between a target precision of a target computing unit in the machine learning chip and a parameter precision of the target model to be adapted.

[0090] S230, if yes, loading an input vector set and a hidden state vector data range respectively matched with each recurrent neural network.

[0091] S240, carrying each of the recurrent neural network, the input vector set, and the hidden state vector data range to a target memory close to the computing unit in the machine learning chip by at least one level of memory carrying.

[0092] S250, calling each computing instruction in the machine learning chip according to the data in the target memory by each computing unit to execute the following S260-S2160.

[0093] S260, calculating an input vector numerical range corresponding to the input vector set according to a value range distribution of the input vector set, and determining an input vector quantization threshold according to the input vector numerical range.

[0094] ​S270, calculating an input vector quantization scale according to the input vector quantization threshold, the preset quantization range, and the quantization manner.

[0095] S280, determining a hidden state vector quantization threshold according to the hidden state vector data range, and calculating a hidden state vector quantization scale according to the hidden state vector quantization threshold, the preset quantization range, and the quantization manner.

[0096] S290, obtaining a target input weight matrix and a target loop weight matrix corresponding to a target gate structure being processed.

[0097] S2100, splicing a first result obtained by multiplying the target input weight matrix by the input vector quantization scale and a second result obtained by multiplying the target loop weight matrix by the hidden state vector quantization scale to obtain a target splicing matrix.

[0098] S2110, obtaining a maximum value of matrix elements in the target splicing matrix after taking an absolute value of each matrix element in the target splicing matrix.

[0099] It is worth noting that in the embodiment of the present application, the value type of each value in the input vector set is float32 (i.e. fp32), in order to improve the efficiency of parameter quantization, it is expected to quantize each value in the input vector set to the target precision of the target computing unit in the machine learning chip, for example, it can be of int8 type.

[0100] Wherein, taking the input gate i as an example, the maximum value of the matrix elements can be calculated by function, is the target splicing matrix.

[0101] S2120, calculating a target quantization scale corresponding to the target gate structure according to the maximum value of the matrix elements, the preset quantization range, and the quantization manner.

[0102] Specifically, the preset quantization range is determined by the target precision of the target computing unit in the machine learning chip, when the target precision of the target computing unit in the machine learning chip is int8, the preset quantization range is (-127, 127).

[0103] Specifically, taking the input gate i as an example, it is known that the maximum value of the matrix elements is max (fabs (W i_concat_fp32 )), the preset quantization range is (-127, 127), and the quantization manner is a symmetric quantization method, thus the target quantization scale

[0104] S2130, dividing the target splicing matrix by the target quantization scale to obtain an intermediate result matrix.

[0105] The intermediate result matrix can refer to a matrix obtained by dividing the target splicing matrix by the target quantization scale. Specifically, taking the input gate i as an example, the intermediate result matrix is

[0106] S2140, according to the target precision, rounding and truncation processing is performed on the intermediate result matrix to obtain a target quantization splicing matrix.

[0107] The rounding can refer to an operation of rounding each matrix element in the intermediate result matrix to only retain an integer. For example, a rounding to even (Round2even) method can be used to perform the rounding operation. That is, when the matrix element is exactly X.5 data, the above data needs to be rounded to the nearest even integer, for example, 4.5 is rounded to even integer 4, and 5.5 is rounded to even integer 6. When the matrix element is not exactly X.5 data, a normal rounding operation can be directly performed, for example, 4.4 is rounded to 4, and 5.6 is rounded to 6.

[0108] The truncation can refer to an operation of cutting each matrix element in the intermediate result matrix after rounding according to a set requirement. For example, the minimum value and the maximum value in the quantization range can be used as a standard. Specifically, taking the target precision of the machine learning chip as int8 as an example, the values between (-127, 127) can be cut off for subsequent use. That is, the matrix element after rounding falls within the range of (-127, 127) is kept as it is, and the matrix element after rounding falls outside the range of (-127, 127) is cut off to -127 or 127. The target quantization splicing matrix can refer to a matrix obtained by rounding and truncation processing on the intermediate result matrix.

[0109] It is worth noting that in the embodiments of the present application, the target precision of the target computing unit in the machine learning chip is preferably used to round and truncate the intermediate result matrix.

[0110] Specifically, taking the input gate i as an example, the rounding processing on the intermediate result matrix can be performed by Further, the target quantization splicing matrix:

[0111]

[0112] S2150, the target quantization splicing matrix is disassembled to obtain the target input weight quantization matrix and the target cycle weight quantization matrix.

[0113] The de-splicing processing can refer to inverse order processing according to the acquisition process of the target splicing matrix, to obtain the target input weight quantization matrix and the target cycle weight quantization matrix corresponding to the target door structure currently processed.

[0114] S2160, output the target input weight quantization matrix, the target cycle weight quantization matrix and the target quantization scale corresponding to each door structure respectively to the engine file, to generate the adapted target model.

[0115] Specifically, based on the above method, the input door i, the output door o, the forget door f and the cell door c of each network unit are processed respectively to obtain the scale matrix [s wi , s wo , s wf , s wc ] composed of target quantization scales. It is worth noting that if the recurrent neural network is a bidirectional LSTM, the scale matrix contains two scale matrices of forward and reverse directions. Further, the scale matrix [s wi , s wo , s wf , s wc ] is stored in the Tensor processor of the LSTM, the quantized Wi_int8, Wo_int8, Wf_int8 and Wc_int8 are stored in the W_int8 in the Tensor processor; the quantized Ri_int8, Ro_int8, Rf_int8 and Rc_int8 are stored in the R_int8 in the Tensor processor; and then, the stored W_int8, R_int8 and [s wi , s wo , s wf , s wc ] are output to the Engine file, to complete the generation of the adapted target model corresponding to the recurrent neural network.

[0116] The technical scheme of the embodiment of the present application loads a target model comprising at least one recurrent neural network, verifies whether the chip adaptation condition for the target model is met according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted, if yes, loads the input vector set and the hidden state vector data range respectively matched with each recurrent neural network, carries the recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip through at least one level of memory carrying, then calls each computing instruction built-in in the machine learning chip according to the data in the target memory, executes the input vector numerical range corresponding to the input vector set according to the value range distribution of the input vector set, determines the input vector quantization threshold according to the input vector numerical range, calculates the input vector quantization scale according to the input vector quantization threshold, the preset quantization range and the quantization mode, determines the hidden state vector quantization threshold according to the hidden state vector data range, calculates the hidden state vector quantization scale according to the hidden state vector quantization threshold, the preset quantization range and the quantization mode, splices the first result obtained by multiplying the target input weight matrix of the recurrent neural network and the input vector quantization scale and the second result obtained by multiplying the target recurrent weight matrix and the hidden state vector quantization scale to obtain a target splicing matrix, obtains the maximum value of the matrix elements in the target splicing matrix after taking the absolute value of each matrix element in the target splicing matrix, calculates the target quantization scale corresponding to the target gate structure according to the maximum value of the matrix elements, the preset quantization range and the quantization mode, divides the target splicing matrix by the target quantization scale to obtain an intermediate result matrix, and performs rounding and truncation processing on the intermediate result matrix according to the target accuracy to obtain a target quantization splicing matrix, finally, performs dis-splicing processing on the target quantization splicing matrix to obtain a target input weight quantization matrix and a target recurrent weight quantization matrix, and outputs the target input weight quantization matrix, the target recurrent weight quantization matrix and the target quantization scale corresponding to each gate structure to the engine file to generate the adapted target model, solving the problem of long process time and high computational complexity of the parameter quantization method of the recurrent neural network in the prior art, bringing huge computing power cost and time cost, realizing great reduction of the complexity and calculation time of quantization calculation with less accuracy loss, and improving the efficiency of parameter quantization in the recurrent neural network.

[0117] Figure 2bA parameter quantization process diagram of a recurrent neural network is provided for an embodiment of the present application. Specifically, first, an already trained Open Neural Network Exchange (ONNX) model is loaded, and a corresponding input vector set is obtained. Then, the value range distribution of the input vector set is determined to obtain the input vector numerical range and the input vector quantization threshold corresponding to the input vector set, and the input vector quantization scale is calculated. Further, the hidden state vector quantization scale is set to 1 / 127 according to the hidden state vector quantization threshold, the preset quantization range and quantization mode matched with the target accuracy of the target computing unit in the machine learning chip. Then, the target input weight matrix and the target recurrent weight matrix corresponding to each target gate structure included in the ONNX model are obtained, and the target splicing matrix is obtained by splicing processing. After taking the absolute value of each matrix element in the target splicing matrix, the maximum value of the matrix elements in the target splicing matrix is obtained, and the target quantization scale corresponding to the target gate structure is calculated. The target splicing matrix is divided by the target quantization scale, and then the integer and truncation processing are performed to obtain the target quantization splicing matrix. Then, the target quantization splicing matrix is disassembled to obtain the target input weight quantization matrix and the target recurrent weight quantization matrix. Further, the calculation of all gate structures is completed. When the calculation of all gate structures is completed, the quantized target input weight quantization matrix Wi_int8, Wo_int8, Wf_int8 and Wc_int8 are stored in W_int8 in the Tensor processor, and the quantized target recurrent weight quantization matrix Ri_int8, Ro_int8, Rf_int8 and Rc_int8 are stored in R_int8 in the Tensor processor. Finally, the stored W_int8, R_int8 and scale matrix [s wi ,s wo ,s wf ,s wc ] are output to the Engine file to complete the generation of the adapted target model corresponding to the recurrent neural network. Thus, by independently quantizing the input weight matrix and the recurrent weight matrix in the recurrent neural network according to the input gate, the output gate, the forget gate and the cell gate respectively, and by symmetrically quantizing the hidden state vector according to the hidden state vector data range, the floating point multiplication calculation of the gate structure matrix is converted to int8 multiplication, so that the storage amount is reduced to 1 / 4 of the previous one. In the case of small accuracy loss, the efficiency of on-chip parameter quantization is improved, and the consumption of on-chip storage space is reduced.

[0118] Embodiment three

[0119] Figure 3aA flowchart of a chip adaptation method of a deep learning model provided for the third embodiment of the present application is provided, which is based on the above-mentioned embodiments and additionally includes the following operations: after the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the adapted target model, the source language sequence and the adapted language translation model in the engine file are loaded; the source language sequence and the adapted language translation model are carried to the target memory through at least one level of memory carrying; each calculation unit calls the built-in calculation instructions in the machine learning chip according to the data in the target memory to perform the following operations: the source language sequence is converted into a quantized input sequence according to the target accuracy; the quantized input sequence is input into the adapted language translation model; the calculation result output by each LSTM network unit is obtained through step-by-step iterative calculation according to the quantized sequence value at the matching time in the quantized input sequence, the hidden state quantized value at the previous time of the matching time, and the quantization parameters of each gate structure in the LSTM network unit, so as to finally output the target translation speech sequence.

[0120] As shown in Figure 3a , the method specifically includes the following steps:

[0121] S310, loading a target model to be adapted, the target model including at least one recurrent neural network, the recurrent neural network including at least one network unit, the network unit including at least one gate structure, the gate structure having an input weight matrix and a cycle weight matrix.

[0122] In an optional embodiment, the target model is a language translation model, as shown in Figure 3b , which is a schematic diagram of a language translation model. The language translation model specifically includes a long short-term memory network (LSTM) encoding layer and an LSTM network decoding layer, which respectively contain a plurality of sequentially connected LSTM network units, each LSTM network unit containing four gate structures, each gate structure being an input gate, an output gate, a forget gate and a cell gate. Finally, the data in the target memory (GPU 8 in the figure) in the LSTM network decoding layer is processed using a softmax function to obtain the final output target translation speech sequence (y1, y2, etc.).

[0123] S320, according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted, verifying whether the chip adaptation condition of the target model is met.

[0124] S330, if yes, load the input vector set and the hidden state vector data range matched with each recurrent neural network respectively.

[0125] S340, carry the recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip by at least one level of memory transfer.

[0126] S350, call each computing instruction in the machine learning chip according to the data in the target memory by each computing unit.

[0127] S360, calculate the input vector quantization scale and the hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network.

[0128] S370, calculate the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure respectively according to the target precision, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the recurrent weight matrix of each gate structure.

[0129] S380, output the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure respectively to the engine file to generate the adapted target model.

[0130] S390, load the source language sequence and the adapted language translation model in the engine file.

[0131] The source language sequence can refer to the language sequence to be translated input into the adapted language translation model. The data precision of the source language sequence matches the data precision of each model parameter in the language translation model before adaptation, for example, the sequence can be in the fp32 data format.

[0132] S3100, carry the source language sequence and the adapted language translation model to the target memory by at least one level of memory transfer.

[0133] S3110, call each computing instruction in the machine learning chip according to the data in the target memory by each computing unit.

[0134] S3120, convert the source language sequence into a quantized input sequence according to the target precision.

[0135] The quantized input sequence can refer to an input sequence that meets the data precision requirement of the adapted language translation model. For example, in an embodiment of the present application, the data format of the quantized input sequence can be int8. Specifically, the quantized input sequence can be obtained by dividing the source language sequence by the input vector quantization scale.

[0136] It should be noted that if the adapted language translation model is an independent and complete calculation network required to implement a calculation process, the source language sequence is the original data sequence provided by the user. If the adapted language translation model is a part (intermediate network) of a large calculation network required to implement a calculation process, the source language sequence is the output of the previous network (or calculation node) adjacent to the adapted language translation model in the large calculation network.

[0137] S3130, inputting the quantized input sequence into the adapted language translation model.

[0138] S3140, through each LSTM network unit in the adapted language translation model, iteratively calculating the calculation result output by each LSTM network unit according to the quantized sequence value at the matching time in the quantized input sequence, the hidden state quantization value at the previous time of the matching time, and the quantization parameter of each gate structure in the LSTM network unit, to finally output the target translation speech sequence.

[0139] The quantized sequence value at the matching time can refer to the quantized input sequence value corresponding to the current time in the quantized input sequence. The hidden state quantization value at the previous time of the matching time can refer to the quantization value of the hidden state generated at the previous time of the current time. The quantization parameter of each gate structure can refer to the target quantization scale corresponding to each gate structure. The target translation speech sequence can refer to the sequence generated after the source language sequence is translated by the adapted language translation model.

[0140] As shown in Figure 3c The schematic diagram of the iterative calculation of the LSTM network unit is shown. In an optional embodiment, through each LSTM network unit in the adapted language translation model, iteratively calculating the calculation result output by each LSTM network unit according to the quantized sequence value at the matching time in the quantized input sequence, the hidden state quantization value at the previous time of the matching time, and the quantization parameter of each gate structure in the LSTM network unit, can include:

[0141] The target LSTM network unit obtains an input target quantization sequence value and a target hidden state value, quantizes the target hidden state value through the target LSTM network unit, and obtains a target hidden state quantization value; the target LSTM network unit calculates a target quantization calculation result vector corresponding to each gate structure according to an input weight quantization matrix, a recurrent weight quantization matrix, the target quantization sequence value, and the target hidden state quantization value; the target LSTM network unit converts the target quantization calculation result vector corresponding to each gate structure into a target calculation result vector, and the numerical precision of the target calculation result vector matches a recurrent neural network before quantization of a post-training quantization neural network; the target LSTM network unit multiplies the target calculation result vector corresponding to each gate structure by a quantization scale in a quantization parameter of each gate structure, and adds the target calculation result vector corresponding to each gate structure to a gate offset of each gate structure to obtain a gate calculation result vector corresponding to each gate structure; the target LSTM network unit processes each gate calculation result vector using an internal activation function, and outputs a calculation result; wherein the numerical precision of the gate offset of each gate structure and the activation function matches the language translation model to be adapted.

[0142] The target LSTM network unit can refer to a network unit currently processed in the adapted language translation model. The target quantization sequence value can refer to a quantization sequence value at a matching time; the target hidden state value can refer to a hidden state value at a previous time of the matching time; the target hidden state quantization value can refer to a value obtained by quantizing the target hidden state value. For example, at the current time t, the target hidden state quantization value h The target hidden state value h t-1_int8 is calculated according to the formula t_1 The target hidden state value h t-1_int8 The target quantization calculation result vector h int8 T The target quantization calculation result vector h t_int8 The target quantization calculation result vector h int8 The target quantization calculation result vector h T The target quantization calculation result vector h int8 The target quantization calculation result vector h int8 The target quantization calculation result vector h

[0143] The gate offset can refer to an offset of each gate structure. Specifically, the gate offset can include an input weight quantization matrix offset and a recurrent weight quantization matrix offset. The calculation result output by each network unit can refer to a result generated in the calculation process. For example, it can be a hidden state value h corresponding to a matching time and a corresponding cell state C. For example, Figure 3dAs shown, an inference process diagram in an adapted language translation model provided by an embodiment of the present application is shown. Specifically, still taking the current time t as an example, the input target quantization sequence value X t_int8 and the hidden state value (target hidden state value) h t_1 of the previous time t-1 are obtained The target hidden state value h t_1 is quantized to obtain the target hidden state quantization value h t-1_int8 It is worth noting that since the initial target hidden state value is 0, h t_1 is 0 after quantization; further, h t-1_int8 *R int8 T and X t_int8 *W int8 T are calculated respectively, and then summed to obtain a target quantization calculation result vector with data format int32; then, the target quantization calculation result vector is converted by a converter to match the data format fp32 of the recurrent neural network; further, the converted result is multiplied by the quantization scale in the quantization parameter [s wi , s wo , s wf , s wc ] corresponding to each gate structure, and then added to the gate offset (b i ) of each gate structure bias i =Wb i +Rb i (taking the input gate i as an example), since the numerical accuracy of b i matches the recurrent neural network, the data format is also fp32, and finally the calculation is performed through the activation function with numerical accuracy matching the recurrent neural network, so that the hidden state value h t corresponding to the current time t and the cell state Ct can be obtained as the calculation result of the network unit output matching the current time t.

[0144] Correspondingly, as shown in Figure 3e , each gate structure of each grid unit is calculated in int8 accuracy in the matrix multiplication stage (Matmul-D), and after converting the calculation result in int32 accuracy obtained by matrix multiplication to fp32, all subsequent calculations are performed in fp32 accuracy.

[0145] The technical scheme of the embodiment of the application comprises the following steps: loading a target model comprising at least one recurrent neural network, verifying whether a chip adaptation condition for the target model is met according to a numerical relationship between target accuracy of a target computing unit in a machine learning chip and parameter accuracy of the target model to be adapted, if yes, loading an input vector set and a hidden state vector data range respectively matched with each recurrent neural network, carrying the recurrent neural network, the input vector set and the hidden state vector data range to a target memory close to the computing unit in the machine learning chip through at least one level of memory carrying, then calling each computing instruction built in the machine learning chip according to data in the target memory through each computing unit to calculate an input vector quantization scale and a hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network, further calculating an input weight quantization matrix, a recurrent weight quantization matrix and a quantization scale corresponding to each gate structure respectively according to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale and an input weight matrix and a recurrent weight matrix of each gate structure, and outputting the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure respectively to an engine file to generate an operation of the adapted target model, further loading a source language sequence and the adapted language translation model in the engine file, and carrying the source language sequence and the adapted language translation model to the target memory through at least one level of memory carrying, calling each computing instruction built in the machine learning chip according to data in the target memory through each computing unit to convert the source language sequence into a quantized input sequence according to the target accuracy, inputting the input data sequence converted into the quantized input sequence into the adapted language translation model, and iteratively calculating a calculation result output by each LSTM network unit in the adapted language translation model according to a quantized sequence value at a matching time in the quantized input sequence, a hidden state quantized value at a previous time of the matching time and a quantization parameter of each gate structure in the LSTM network unit, so as to finally output a target translation voice sequence, which can provide a calculation result in a quantization process for a staff and provide an effective data basis for a subsequent work process.

[0146] As Figure 3f shown, a flowchart of an inference method in an adapted language translation model provided by the embodiment of the application, specifically, X t_int8 is multiplied by W int8 , and int32 data format data is output; a target hidden state value at t-1 time is obtained, a target hidden state quantized value h t-1_int8 is obtained through quantization; h t - 1_int8 *R int8 is calculated, and Xt_int8 *W int8 Summing up, the target quantization calculation result vector is obtained; further, the target quantization calculation result vector is converted from int32 to the target calculation result vector of the fp32 type; then, the target calculation result vector is multiplied by the quantization parameter in sequence, so that each gate structure is multiplied by the corresponding quantization parameter [s wi , s wo , s wf , s wc ]; further, the corresponding addition of the gate offset composed of W_bais and R_bais is added; then, the gate calculation result vector corresponding to each gate structure is obtained, the subsequent activation function calculation and other calculations are performed under the fp32 precision, and the hidden state value ht and the cell state Ct at the t time are output, until all the sequences are calculated, the hidden state sequence, the final hidden state and the final cell state are output. Therefore, by converting the floating point operation into the int8 operation, the calculation result in the quantization process can be provided for the staff, the calculation efficiency is improved, and an effective data basis is provided for the subsequent work process.

[0147] In particular, the entire parameter quantization and inference method can be implemented by hardware, and by configuring the matching quantized precision 2D calculation capability and 1D calculation capability, special 1D calculation instructions such as round, clip and the like are supported, while the inference capability is greatly improved, the storage space occupation is greatly reduced.

[0148] Embodiment four

[0149] Figure 4 A structural schematic diagram of a deep learning model chip adaptation device provided by the fourth embodiment of the present application is shown in the figure. The device can execute the deep learning model chip adaptation method involved in each of the above embodiments. The device can be realized in the form of software and / or hardware, as shown in the figure, the deep learning model chip adaptation device specifically comprises: a target model loading module 410, an adaptation condition verification module 420, a data loading module 430, a data carrying module 440, an instruction calling module 450, a quantization scale calculation module 460, a quantization parameter calculation module 470, and a target model adaptation module 480. Figure 4

[0150] The adaptation condition verification module 410 is configured to load a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, the gate structure having a matching input weight matrix and a recurrent weight matrix;

[0151] ​The adaptation condition verification module 420 is configured to verify whether a chip adaptation condition for the target model is met according to a numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted.

[0152] The data loading module 430 is configured to load the input vector set and the hidden state vector data range matched with each recurrent neural network, if yes.

[0153] The data carrying module 440 is configured to carry each recurrent neural network, the input vector set and the hidden state vector data range to a target memory close to the computing unit in the machine learning chip by at least one level of memory carrying.

[0154] The instruction calling module 450 is configured to call each computing instruction built in the machine learning chip according to the data in the target memory by each computing unit, and perform the following operations:

[0155] The quantization scale calculation module 460 is configured to calculate the input vector quantization scale and the hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network.

[0156] The quantization parameter calculation module 470 is configured to calculate the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure respectively according to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the cycle weight matrix of each gate structure.

[0157] The target model adaptation module 480 is configured to output the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure respectively to the engine file to generate the adapted target model.

[0158] The application loads a target model including at least one recurrent neural network, verifies whether the chip adaptation condition of the target model is met according to the numerical relationship between the target accuracy of the target computing unit in the machine learning chip and the parameter accuracy of the target model to be adapted, if yes, loads the input vector set and the hidden state vector data range respectively matched with each recurrent neural network, carries the recurrent neural network, the input vector set and the hidden state vector data range to the target memory close to the computing unit in the machine learning chip through at least one memory carrying mode, then calls each computing instruction in the machine learning chip according to the data in the target memory, executes the input vector quantization scale and the hidden state vector quantization scale calculated according to the input vector set and the hidden state vector data range matched with the recurrent neural network, calculates the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure respectively according to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the cycle weight matrix of each gate structure, and outputs the input weight quantization matrix, the cycle weight quantization matrix and the quantization scale corresponding to each gate structure to the engine file to generate the operation of the adapted target model, solve the problem that the parameter quantization method process of the recurrent neural network in the prior art is time-consuming and has high computational complexity, brings huge computing power cost and time cost, realize the great reduction of quantization calculation complexity and calculation time consumption with small accuracy loss, and improve the efficiency of parameter quantization in the recurrent neural network.

[0159] In particular, the chip adaptation method of the whole deep learning model can be realized by hardware, the matched quantized accuracy 2D computing capability and 1D computing capability are configured, special 1D computing instructions such as round, clip and the like are supported, the inference capability is greatly improved, and the storage space occupation is greatly reduced.

[0160] Optionally, the quantization scale calculation module 460 can be specifically used for calculating the input vector numerical range corresponding to the input vector set according to the value range distribution of the input vector set, determining the input vector quantization threshold according to the input vector numerical range, calculating the input vector quantization scale according to the input vector quantization threshold, the preset quantization range and the quantization mode, determining the hidden state vector quantization threshold according to the hidden state vector data range, and calculating the hidden state vector quantization scale according to the hidden state vector quantization threshold, the preset quantization range and the quantization mode.

[0161] Optionally, the quantization scale calculation module 460 can be specifically configured to calculate the input vector quantization scale, the hidden state vector quantization scale, the input weight quantization matrix corresponding to each gate structure, and the recurrent weight quantization matrix using special calculation instructions encapsulated by calculation logic constructed using any one of the maximum and minimum value, relative entropy divergence, and percentile.

[0162] Optionally, the quantization parameter calculation module 470 can specifically include a data acquisition unit, a splicing calculation unit, a quantization scale calculation unit, and a target matrix acquisition unit.

[0163] The data acquisition unit is configured to acquire a target input weight matrix and a target recurrent weight matrix corresponding to a target gate structure currently being processed.

[0164] The splicing calculation unit is configured to splice a first result obtained by multiplying the target input weight matrix by the input vector quantization scale and a second result obtained by multiplying the target recurrent weight matrix by the hidden state vector quantization scale to obtain a target splicing matrix.

[0165] The quantization scale calculation unit is configured to calculate a target quantization scale corresponding to the target gate structure according to the target splicing matrix.

[0166] The target matrix acquisition unit is configured to quantize the target splicing matrix according to the target precision and the target quantization scale corresponding to the target gate structure to obtain a target input weight quantization matrix and a target recurrent weight quantization matrix corresponding to the target gate structure.

[0167] Optionally, the quantization scale calculation unit can be specifically configured to obtain a maximum value of matrix elements in the target splicing matrix after performing absolute value processing on the matrix elements in the target splicing matrix, and calculate the target quantization scale corresponding to the target gate structure according to the maximum value of the matrix elements, a preset quantization range, and a quantization manner.

[0168] Optionally, the target matrix acquisition unit can be specifically configured to divide the target splicing matrix by the target quantization scale to obtain an intermediate result matrix, perform rounding and truncation processing on the intermediate result matrix according to the target precision to obtain a target quantization splicing matrix, and perform de-splicing processing on the target quantization splicing matrix to obtain the target input weight quantization matrix and the target recurrent weight quantization matrix.

[0169] Optionally, the target model is a language translation model, and the language translation model specifically includes:

[0170] The long short-term memory network LSTM encoding layer and the LSTM decoding layer each include a plurality of LSTM network units connected in sequence, each of the LSTM network units including four gate structures, and each of the gate structures being an input gate, an output gate, a forget gate, and a cell gate.

[0171] Optionally, the chip adaptation device of the deep learning model can further include a model running module configured to load a source language sequence and the adapted language translation model in the engine file after outputting the input weight quantization matrix, the recurrent weight quantization matrix, and the quantization scale corresponding to each gate structure to the engine file to generate the adapted target model; carry the source language sequence and the adapted language translation model in the target memory through at least one level of memory carrying; and call each computing instruction built in the machine learning chip according to the data in the target memory through each computing unit to perform the following operations: convert the source language sequence into a quantized input sequence according to the target precision; input the quantized input sequence into the adapted language translation model; and iteratively calculate the calculation result output by each LSTM network unit through each LSTM network unit in the adapted language translation model according to the quantized sequence value at the matching time in the quantized input sequence, the hidden state quantized value at the previous time of the matching time, and the quantization parameters of each gate structure in the LSTM network unit to finally output a target translation speech sequence.

[0172] Optionally, the model running module can be further configured to: obtain a target quantized sequence value and a target hidden state value through the target LSTM network unit; perform quantization processing on the target hidden state value through the target LSTM network unit to obtain a target hidden state quantized value; calculate a target quantized calculation result vector corresponding to each gate structure according to the input weight quantization matrix, the recurrent weight quantization matrix, the target quantized sequence value, and the target hidden state quantized value of each gate structure through the target LSTM network unit; convert the target quantized calculation result vector corresponding to each gate structure into a target calculation result vector through the target LSTM network unit, the numerical precision of the target calculation result vector matching the recurrent neural network before quantization of the post-training quantized neural network; multiply the target calculation result vector corresponding to each gate structure by the quantization scale in the quantization parameters of each gate structure and then add the gate offset of each gate structure to obtain a gate calculation result vector corresponding to each gate structure; and process each gate calculation result vector through the target LSTM network unit using an internal activation function and output a calculation result; wherein the numerical precision of the gate offset of each gate structure and the activation function matches the language translation model to be adapted.

[0173] The chip adaptation apparatus for a deep learning model provided in the embodiments of the present application can execute the chip adaptation method for a deep learning model provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0174] Embodiment five

[0175] Figure 5 A structural schematic diagram of a machine learning chip provided in embodiment five of the present application is shown in the figure, which includes a processor 510, a memory 520, an input device 530 and an output device 540; the number of the processor 510 in the machine learning chip can be one or more, Figure 5 one processor 510 is taken as an example; the processor 510, the memory 520, the input device 530 and the output device 540 in the machine learning chip can be connected through a bus or other means, Figure 5 and a connection through a bus is taken as an example. Figure 5

[0176] The memory 520, as a kind of computer readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules (for example, the target model loading module 410, the adaptation condition verification module 420, the data loading module 430, the data carrying module 440, the instruction calling module 450, the quantization scale calculation module 460, the quantization parameter calculation module 470, the target model adaptation module 480 in the chip adaptation apparatus for a deep learning model) corresponding to the chip adaptation method for a deep learning model in the embodiments of the present application. The processor 510 executes various functional applications and data processing of the machine learning chip by running the software programs, instructions and modules stored in the memory 520, that is, implements the chip adaptation method for a deep learning model described above.

[0177] The chip adaptation method for a deep learning model includes:

[0178] loading a target model to be adapted, the target model including at least one recurrent neural network, the recurrent neural network including at least one network unit, the network unit including at least one gate structure, the gate structure having an input weight matrix and a recurrent weight matrix;

[0179] verifying whether a chip adaptation condition for the target model is met according to a numerical relationship between a target precision of a target computing power calculation unit in the machine learning chip and a parameter precision of the target model to be adapted;

[0180] if yes, loading an input vector set and a hidden state vector data range respectively matched with each recurrent neural network;

[0181] ​The recurrent neural network, the input vector set and the hidden state vector data range are carried to the target memory close to the computing unit in the machine learning chip by at least one level of memory transfer;

[0182] According to the data in the target memory, each computing unit calls each computing instruction built in the machine learning chip to perform the following operations:

[0183] According to the input vector set and the hidden state vector data range matched with the recurrent neural network, the input vector quantization scale and the hidden state vector quantization scale are calculated;

[0184] According to the target accuracy, the input vector quantization scale, the hidden state vector quantization scale, and the input weight matrix and the recurrent weight matrix of each gate structure, the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated respectively.

[0185] The input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the adapted target model.

[0186] The memory 520 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the memory 520 can further include a memory remotely arranged with respect to the processor 510, which can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0187] The input device 530 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the machine learning chip. The output device 540 can include a display device such as a display screen.

[0188] Embodiment six

[0189] The embodiment six of the present application also provides a storage medium containing computer executable instructions, which when executed by a computer processor, are used to perform a chip adaptation method of a deep learning model;

[0190] The chip adaptation method of the deep learning model comprises:

[0191] loading a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, the gate structure having an input weight matrix and a recurrent weight matrix;

[0192] According to the numerical relationship between the target precision of the target computing unit in the machine learning chip and the parameter precision of the target model to be adapted, it is verified whether the chip adaptation condition of the target model is met;

[0193] If yes, load the input vector set and the hidden state vector data range respectively matched with each recurrent neural network;

[0194] Each of the recurrent neural networks, the input vector set and the hidden state vector data range are carried to the target memory close to the computing unit in the machine learning chip through at least one level of memory carrying;

[0195] According to the data in the target memory, each computing instruction in the machine learning chip is called by each computing unit to perform the following operations:

[0196] According to the input vector set and the hidden state vector data range matched with the recurrent neural network, the input vector quantization scale and the hidden state vector quantization scale are calculated;

[0197] According to the target precision, the input vector quantization scale, the hidden state vector quantization scale, and the input weight matrix and the recurrent weight matrix of each gate structure, the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are calculated respectively;

[0198] The input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure are output to the engine file to generate the adapted target model.

[0199] Of course, the storage medium provided by the embodiment of the application contains computer executable instructions, which are not limited to the method operations described above, and can also perform related operations in the chip adaptation method of the deep learning model provided by any embodiment of the application.

[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art that makes a contribution. The computer software product can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH memory, a hard disk, or an optical disk, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0201] It is worth noting that in the embodiments of the chip adaptation device of the deep learning model described above, each unit and module included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not limit the protection scope of the present application.

[0202] Note that the above are only preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A chip adaptation method for a deep learning model, executed by a machine learning chip, characterized in that, The method comprises the following steps: loading a target model to be adapted, the target model comprising at least one recurrent neural network, the recurrent neural network comprising at least one network unit, the network unit comprising at least one gate structure, the gate structure having an input weight matrix and a recurrent weight matrix; verifying whether a chip adaptation condition for the target model is met according to a numerical relationship between a target precision of a computing unit in a machine learning chip and a parameter precision of the target model to be adapted; if yes, loading an input vector set and a hidden state vector data range respectively matched with each recurrent neural network; carrying the recurrent neural network, the input vector set and the hidden state vector data range to a target memory close to the computing unit in the machine learning chip through at least one level of memory carrying; calling each computing instruction in the machine learning chip according to the data in the target memory through each computing unit to perform the following operations: calculating an input vector quantization scale and a hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network; calculating an input weight quantization matrix, a recurrent weight quantization matrix and a quantization scale corresponding to each gate structure respectively according to the target precision, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the recurrent weight matrix of each gate structure; outputting the input weight quantization matrix, the recurrent weight quantization matrix and the quantization scale corresponding to each gate structure respectively to an engine file to generate an adapted target model.

2. The method of claim 1, wherein, calculating an input vector quantization scale and a hidden state vector quantization scale according to the input vector set and the hidden state vector data range matched with the recurrent neural network, comprising: calculating an input vector numerical range corresponding to the input vector set according to the value range distribution of the input vector set, and determining an input vector quantization threshold according to the input vector numerical range; calculating an input vector quantization scale according to the input vector quantization threshold, a preset quantization range and a quantization mode; determining a hidden state vector quantization threshold according to the hidden state vector data range, and calculating a hidden state vector quantization scale according to the hidden state vector quantization threshold, a preset quantization range and a quantization mode.

3. The method of claim 2, wherein: a dedicated computing instruction is used to calculate the input vector quantization scale, the hidden state vector quantization scale, the input weight quantization matrix and the recurrent weight quantization matrix corresponding to each gate structure respectively; wherein the dedicated computing instruction is encapsulated by using a calculation logic constructed by any one of a maximum minimum value, a relative entropy divergence and a percentile.

4. The method according to claim 1 or 2, characterized in that, calculating an input weight quantization matrix, a recurrent weight quantization matrix and a quantization scale corresponding to each gate structure respectively according to the target precision, the input vector quantization scale, the hidden state vector quantization scale and the input weight matrix and the recurrent weight matrix of each gate structure, comprising: obtaining a target input weight matrix and a target recurrent weight matrix corresponding to a target gate structure currently processed; The first result obtained by multiplying the target input weight matrix by the input vector quantization scale and the second result obtained by multiplying the target recurrent weight matrix by the hidden state vector quantization scale are spliced to obtain a target splicing matrix; According to the target splicing matrix, a target quantization scale corresponding to the target gate structure is calculated; According to the target precision, the target quantization scale corresponding to the target gate structure, the target splicing matrix is quantized to obtain a target input weight quantization matrix and a target recurrent weight quantization matrix corresponding to the target gate structure.

5. The method of claim 4, wherein, According to the target splicing matrix, a target quantization scale corresponding to the target gate structure is calculated, comprising: After the absolute value of each matrix element in the target splicing matrix is processed, the maximum value of the matrix elements in the target splicing matrix is obtained; According to the maximum value of the matrix elements, the preset quantization range and the quantization mode, the target quantization scale corresponding to the target gate structure is calculated.

6. The method of claim 4, wherein, According to the target precision, the target quantization scale corresponding to the target gate structure, the target splicing matrix is quantized to obtain a target input weight quantization matrix and a target recurrent weight quantization matrix corresponding to the target gate structure, comprising: Divide the target splicing matrix by the target quantization scale to obtain an intermediate result matrix; According to the target precision, the intermediate result matrix is rounded and truncated to obtain a target quantization splicing matrix; The target input weight quantization matrix and the target recurrent weight quantization matrix are obtained by disassembling the target quantization splicing matrix.

7. The method according to any one of claims 1 to 6, characterized in that, The target model is a language translation model, and the language translation model specifically comprises: A long short-term memory network (LSTM) encoding layer and an LSTM decoding layer, the LSTM encoding layer and the LSTM decoding layer each comprising a plurality of LSTM network units connected in sequence, each LSTM network unit comprising four gate structures, and each gate structure being an input gate, an output gate, a forget gate, and a cell gate.

8. The method of claim 7, wherein, After outputting the input weight quantization matrix, the recurrent weight quantization matrix, and the quantization scale corresponding to each gate structure to the engine file to generate an adapted target model, the method further comprises: Loading a source language sequence and the adapted language translation model in the engine file; Transferring the source language sequence and the adapted language translation model to the target memory through at least one level of memory transfer; Through the data in the target memory, each computing unit invokes each computing instruction built in the machine learning chip to perform the following operations: According to the target precision, the source language sequence is converted into a quantized input sequence; Inputting the quantized input sequence into the adapted language translation model; The calculation result output by each LSTM network unit in the adapted language translation model is obtained by iterative calculation according to the quantized sequence value at the matching moment in the quantized input sequence, the hidden state quantized value at the previous moment of the matching moment, and the quantized parameters of each gate structure in the LSTM network unit.

9. The method of claim 8, wherein, The calculation result output by each LSTM network unit in the adapted language translation model is obtained by iterative calculation according to the quantized sequence value at the matching moment in the quantized input sequence, the hidden state quantized value at the previous moment of the matching moment, and the quantized parameters of each gate structure in the LSTM network unit, including: The target LSTM network unit obtains the target quantized sequence value and the target hidden state value input by the target LSTM network unit; The target LSTM network unit quantizes the target hidden state value to obtain a target hidden state quantized value; The target LSTM network unit calculates a target quantized calculation result vector corresponding to each gate structure according to the input weight quantized matrix, the recurrent weight quantized matrix, the target quantized sequence value and the target hidden state quantized value of each gate structure; The target LSTM network unit converts the target quantized calculation result vector corresponding to each gate structure into a target calculation result vector, and the numerical precision of the target calculation result vector matches the recurrent neural network before quantization of the post-training quantized neural network; The target LSTM network unit multiplies the target calculation result vector corresponding to each gate structure by the quantization scale in the quantized parameter of each gate structure, and adds the gate offset of each gate structure to obtain a gate calculation result vector corresponding to each gate structure; The target LSTM network unit processes each gate calculation result vector using an internal activation function and outputs a calculation result. The numerical precision of the gate offset of each gate structure and the activation function matches the language translation model to be adapted.

10. A chip adaptation apparatus of a deep learning model, characterized by, The target model loading module loads the target model to be adapted, and the target model includes at least one recurrent neural network, the recurrent neural network includes at least one network unit, the network unit includes at least one gate structure, and the gate structure has a matched input weight matrix and a recurrent weight matrix; The adaptation condition verification module verifies whether the chip adaptation condition of the target model is met according to the numerical relationship between the target precision of the target computing unit in the machine learning chip and the parameter precision of the target model to be adapted; The data loading module loads the input vector set and the hidden state vector data range matched with each recurrent neural network if the chip adaptation condition is met. The data loading module loads the input vector set and the hidden state vector data range matched with each recurrent neural network if the chip adaptation condition is met. The data loading module loads the input vector set and the hidden state vector data range matched with each recurrent neural network if the chip adaptation condition is met. The instruction calling module is configured to call each computing instruction built in the machine learning chip according to data in the target memory through each computing unit, and perform the following operations: The quantization scale calculation module is configured to calculate input vector quantization scales and hidden state vector quantization scales according to the input vector set matched with the recurrent neural network and the hidden state vector data range. The quantization parameter calculation module is configured to calculate input weight quantization matrices, recurrent weight quantization matrices and quantization scales corresponding to each gate structure respectively according to the target accuracy, the input vector quantization scales, the hidden state vector quantization scales and the input weight matrices and the recurrent weight matrices of each gate structure. The target model adaptation module is configured to output the input weight quantization matrices, the recurrent weight quantization matrices and the quantization scales corresponding to each gate structure respectively to the engine file to generate an adapted target model.

11. A machine learning chip comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the program to implement the chip adaptation method of the deep learning model in any one of claims 1-9.

12. A storage medium of computer executable instructions, on which a computer program is stored, characterized in that, The program is executed by the processor to implement the chip adaptation method of the deep learning model in any one of claims 1-9.

Images