Data calculation method, device, storage medium and equipment

Abstract

The present application relates to the field of data calculation and provides a data calculation method, apparatus, storage medium and device. The method comprises: obtaining a target activation value and a target weight of a floating-point type to be involved in a target matrix multiplication operation; quantizing the target activation value and the target weight respectively to obtain a fixed-point type quantized activation value corresponding to the target activation value and a fixed-point type quantized weight corresponding to the target weight; performing the target matrix multiplication operation using the quantized activation value and the quantized weight; dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain a floating-point type target output data. By quantizing a limited number of target activation values and target weights that meet the conditions, the present application reduces the resources and time consumed during calculation, saves video memory, and does not reduce the calculation accuracy.

Description

Technical Field

[0001] The present application relates to the field of data computing, and more specifically to a data computing method, apparatus, storage medium, and device. Background Art

[0002] In some data computation scenarios, data is typically input into a model corresponding to the scenario, and the corresponding results are obtained through calculations at each layer of the model. The activation values involved in the calculations at each layer of the model are typically floating-point data.

[0003] For example, when generating images or text using a generative model, the target text is obtained by inputting the generation target and latent variables into the model. The activation values involved in the calculations at each layer in the model are typically corresponding 32-bit floating-point data. After the model performs operations, the target text is obtained. The activation values can refer to either the input or output data of each layer in the model.

[0004] Existing data calculation methods often quantize all floating-point data involved in the calculation to fixed-point data to speed up the calculation and improve efficiency. However, quantizing floating-point data to fixed-point data before performing the calculation sacrifices accuracy. If all floating-point data is quantized to fixed-point data before the calculation is performed, the accuracy of the final result will be significantly deviated. Summary of the Invention

[0005] In this context, the embodiments of the present application hope to provide a data calculation method, apparatus, storage medium and device to quantize a limited number of target activation values and target weights to be involved in matrix multiplication operations, that is, to convert a limited number of floating-point type data into fixed-point type data, rather than converting all floating-point type data into fixed-point type data, thereby improving calculation efficiency while ensuring calculation accuracy.

[0006] In a first aspect of the present application, a data calculation method is provided, comprising:

[0007] Obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point type data within a first preset threshold range;

[0008] quantizing the target activation value and the target weight to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein both the quantized activation value and the quantized weight are fixed-point data within a second preset threshold range;

[0009] Performing a target matrix multiplication operation using the quantized activation value and the quantized weight;

[0010] The result of the target matrix multiplication operation is inversely quantized according to the quantization method of the target activation value or the target weight to obtain target output data, where the target output data is floating-point type data within a first preset threshold range.

[0011] In one embodiment of the present application, it is applied to a neural network model;

[0012] wherein the neural network model includes at least one target matrix multiplication operation;

[0013] The target matrix multiplication operation is one of the following matrix multiplications:

[0014] QKV matrix multiplication of the attention calculation layer;

[0015] Mapping matrix multiplication of the mapping layer;

[0016] The first fully connected matrix multiplication of the feedforward neural network layer;

[0017] The second fully connected matrix multiplication of the feedforward neural network layer.

[0018] In one embodiment of the present application, in the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight, keyword weight and value weight of the attention calculation layer;

[0019] The target weight is quantified, including:

[0020] Quantizing the query weight, the keyword weight, and the value weight according to their respective channel dimensions to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight;

[0021] The performing the target matrix multiplication operation using the quantized activation value and the quantized weight includes:

[0022] Performing matrix multiplication operations on the quantized activation value, the quantized query weight, the quantized keyword weight, and the quantized value weight respectively;

[0023] Dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, including:

[0024] Dequantize the results of the three matrix multiplication operations according to the quantization method of the target activation value or the target weight;

[0025] The three inverse quantized matrix multiplication results are used to calculate the attention according to the preset rules as the output data of the attention calculation layer.

[0026] In one embodiment of the present application, when the attention calculation layer is a masked multi-head attention calculation layer, the channel dimension of the weight is the attention head dimension of the weight.

[0027] In one embodiment of the present application, quantization processing or inverse quantization processing is performed by a preset fusion operator;

[0028] The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator;

[0029] The quantization fusion operator is used to fuse the data calculation before the target activation value quantization processing and the quantization processing of the target activation value;

[0030] The dequantization fusion operator is used to perform a fusion calculation on the dequantized data and the dequantization processing.

[0031] In one embodiment of the present application, each target matrix multiplication corresponds to a quantization fusion operator and / or an inverse quantization fusion operator;

[0032] Among them, the quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the inverse quantization processing of the calculation result of the QKV matrix multiplication and the addition of the bias term;

[0033] The quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the permutation conversion process and the quantization process of the activation value after the permutation conversion process; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the inverse quantization process of the operation result of the mapping matrix multiplication, the addition of the bias term, and the addition of the residual;

[0034] The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the inverse quantization processing of the operation result of the first fully connected matrix multiplication, the addition of the bias term, and the activation operation;

[0035] The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to perform fusion calculation on the inverse quantization processing of the operation result of the second fully connected matrix multiplication, as well as the addition and normalization of the bias terms.

[0036] In one embodiment of the present application, if the neural network model is in a parallel training state, the target activation value and the target weight are quantized respectively according to the parallel training mode of the neural network model;

[0037] When the parallel training mode of the neural network model is data parallel, the target weights that meet the conditions are quantized in the channel dimension, and the target activation values and the target weights that do not meet the conditions are quantized in the tensor dimension;

[0038] When the parallel training mode of the neural network model is model parallel, the target activation value and the target weight are respectively quantized in data block dimensions, wherein the target activation value and the target weight are quantized in different data block dimensions.

[0039] In one embodiment of the present application, the quantization processing of the target weight that meets the conditions in the channel dimension includes:

[0040] Get the attention head dimension of the target weight;

[0041] Divide the target weight into blocks according to the focus head dimension to obtain each target weight sub-block;

[0042] Each target weight sub-block is quantized separately.

[0043] In one embodiment of the present application, the quantization processing of the target activation value and the target weight in data block dimensions respectively includes:

[0044] Obtaining the scale of the model parallelism and the attention head dimension of the target weight;

[0045] The target activation value is divided into blocks according to the scale of the model parallelism to obtain target activation value sub-blocks; and the target weight is divided into blocks according to the scale of the model parallelism and the focus head dimension to obtain target weight sub-blocks;

[0046] The target activation value sub-blocks and the target weight sub-blocks are quantized respectively.

[0047] In one embodiment of the present application, partitioning the target weight according to the scale of the model parallelism and the focus head dimension includes:

[0048] The product of the attention head dimension of the target weight and the scale of the model parallelism is used as the divisor for data partitioning.

[0049] In a second aspect of the present application, a data computing device is provided, comprising:

[0050] an acquisition module configured to acquire a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point data within a first preset threshold range;

[0051] a quantization module configured to quantize the target activation value and the target weight, respectively, to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized weight are both fixed-point type data within a second preset threshold range;

[0052] a computing module configured to perform the target matrix multiplication operation using the quantized activation value and the quantized weight;

[0053] The inverse quantization module is configured to inversely quantize the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is floating-point type data within a first preset threshold range.

[0054] In a third aspect of the present application, a computer-readable storage medium is provided, which includes instructions. When the instructions are executed on a computer, the computer is caused to execute the method according to the first aspect.

[0055] In a fourth aspect of the present application, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0056] Compared with the prior art, the data calculation method, apparatus, storage medium and device according to the embodiments of the present application quantize a limited number of target activation values and target weights to be involved in the matrix multiplication operation, that is, convert a limited number of floating-point type data into fixed-point type data, instead of converting all floating-point type data into fixed-point type data, thereby reducing the resources and time consumed during calculation, saving video memory, and not reducing the calculation accuracy, thereby bringing a better experience to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an illustrative and non-limiting manner, in which:

[0058] Figure 1 Schematic diagram of application scenarios of the data calculation method in some embodiments of the present application;

[0059] Figure 2This is a flow chart of a data calculation method according to an embodiment of the present application;

[0060] Figure 3 This is a GPT model quantization calculation diagram for one embodiment of the present application;

[0061] Figure 4 This is a GPT model dequantization calculation diagram for one embodiment of the present application;

[0062] Figure 5 A schematic diagram of quantization cutting during parallel training of a model according to another embodiment of the present application;

[0063] Figure 6 This is a schematic diagram of the structure of a data computing device according to an embodiment of the present application;

[0064] Figure 7 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of a computing device according to an embodiment of the present application.

[0066] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0067] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] Those skilled in the art will appreciate that the embodiments of the present application may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0069] It should be noted that the terms used in the various embodiments or drawings of this application are as follows:

[0070] float: floating point type data;

[0071] FP16 / FP32: 16-bit / 32-bit floating-point data;

[0072] Per-tensor: each tensor;

[0073] Per-channel: each channel;

[0074] Per-block: Each data block.

[0075] Currently, data calculations for models built based on neural networks usually require calculations on 32-bit data, which puts a lot of pressure on the bandwidth of data calculations and reduces computing performance.

[0076] To this end, an embodiment of the present application provides a data calculation method, which can quantize a limited number of target activation values and target weights to be involved in the matrix multiplication operation, that is, convert a limited number of floating-point type data into fixed-point type data, without converting all floating-point type data into fixed-point type data, while having little impact on the data calculation results. This reduces the bandwidth pressure of data calculation, improves the computing power of the data calculation device, and ensures the accuracy of the data calculation results.

[0077] The data calculation method provided in the embodiments of the present application can be applied to a neural network model based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0078] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0079] In the embodiments of this application, the main artificial intelligence software technologies involved include the above-mentioned natural language processing technology and deep learning.

[0080] For example, it may involve deep learning in machine learning (ML), including various types of artificial neural networks.

[0081] First, the execution subject of the embodiment of the present application is introduced. The data calculation method provided in the present application can be executed by a data computing device. A neural network model applicable to the data calculation method can be deployed in the data computing device, and the data computing device can be a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The data computing device can be a server, and the terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0082] The data computing device may have the capability to implement automatic sentence generation and translation technology in natural language processing technology.

[0083] The data computing device can be equipped with machine learning (ML) capabilities. ML is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include technologies such as artificial neural networks.

[0084] In the embodiments of the present application, the neural network model using the above-mentioned data calculation method mainly involves the application of various artificial neural networks, such as sequence generation through the neural network model.

[0085] It should be noted that the embodiments of the present application do not limit the model type for data calculation using this method. The model can be any type of model. In one possible implementation, the model can be a recurrent neural network (RNN) model.

[0086] Next, the data calculation method provided in the embodiment of this application is introduced with the server as the execution entity and combined with actual application scenarios.

[0087] See also Figure 1 , which shows a schematic diagram of an application scenario of a data calculation method provided by an embodiment of the present application. Figure 1As shown, the application scenario includes a server 101, and the server 101 executes the data calculation method provided in the embodiment of the present application.

[0088] In an embodiment of the present application, when data calculation needs to be performed on input data, the server 101 can input the input data into a model for performing data calculation to determine output data corresponding to the input data through the model.

[0089] The application scenarios of this application include speech analysis, speech noise reduction, speech translation, text translation, text recognition, sequencing and other scenarios.

[0090] In a speech analysis scenario, the model can be a speech analysis model, the input data can be the speech data to be analyzed, and the output data corresponding to the input data can be the data after the speech analysis is completed. In a speech noise reduction scenario, the model can be a speech noise reduction model, the input data can be the speech data to be denoised, and the output data corresponding to the input data can be the speech data after the denoising is performed on the speech data to be denoised.

[0091] In the case of sentence translation, the model can be a sentence translation model. The input data can correspond to sentence data in the first language to be translated, and the output data corresponding to the input data can be the translated sentence data in the second language. In the case of sequence generation, the model can be a sequence generation model. The input data can include data including the sequence to be generated, and the output data corresponding to the input data can be the sequence obtained by performing data calculations based on the sequence to be generated. This will not be elaborated further.

[0092] After input data is fed into the model, it processes it and generates activation values that flow through each layer. The activation values can be the activation values input to a particular neural network layer or the activation values output by a particular neural network layer, and the weight values can be the inherent weight values of a particular neural network layer. During this activation value flow, target activation values and target weights can be obtained and quantized, and then the quantized target activation values and target weights can be used to perform the matrix multiplication operation.

[0093] The principles and spirit of the present application are explained in detail below with reference to several representative embodiments of the present application.

[0094] For example, see Figure 1 , the server 101 can use the input activation value of the linear superposition layer in the model as the target activation value, and obtain the target activation value as a one-dimensional sequence [-0.127, 0.126, 0.08, -0.07].

[0095] After obtaining the target activation values, server 101 can quantize the target activation values to determine the quantized activation values corresponding to each target activation value, which are recorded as quantized activation values. The quantized activation values are fixed-point data. Specifically, in this example, the target activation values are quantized into integer values.

[0096] In this embodiment, the target activation value can be quantized to the data range [-127, 127] by expanding it by a factor of 1000. Thus, the determined quantized activation value is: "-127, 126, 8, -7", resulting in a one-dimensional sequence [-127, 126, 8, -7].

[0097] Thus, the quantized activation values and quantized weights are forward calculated.

[0098] After the calculation is complete, the forward-calculated results can be dequantized based on the quantization method used for the target activations or weights to obtain the corresponding output results. This reduces the impact of the quantization of the activations and weights on the forward-calculated output results. In other words, the difference between the results obtained by calculating the activations and target weights and the results obtained by calculating the quantized activations and weights is within an acceptable range.

[0099] In this embodiment, the result of the forward calculation can be scaled 1000 times according to the quantization method to achieve inverse quantization. Figure 1 As shown in the figure, after forward calculation based on the quantized activation value and the quantized weight, the one-dimensional sequence is [102, 103, -76, 8]. The data in the sequence can be dequantized and scaled 1000 times to obtain the sequence [0.102, 0.103, -0.76, 0.08], which is recorded as the corresponding output result.

[0100] After the model completes the calculation for the input data, the output data corresponding to the input data is determined.

[0101] In this example, without affecting the data calculation results as much as possible, the target activation value and target weight are quantized into integer values with fewer bits, thereby reducing the bandwidth pressure of data calculation and improving the computing power of the data calculation device.

[0102] It should be noted that although the above examples only specifically explain how to quantize the target activation value and dequantize the output data, in some embodiments of the present application, the target weight is also quantized, and the quantized weight and quantized activation value are processed according to the original activation value and weight calculation method; for example, inputting a target activation value will perform matrix multiplication calculation with the target weight, then when the data calculation method of the present application is used to calculate the data, the target activation value and the target weight will be quantized respectively, and then the quantized activation value and the quantized weight will be matrix multiplied, and the calculation result will be used as the corresponding output data, or the calculation result will be dequantized and used as the corresponding output data.

[0103] It can be understood that the quantization method of the corresponding weight is similar to the quantization method of the corresponding activation value, that is, the corresponding weight is processed using a quantization coefficient to obtain a quantized corresponding weight.

[0104] The various technical solutions of this application are described below with reference to several specific embodiments.

[0105] The following combination Figure 1 For application scenarios, refer to Figure 2 The method for data calculation according to the exemplary embodiment of the present application is described. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0106] Next, we will use the server as the above-mentioned data computing device and the natural language generation scenario as an example to introduce the data computing method provided by the embodiment of this application. The server is deployed with the above-mentioned model, which can be a trained model, and the data computing method is the inference process for this model.

[0107] Reference Figure 2 , the data calculation method includes:

[0108] Step S110, obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point type data within a first preset threshold range;

[0109] The data calculation method provided in the embodiments of the present application quantifies each target activation value and target weight in the neural network model. The quantified target activation values and target weights can be divided into multiple groups, and a group of target activation values and target weights can include multiple target activation values and target weights.

[0110] It should be noted that a set of target activation values and target weights should be subject to the same matrix multiplication operation. That is, this embodiment determines whether the activation value is a target activation value based on whether the activation value is to be matrix multiplied with the weight. Similarly, it determines whether the weight is a target weight based on whether the weight is to be matrix multiplied with the activation value. In addition, if the activation value and the weight are to be matrix multiplied, the activation value and the weight are determined as a set of target activation value and target weight. For example, if activation value 1 is to be matrix multiplied with weight 1 by 1, then activation value 1 and weight 1 are a set of target activation value and target weight.

[0111] It is understandable that since the activation values and weights in the neural network model are inherently floating-point data, this embodiment does not determine whether the data types of the activation values and weights in the neural network model are floating-point data. That is, the activation values and weights in the neural network model are automatically assumed to be floating-point data. Those skilled in the art can further determine whether the activation values and weights involved in the matrix multiplication operation are floating-point data based on actual application scenarios, and this embodiment does not limit this.

[0112] The floating-point data is a real number used to indicate a decimal. The first preset threshold range may be a threshold range of floating-point data recognized in the prior art, for example, 1.8E-308 to 1.8E+308.

[0113] It should be noted that floating-point data can express more information, that is, the results of calculations based on floating-point data will be more accurate. However, since floating-point data will take more time and computing resources to calculate, it is necessary to quantize the floating-point data. Considering that if all floating-point data in the neural network model are quantized, the accuracy of the final output result of the model will be greatly reduced, therefore, in the embodiment, the target activation value and target weight that meet certain conditions are obtained and quantized.

[0114] The reason is that matrix multiplication of floating-point data consumes extremely high computational resources and time, resulting in a very high computational complexity. Compared to matrix multiplication, other multiplication and addition operations consume fewer computational resources. Therefore, quantizing only the weights and activations that require matrix multiplication can achieve significant resource savings. This significantly reduces the computational speed and resource consumption of the quantized model, while also minimizing the loss of accuracy.

[0115] Based on the above principles, the inventors discovered that a neural network model includes at least one matrix multiplication operation that requires the participation of weights and activation values. Therefore, in one embodiment, the target weights and target activation values can be determined based on the matrix multiplication operation. That is, the matrix multiplication operation involving activation values and weights is identified as the target matrix multiplication operation; specifically, the target matrix multiplication operation is one of the following matrix multiplications:

[0116] QKV matrix multiplication of the attention calculation layer;

[0117] Mapping matrix multiplication of the mapping layer;

[0118] The first fully connected matrix multiplication of the feedforward neural network layer;

[0119] The second fully connected matrix multiplication of the feedforward neural network layer.

[0120] In one embodiment, in the generative unsupervised pre-training model Gererate Pre-Training Model (GPT), all target matrix multiplication operations listed in the above embodiment are included.

[0121] It should be noted that the GPT model includes multiple stacked decoder-encoders (transformer-decoder), each transformer-decoder has the same structure, for example, including a masked multi-head attention layer, a mapping layer, a feedforward neural network layer, a residual connection layer and a normalization layer. Therefore, each transformer-decoder includes the various target matrix multiplication operations listed in the above embodiments.

[0122] Step S120: quantizing the target activation value and the target weight to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein both the quantized activation value and the quantized weight are fixed-point data within a second preset threshold range.

[0123] As mentioned in the application scenario section, quantizing floating-point data means converting floating-point data into fixed-point data. The conversion formula between floating-point data and fixed-point data is as follows:

[0124] x_out = (clamp(round(x / scale+zero_point),quant_min,quant_max)-zero_point)*scale, where x_out represents the fixed-point data after quantization, x represents the floating-point data before quantization, quant_min and quant_max represent the maximum and minimum values that can be represented by the fixed-point data at a specific bit width, respectively, and scale is the scaling factor (for example, 1000 times the example in the application scenario section).

[0125] Calculating the scale factor is crucial in quantization. This factor is calculated as the ratio of the tensor's statistical range to the fixed-point range. Therefore, determining the tensor's statistical range is crucial to quantization accuracy. The goal is to maximize the range while ensuring that closely matched numbers are distinguished. Common methods for calculating scale include max and percentage.

[0126] In addition, quantization methods can be divided into symmetric quantization and asymmetric quantization based on whether the zero_point is 0. Zero_point refers to the value of the floating-point number 0 mapped to the fixed-point number. If the floating-point number 0 is mapped to the fixed-point number 0, it is symmetric quantization; otherwise, it is asymmetric quantization. In deep learning, since activation values and weights generally conform to a normal distribution with a mean of 0, the embodiments of the present application use symmetric quantization to achieve the purpose of saving computation.

[0127] The fixed-point type data is data with a decimal point implicitly located at a fixed position. The second preset threshold range may be a threshold range of fixed-point type data recognized in the prior art, for example, 1.8E-308 to 1.8E+308.

[0128] In order to describe in more detail which activation values and weights need to be quantized in the neural network model, refer to Figure 3 , taking how to quantize each target activation value and target weight in the GPT model as an example to introduce, where float input is the input of floating-point type data: that is, the output activation value of the embedding layer. Since the embedding layer does not involve matrix multiplication operations of weights and activation values, even if quantized, it will not bring performance improvement, and the proportion of video memory is small. Therefore, this embodiment does not quantize the activation values and weights of the embedding layer to ensure the accuracy of the model's data calculation.

[0129] In the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight, keyword weight and value weight of the attention calculation layer;

[0130] The target weight is quantified, including:

[0131] Quantizing the query weight, the keyword weight, and the value weight according to their respective channel dimensions to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight;

[0132] The performing the target matrix multiplication operation using the quantized activation value and the quantized weight includes:

[0133] Performing matrix multiplication operations on the quantized activation value, the quantized query weight, the quantized keyword weight, and the quantized value weight respectively;

[0134] Dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, including:

[0135] Dequantize the results of the three matrix multiplication operations according to the quantization method of the target activation value or the target weight;

[0136] The three inverse quantized matrix multiplication results are used to calculate the attention according to the preset rules as the output data of the attention calculation layer.

[0137] Reference Figure 3 , query-scale corresponds to the query quantization coefficient in the transformer-decoder structure. The activation value query input to the attention calculation layer will be matrix multiplied with the query weight Q, keyword weight K, and value weight V of the attention calculation layer (i.e., QKV multiplication of the attention calculation layer, Figure 3 The Gemm shown in represents matrix multiplication). According to Figure 3 It can be seen that the input of each of the above three matrix multiplications is essentially the same activation value. Therefore, this embodiment combines the quantization processing operations of the inputs of each of the above three matrix multiplications into one, reducing the number of quantization steps and ensuring quantization accuracy.

[0138] In addition, in some embodiments, the attention calculation layer may be a masked multi-head attention calculation layer, that is, the attention calculation layer includes multiple attention heads. Therefore, in order to improve the quantization granularity and reduce quantization error, in one embodiment, if the dimension of the target weight is (head_num, size_per_head), the attention head dimension, i.e., size_per_head, is regarded as the channel dimension, and the target weight is quantized separately according to the channel dimension, that is, the target weight is first divided into each target weight sub-block according to the attention head dimension, and then quantized separately.

[0139] Since the two matrix multiplications Q*K and attention attn*v have no weights and are both multiplied by activation values, which cannot reduce the memory usage, this embodiment does not quantize the data involved in the two matrix multiplications.

[0140] Continue to refer to Figure 3 Considering that Out_scale / fc1_scale and fc2_scale respectively correspond to the scaling factor of the project matrix multiplication of the deocder structure (i.e., the mapping matrix multiplication of the mapping layer), the scaling factor of the fc1 matrix multiplication (i.e., the first fully connected matrix multiplication of the feedforward neural network layer), and the scaling factor of the fc2 matrix multiplication (i.e., the second fully connected matrix multiplication of the feedforward neural network layer), this embodiment performs normal quantization operations to save computing resources and time.

[0141] After introducing how to determine the target activation value and the target weight, and how to quantize each target activation value and target weight, step S130 is then executed to perform the target matrix multiplication operation using the quantized activation value and the quantized weight.

[0142] In order to further improve the computational efficiency, in one embodiment of the present application, quantization processing is performed by a pre-set fusion operator; the fusion operator includes a quantization fusion operator;

[0143] The quantization fusion operator is used to fuse the data calculation before the target activation value quantization processing and the quantization processing of the target activation value.

[0144] Specifically, each target matrix multiplication can correspond to a quantized fusion operator;

[0145] The quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; Figure 3 The layer normalization layernorm in the dotted box and the input queue scaling factor query-scale (equivalent to Figure 4 Quant in ).

[0146] The quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the permutation conversion process and the quantization process of the activation value after the permutation conversion process; Figure 4 The Transpose arrangement conversion process in the dotted box and the quantization process of the activation value Quant.

[0147] The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; Figure 4 The layer normalization LayerNorm processing and the quantization processing of the activation value Quant in the dotted box before the first fully connected matrix multiplication FC1 Gemm.

[0148] The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to fuse the activation processing and the quantization processing of the activation value after the activation processing; Figure 4 The second fully connected matrix in the dotted box is activated before the GELU processing of the fc2_scale and the quantization of the activation value according to fc2_scale.

[0149] After using the quantized activation value and the quantized weight to perform the target matrix multiplication operation, in order to ensure the accuracy of the final calculation result, step S140 needs to be executed, and the result of the target matrix multiplication operation is dequantized according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is floating-point type data within the first preset threshold range.

[0150] The inverse quantization process is the inverse operation of the quantization process. For example, if the quantization process amplifies the data by 1000 times, the inverse quantization process reduces the data by 1000 times.

[0151] In addition, since some target weights are quantized in the channel dimension, in some embodiments, the quantization method of the target weight is different from the target activation value. At this time, the result of the target matrix multiplication operation can be dequantized according to the quantization method of the target activation value or the target weight according to actual needs.

[0152] Similar to the quantization embodiment, in one embodiment of the present application, inverse quantization processing is also performed through a preset fusion operator;

[0153] The fusion operator is an inverse quantization fusion operator;

[0154] The dequantization fusion operator is used to perform a fusion calculation on the dequantized data and the dequantization processing.

[0155] In one embodiment of the present application, each target matrix multiplication also corresponds to an inverse quantization fusion operator;

[0156] Among them, the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used to perform a fusion calculation by adding the inverse quantization processing of the operation result of the QKV matrix multiplication and the bias term; specifically, in the QKV multiplication step, after the QKV matrix multiplication operation is completed, this embodiment combines the inverse quantization process with the bias term bias, for example Figure 4 The dequantization deQuant in is added to the query weight Q bias term Qbias.

[0157] The inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to perform a fusion calculation on the inverse quantization processing of the operation result of the mapping matrix multiplication, the addition of the bias term, and the addition of the residual; specifically, for example Figure 4 The dequantization deQuant, mapping bias term addition Proj Bias and Add input residual are added.

[0158] The inverse quantization fusion operator corresponding to the first fully connected matrix multiplication is used to perform a fusion calculation on the inverse quantization processing of the operation result of the first fully connected matrix multiplication, the addition of the bias term, and the activation operation; specifically, for example Figure 4 DeQuant, bias addition FC1 Bias&act and GELU activation operation ( Figure 3 shown).

[0159] The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to perform a fusion calculation on the inverse quantization processing of the operation result of the second fully connected matrix multiplication, the addition of the bias term, and the normalization processing; specifically, for example Figure 4 Dequantization deQuant, bias addition FC2Bias&act and normalization processing Add&Norm.

[0160] In addition, the quantization fusion operator can be obtained by merging the original kernel function calculated before the quantization processing and the quantization processing; the inverse quantization fusion operator can be obtained by merging the original kernel function calculated after the inverse quantization processing and the inverse quantization processing.

[0161] The embodiment of the present application integrates the quantization and dequantization processes into the front and back operators when performing quantized inference, thereby reducing video memory access and improving performance. Since the weights and their quantization coefficients are known, this embodiment can pre-quantize the weights.

[0162] Some neural network models, such as the GPT model, have a large number of parameters, so a single graphics card may not have enough memory to store all the models. To address this issue, you can split the weights into tensors and store the split tensors on different graphics cards, allowing you to load a larger model.

[0163] Specifically, in one embodiment, if the neural network model is in a parallel training state, the target activation value and the target weight are quantized respectively according to the parallel training mode of the neural network model;

[0164] When the parallel training mode of the neural network model is data parallel, the target weights that meet the conditions are quantized in the channel dimension, and the target activation values and the target weights that do not meet the conditions are quantized in the tensor dimension. The qualified condition may be that the target weights are weights output by multiple attention heads.

[0165] In one embodiment, the quantization processing of the target weight that meets the conditions in terms of channel dimension includes:

[0166] Get the attention head dimension of the target weight;

[0167] Divide the target weight into blocks according to the focus head dimension to obtain each target weight sub-block;

[0168] Each target weight sub-block is quantized separately.

[0169] During parallel model training, weight quantization typically involves distributing the data within a weight channel to different graphics cards. This results in an inability to obtain the statistical range of the data within the channel and, consequently, to calculate the weight scale coefficient. As for activation values, since different machines calculate different parts of the activation values, it is also impossible to determine the overall range of the activation values and, consequently, to calculate the activation scale coefficient.

[0170] In order to solve the problem that the scale of activation values and weights cannot be counted across machines due to parallel model training, the embodiments of the present application propose a per-block data block granularity quantization method.

[0171] When the parallel training mode of the neural network model is model parallel, the target activation value and the target weight are respectively quantized in data block dimensions, wherein the target activation value and the target weight are quantized in different data block dimensions.

[0172] In one embodiment, the quantizing of the target activation value and the target weight in terms of data block dimensions respectively includes:

[0173] Obtaining the scale of the model parallelism and the attention head dimension of the target weight;

[0174] The target activation value is divided into blocks according to the scale of the model parallelism to obtain target activation value sub-blocks; and the target weight is divided into blocks according to the scale of the model parallelism and the focus head dimension to obtain target weight sub-blocks;

[0175] The target activation value sub-blocks and the target weight sub-blocks are quantized respectively.

[0176] In one embodiment, partitioning the target weight according to the scale of the model parallelism and the focus head dimension includes:

[0177] The product of the attention head dimension of the target weight and the scale of the model parallelism is used as the divisor for data partitioning.

[0178] The per-block quantization granularity is finer than the per-channel quantization granularity. For activation values, they are divided into blocks according to the scale of model parallelism. Assuming that the size of model parallelism is K and the number of original activation value data is n, then the per-tensor quantization granularity is n, and the per-block quantization granularity is n / K; for weights, they are also divided into blocks according to the scale of model parallelism. Assuming that the size of model parallelism is K, the number of original weight data is n, and the number of weight attention heads is n, then the per-channel quantization granularity is n / h, and the per-block quantization granularity is n / (h*K).

[0179] Reference Figure 5 In this figure, a rectangle represents the per-tensor quantization process (a rectangle represents a tensor), the horizontal division into two represents the per-channel quantization process, and the horizontal + vertical division into four represents the per-block quantization process.

[0180] exist Figure 5 In the example, the parameter starting with quant represents the quantization of the activation value, and the parameter ending with weight represents the quantization of the weight.

[0181] The choice of the three quantization processes depends on the training method. In the case of data parallelism, this embodiment uses a per-tensor quantization method for activation values and a per-channel quantization method for weights. Therefore, in data parallelism, the word "quant" at the beginning is a whole rectangle, and the word "weight" at the end is a rectangle divided in half.

[0182] In model parallelism, attention is first processed in parallel. Due to model parallelism, the QKV weights undergo a tensor split, and this split is not performed on the channel dimension. Therefore, they are represented as two vertical lines, forming a quadrant. Quant_out, the output of the QKV matrix multiplication, is the result of calculating the attention. In the case of tensor split, it is also split vertically. Similarly, out_weight also needs to be split vertically until the attention calculation is complete (i.e., the node represented by quant_fc1).

[0183] Similar to the model parallelism of attention, the calculation process of the feedforward neural network layer ffn also follows this path. Due to the dual effects of model parallelism and per-channel quantization, FC1_weight and FC2_weight are quantized in a four-way per-block manner. The matrix multiplication result of fc1, quant_fc2, changes from per-tensor quantization to per-block quantization in a two-way vertical division. Once the calculation of ffn is complete, the output of the matrix multiplication of fc2 returns to per-tensor quantization.

[0184] For example, for a matrix [[1,2,3,4],[5,6,7,8]], the per-tensor quantization method quantizes all 8 values in the matrix together; the per-channel quantization method quantizes [1,2,3,4] and [5,6,7,8]; the per-block quantization method first splits [1,2,3,4] into two data blocks: [1,2] and [3,4], then quantizes [1,2] and [3,4], and then repeats the above operation for [5,6,7,8].

[0185] According to the data calculation method of the embodiment of the present application, a limited number of target activation values and target weights to be involved in the matrix multiplication operation are quantized, that is, a limited number of floating-point type data are converted into fixed-point type data, rather than converting all floating-point type data into fixed-point type data, so that the resources and time consumed during calculation are reduced, video memory is saved, and the calculation accuracy is not reduced, which brings a better experience to the user. In addition, in some embodiments, quantization and dequantization are integrated with the front and back operators to further improve the calculation efficiency. In addition, in some embodiments, the model in the parallel training state is quantized at the data block granularity. Since the data is quantized at a finer granularity, the quantization error is effectively reduced, and the communication overhead between machines is avoided, thereby improving the training efficiency.

[0186] After introducing the method of the exemplary embodiment of the present application, next, reference is made to Figure 6 In the device for data calculation of the exemplary embodiment of the present application, the device 60 includes:

[0187] An acquisition module 610 is configured to acquire a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point data within a first preset threshold range;

[0188] a quantization module 620 configured to quantize the target activation value and the target weight, respectively, to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized weight are both fixed-point data within a second preset threshold range;

[0189] A calculation module 630 is configured to perform the target matrix multiplication operation using the quantized activation value and the quantized weight;

[0190] The inverse quantization module 640 is configured to inverse quantize the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is floating-point type data within a first preset threshold range.

[0191] In one embodiment of the present application, the device 60 is applied to a neural network model;

[0192] wherein the neural network model includes at least one target matrix multiplication operation;

[0193] The target matrix multiplication operation is one of the following matrix multiplications:

[0194] QKV matrix multiplication of the attention calculation layer;

[0195] Mapping matrix multiplication of the mapping layer;

[0196] The first fully connected matrix multiplication of the feedforward neural network layer;

[0197] The second fully connected matrix multiplication of the feedforward neural network layer.

[0198] In one embodiment of the present application, in the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight, keyword weight and value weight of the attention calculation layer;

[0199] The quantization module 620 is further configured to perform quantization processing on the query weight, the keyword weight, and the value weight according to their respective channel dimensions, to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight;

[0200] The calculation module 630 is further configured to perform matrix multiplication operations using the quantized activation value and the quantized query weight, the quantized keyword weight, and the quantized value weight respectively;

[0201] The dequantization module 640 is further configured to perform dequantization processing on the results of the three matrix multiplication operations according to the quantization method of the target activation value or the target weight; and

[0202] The three inverse quantized matrix multiplication results are used to calculate the attention according to the preset rules as the output data of the attention calculation layer.

[0203] In one embodiment of the present application, when the attention calculation layer is a masked multi-head attention calculation layer, the channel dimension of the weight is the attention head dimension of the weight.

[0204] In one embodiment of the present application, quantization processing or inverse quantization processing is performed by a preset fusion operator;

[0205] The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator;

[0206] The quantization fusion operator is used to fuse the data calculation before the target activation value quantization processing and the quantization processing of the target activation value;

[0207] The dequantization fusion operator is used to perform a fusion calculation on the dequantized data and the dequantization processing.

[0208] In one embodiment of the present application, each target matrix multiplication corresponds to a quantization fusion operator and / or an inverse quantization fusion operator;

[0209] Among them, the quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the inverse quantization processing of the calculation result of the QKV matrix multiplication and the addition of the bias term;

[0210] The quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the permutation conversion process and the quantization process of the activation value after the permutation conversion process; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the inverse quantization process of the operation result of the mapping matrix multiplication, the addition of the bias term, and the addition of the residual;

[0211] The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the inverse quantization processing of the operation result of the first fully connected matrix multiplication, the addition of the bias term, and the activation operation;

[0212] The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to perform fusion calculation on the inverse quantization processing of the operation result of the second fully connected matrix multiplication, as well as the addition and normalization of the bias terms.

[0213] In one embodiment of the present application, if the neural network model is in a parallel training state, the target activation value and the target weight are quantized respectively according to the parallel training mode of the neural network model;

[0214] When the parallel training mode of the neural network model is data parallel, the target weights that meet the conditions are quantized in the channel dimension, and the target activation values and the target weights that do not meet the conditions are quantized in the tensor dimension;

[0215] When the parallel training mode of the neural network model is model parallel, the target activation value and the target weight are respectively quantized in data block dimensions, wherein the target activation value and the target weight are quantized in different data block dimensions.

[0216] In one embodiment of the present application, the quantization module 620 is further configured to obtain the focus head dimension of the target weight; divide the target weight into data blocks according to the focus head dimension to obtain each target weight sub-block; and quantize each target weight sub-block separately.

[0217] In one embodiment of the present application, the quantization module 620 is further configured to obtain the scale of the model parallelism and the focus head dimension of the target weight; perform data segmentation on the target activation value according to the scale of the model parallelism to obtain each target activation value sub-block; and perform data segmentation on the target weight according to the scale of the model parallelism and the focus head dimension to obtain each target weight sub-block; and perform quantization processing on each target activation value sub-block and each target weight sub-block respectively.

[0218] In one embodiment of the present application, partitioning the target weight according to the scale of the model parallelism and the focus head dimension includes:

[0219] The product of the attention head dimension of the target weight and the scale of the model parallelism is used as the divisor for data partitioning.

[0220] According to the data computing device of the embodiment of the present application, a limited number of target activation values and target weights to be involved in the matrix multiplication operation are quantized, that is, a limited number of floating-point type data are converted into fixed-point type data, rather than converting all floating-point type data into fixed-point type data, so that the resources and time consumed during calculation are reduced, video memory is saved, and the calculation accuracy is not reduced, which brings a better experience to the user. In addition, in some embodiments, quantization and dequantization are integrated with the previous and next operators to further improve the calculation efficiency. In addition, in some embodiments, the model in the parallel training state is quantized at the data block granularity. Since the data is quantized at a finer granularity, the quantization error is effectively reduced, and the communication overhead between machines is avoided, thereby improving the training efficiency.

[0221] After introducing the method and apparatus of the exemplary embodiment of the present application, Figure 7 The computer-readable storage medium of the exemplary embodiment of the present application is described. The computer-readable storage medium shown is an optical disc 70, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, the computer program will implement the steps described in the above method implementation, for example, obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point type data within a first preset threshold range; quantizing the target activation value and the target weight respectively to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized weight are both fixed-point type data within a second preset threshold range; performing the target matrix multiplication operation on the quantized activation value and the quantized weight; dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, wherein the target output data is floating-point type data within the first preset threshold range; the specific implementation method of each step is not repeated here.

[0222] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0223] After introducing the method, apparatus and storage medium of the exemplary embodiments of the present application, reference is made to Figure 8 A device for data calculation according to an exemplary embodiment of the present application.

[0224] Figure 8 A block diagram of an exemplary computing device 80 suitable for implementing embodiments of the present application is shown. The computing device 80 may be a computer system or a server. Figure 8 The computing device 80 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0225] like Figure 8 As shown, the components of computing device 80 may include, but are not limited to, one or more processors or processing units 801 , a system memory 802 , and a bus 803 connecting various system components (including system memory 802 and processing unit 801 ).

[0226] The computing device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 80, including volatile and non-volatile media, removable and non-removable media.

[0227] The system memory 802 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022. The computing device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 8023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 is not shown in the , usually referred to as "hard drive"). Although not in Figure 8As shown in FIG, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to bus 803 via one or more data medium interfaces. System memory 802 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present application.

[0228] A program / utility 8025 having a set (at least one) of program modules 8024 may be stored, for example, in system memory 802. Such program modules 8024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 8024 generally implement the functions and / or methods of the embodiments described herein.

[0229] The computing device 80 may also communicate with one or more external devices 804 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 805. Furthermore, the computing device 80 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 806. Figure 8 As shown, the network adapter 806 communicates with other modules (such as the processing unit 801, etc.) of the computing device 80 via the bus 803. Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with computing device 80 .

[0230] The processing unit 801 executes various functional applications and data calculations by running the program stored in the system memory 802, for example, obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point type data within a first preset threshold range; quantizing the target activation value and the target weight respectively to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized weight are both fixed-point type data within a second preset threshold range; performing the target matrix multiplication operation using the quantized activation value and the quantized weight; and dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, wherein the target output data is floating-point type data within the first preset threshold range. The specific implementation method of each step will not be repeated here.

[0231] It should be noted that although several units / modules or sub-units / modules of the data computing device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.

[0232] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0233] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the specific embodiments of the application, and the division of various aspects does not mean that the features of these aspects cannot be combined to achieve benefits. Such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A data calculation method, characterized in that: include: Obtaining a target activation value and a target weight, wherein the target activation value is determined based on input data, the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point type data within a first preset threshold range, and when the method is applied to a speech processing scenario, the input data includes speech data; when the method is applied to a sentence translation scenario, the input data includes sentence data; when the method is applied to a sequence generation scenario, the input data includes data of a sequence to be generated; quantizing the target activation value and the target weight to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein both the quantized activation value and the quantized weight are fixed-point data within a second preset threshold range; Performing a target matrix multiplication operation using the quantized activation value and the quantized weight; Dequantizing a result of the target matrix multiplication operation according to a quantization method of the target activation value or the target weight to obtain target output data, where the target output data is floating-point data within a first preset threshold range; The quantization processing is to perform quantization processing or inverse quantization processing through a preset fusion operator; The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator; The quantization fusion operator is used to fuse the data calculation before the target activation value quantization processing and the quantization processing of the target activation value; The dequantization fusion operator is used to perform a fusion calculation on the dequantized data and the dequantization processing.

2. The data calculation method according to claim 1, wherein: Applied to neural network models; wherein the neural network model includes at least one target matrix multiplication operation; The target matrix multiplication operation is one of the following matrix multiplications: QKV matrix multiplication of the attention calculation layer; Mapping matrix multiplication of the mapping layer; The first fully connected matrix multiplication of the feedforward neural network layer; The second fully connected matrix multiplication of the feedforward neural network layer.

3. The data calculation method according to claim 2, wherein: In the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight, keyword weight and value weight of the attention calculation layer; The target weight is quantified, including: Quantizing the query weight, the keyword weight, and the value weight according to their respective channel dimensions to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight; The performing the target matrix multiplication operation using the quantized activation value and the quantized weight includes: Performing matrix multiplication operations on the quantized activation value, the quantized query weight, the quantized keyword weight, and the quantized value weight respectively; Dequantizing the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, including: Dequantize the results of the three matrix multiplication operations according to the quantization method of the target activation value or the target weight; The three inverse quantized matrix multiplication results are used to calculate the attention according to the preset rules as the output data of the attention calculation layer.

4. The data calculation method according to claim 3, wherein: When the attention calculation layer is a masked multi-head attention calculation layer, the channel dimension of the weight is the attention head dimension of the weight.

5. The data calculation method according to claim 2, wherein: Each target matrix multiplication corresponds to a quantization fusion operator and / or an inverse quantization fusion operator; Among them, the quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used to fuse the inverse quantization processing of the calculation result of the QKV matrix multiplication and the addition of the bias term; The quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the permutation conversion process and the quantization process of the activation value after the permutation conversion process; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to fuse the inverse quantization process of the operation result of the mapping matrix multiplication, the addition of the bias term, and the addition of the residual; The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the first fully connected matrix multiplication is used to fuse the inverse quantization processing of the operation result of the first fully connected matrix multiplication, the addition of the bias term, and the activation operation; The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to perform fusion calculation on the inverse quantization processing of the operation result of the second fully connected matrix multiplication, as well as the addition and normalization of the bias terms.

6. The data calculation method according to any one of claims 2 to 5, characterized in that: If the neural network model is in a parallel training state, the target activation value and the target weight are quantized respectively according to the parallel training mode of the neural network model; When the parallel training mode of the neural network model is data parallel, the target weights that meet the conditions are quantized in the channel dimension, and the target activation values and the target weights that do not meet the conditions are quantized in the tensor dimension; When the parallel training mode of the neural network model is model parallel, the target activation value and the target weight are respectively quantized in data block dimensions, wherein the target activation value and the target weight are quantized in different data block dimensions.

7. The data calculation method according to claim 6, wherein: The quantization processing of the target weights that meet the conditions in terms of channel dimension includes: Get the attention head dimension of the target weight; Divide the target weight into blocks according to the focus head dimension to obtain each target weight sub-block; Each target weight sub-block is quantized separately.

8. The data calculation method according to claim 6, wherein: The quantization processing of the target activation value and the target weight in terms of data block dimensions respectively includes: Obtaining the scale of the model parallelism and the attention head dimension of the target weight; The target activation value is divided into blocks according to the scale of the model parallelism to obtain target activation value sub-blocks; and the target weight is divided into blocks according to the scale of the model parallelism and the focus head dimension to obtain target weight sub-blocks; The target activation value sub-blocks and the target weight sub-blocks are quantized respectively.

9. The data calculation method according to claim 8, wherein: The data segmentation of the target weight according to the scale of the model parallelism and the focus head dimension includes: The product of the attention head dimension of the target weight and the scale of the model parallelism is used as the divisor for data partitioning.

10. A data computing device, characterized in that: include: an acquisition module configured to acquire a target activation value and a target weight, wherein the target activation value is determined based on input data, the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point data within a first preset threshold range; when the method is applied to a speech processing scenario, the input data includes speech data; when the method is applied to a sentence translation scenario, the input data includes sentence data; and when the method is applied to a sequence generation scenario, the input data includes data of a sequence to be generated; a quantization module configured to quantize the target activation value and the target weight, respectively, to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized weight are both fixed-point type data within a second preset threshold range; a computing module configured to perform the target matrix multiplication operation using the quantized activation value and the quantized weight; a dequantization module configured to dequantize the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, where the target output data is floating-point type data within a first preset threshold range; The quantization processing is to perform quantization processing or inverse quantization processing through a preset fusion operator; The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator; The quantization fusion operator is used to fuse the data calculation before the target activation value quantization processing and the quantization processing of the target activation value; The dequantization fusion operator is used to perform a fusion calculation on the dequantized data and the dequantization processing.

11. A computer-readable storage medium comprising instructions, characterized in that: When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 9.

12. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images