Data processing method and related equipment

Abstract

The invention provides a data processing method and related equipment, relates to the field of artificial intelligence, and can reduce the overhead of a storage space in a model training process and improve the performance of model training. In the method, a first device performs forward calculation on first data through a first model to obtain second data, the first model is a neural network model, and the digit of the first data and the digit of the second data are first values; the first device quantizes the second data to obtain third data, the number of bits of the third data is a second value, and the second value is smaller than the first value; the first device stores the third data, and the stored third data is used for reverse calculation of the first model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related equipment. Background Art

[0002] With the development of artificial intelligence technology, neural network models are used in more and more fields. In order to improve the accuracy and performance of neural network models, it is necessary to train the neural network models. The training process of neural network models usually includes: forward calculation and reverse calculation. For example, the forward calculation can be the propagation process of the neural network model from the input layer to the output layer. During the forward calculation process, weighted and activated calculations are performed in each layer of the neural network to obtain the output result of the current layer and pass it to the next layer of the neural network. This process continues until the output layer of the neural network model, and finally the loss under the current parameters is obtained. For example, the reverse calculation can be based on the loss obtained by the forward calculation, and the gradient of the loss function for each parameter is calculated from the output layer to the input layer of the neural network model in turn, and the relevant parameters are updated at a given learning rate.

[0003] When training a neural network model, a chip is usually required. However, the computing units of some chips do not support low-precision formats such as 8-bit floating point numbers (8bit float, FP8), but only support high-precision formats such as 16-bit floating point numbers (16bit float, FP16) or 32-bit floating point numbers (32bit float 32, FP32). When using such chips for model training, the accuracy of the data in the model calculation process is higher, for example, the data accuracy of forward calculation and reverse calculation is higher. Compared with low-precision data, high-precision data occupies more storage space, which leads to a large storage space overhead during model training.

[0004] Therefore, how to reduce the storage space overhead of the model training process is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a data processing method and related equipment, which can save storage space occupied during model training and improve the performance of model training.

[0006] In the first aspect, the present application provides a data processing method, which is executed by a first device, or the method is executed by some components in the first device (such as a processor, a chip or a chip system, etc.), or the method can also be implemented by a logic module or software that can realize all or part of the functions of the first device. In the first aspect and its possible implementation, the data processing method is described as being executed by the first device as an example, the first device performs a forward calculation on the first data through a first model to obtain the second data, the first model is a neural network model, and the number of bits of the first data and the number of bits of the second data are first values; the first device quantizes the second data to obtain the third data, the number of bits of the third data is a second value, and the second value is less than the first value; the first device saves the third data, and the saved third data is used for the reverse calculation of the first model.

[0007] In the first aspect, the second data is obtained by forward calculation through the first model. Since the second data is usually needed for the subsequent reverse calculation process, it is generally necessary to save the second data and call the second data when the reverse calculation is performed. To this end, the first aspect proposes that the format of the data can be adjusted before the data is saved, that is, the second data originally needs to be saved, but the first aspect adjusts the data format of the second data before the data is saved, that is, the second data is quantized to obtain the third data, and then the third data is saved. The number of bits of the third data is smaller than that of the second data, that is, the accuracy of the third data is lower and the storage space occupied is smaller. Therefore, compared with directly saving the second data, saving the third data occupies less memory space, thereby reducing the storage space overhead of the model training process and improving the performance of model training.

[0008] In addition, the first aspect does not adjust the format of the data during the forward calculation process or before the forward calculation, but adjusts the format of the saved data. The number of bits of the data during the forward calculation process is still the first value, for example, the number of bits of the first data and the second data is still the first value, and the first value is smaller than the second value, that is, the accuracy of the data during the forward calculation process can be relatively high, for example, the data during the forward calculation process can be in a high-precision format such as FP16 or FP32. It can be seen that the accuracy of the data during the calculation process of the first model in the first aspect can be high, so the method in the first aspect has a wider scope of application, for example, it can be applicable to devices whose configured chip computing units do not support low-precision formats, and it can also be applicable to devices whose configured chip computing units support low-precision formats, which can save dynamic memory in the model training process of such devices, thereby improving the performance of model training.

[0009] Optionally, from the perspective of the neural network level, the first data may include at least one or more of the following data: all or part of the data input in the input layer. In this case, it is described in terms of the dimension of the first model as a whole. Optionally, the data input in the input layer may be data obtained by the first device from the cloud, a local database, or a storage device for model training, or the data input in the input layer may be data sent to the first device by other devices; or, all or part of the data output by the previous layer. In this case, it is described from the dimension of a neural network layer.

[0010] Optionally, from the type of the first data, the first data may be all or part of the data output by the input layer, and the first data includes: an activation tensor output by a previous layer or a feature tensor output by a previous layer.

[0011] In a possible implementation manner of the first aspect, the second data includes: an activation tensor or a feature tensor.

[0012] Based on the above implementation, by quantizing the activation tensor or feature tensor, the size of the activation tensor or feature tensor can be compressed. Since the gradient needs to be calculated based on the activation tensor or feature tensor during the reverse calculation process, the activation tensor or feature tensor needs to be saved. Since the size of the activation tensor or feature tensor is compressed, the above implementation can reduce the dynamic memory occupied by saving the activation tensor or feature tensor.

[0013] Optionally, the second data may be all activation tensors or feature tensors obtained by forward calculation of the first model, or activation tensors or feature tensors obtained by forward calculation of all neural network layers of the first model; or partial activation tensors or feature tensors obtained by forward calculation of the first model, or output activation tensors or feature tensors obtained by forward calculation of some neural network layers of the first model, for example, it may be the output activation tensor or feature tensor of the second neural network layer of the first model; for example, it may be the output activation tensor or feature tensor of the third neural network layer and the fourth neural network layer of the first model.

[0014] Optionally, the number of bits of the first data is the sum of the number of sign bits of the first data, the number of exponent bits of the first data, and the number of mantissa bits of the first data; the number of bits of the second data is the sum of the number of sign bits of the second data, the number of exponent bits of the second data, and the number of mantissa bits of the second data; the number of bits of the third data is the sum of the number of sign bits of the third data, the number of exponent bits of the third data, and the number of mantissa bits of the third data, where the mantissa bits may also be referred to as decimal bits.

[0015] Optionally, when the number of bits of the first data or the number of bits of the second data is a first value, the number of bits of the total storage space occupied by the first data, or the size of the first data, may or may not be the first value.

[0016] Optionally, the above number of bits is related to a data format or a storage format.

[0017] Optionally, in the present application, the number of bits of the data when the first model performs forward calculation is a first value, that is, the input data of the forward calculation process of the first model, and the number of bits of the activation tensor, feature tensor, weight and bias of the forward calculation process of the first model is the first value.

[0018] Optionally, the first value is usually related to the data format supported by the chip computing unit of the first device. For example, when the chip of the first device only supports high-precision formats such as FP16 and FP32, the first value can be 16 or 32, but not 8 or 4, that is, in this case, the number of bits of the first data and the second data can be 16 bits or 32 bits, but not 8 bits or 4 bits. It can be seen that the present application does not change the format of the data in the model calculation process, such as the format of the data in the forward calculation or reverse calculation process. The format of the data in the model calculation process can be a data format supported by the chip computing unit. Therefore, the present application can be applicable to chips or devices whose chip computing units only support high-precision data formats, such as chips or devices whose chip computing units do not support low-precision data formats such as FP8, and can also be applicable to chips or devices whose chip computing units can support low-precision data.

[0019] Optionally, the data formats of the first data, the second data and the third data may include at least one or more of the following: the data formats of the first data and the second data are 64-bit floating point numbers (64bit float, FP64), and the data formats of the third data include: FP32, 32-bit fixed-point integers (32bit integer, int32), 32-bit tensor floating point numbers (32bittensorfloat, TF32), BF16, FP16, 16-bit fixed-point integers (16bit integer, int16), FP8, 8-bit fixed-point integers (8bit integer, int8), 6-bit floating point numbers (6bit float, FP6), 4-bit floating point numbers (4bit float, FP4) or NF4; the data format of the first data and the second data is FP32, and the data format of the third data includes: TF32, BF16, FP16, int16, FP8, int8, FP6, FP4 or NF4; the data format of the first data and the second data is TF32, and the data format of the third data includes: BF16, FP16, int16, FP8, int8, FP6, FP4 or NF4; the data format of the first data and the second data is BF16, and the data format of the third data includes: FP8, int8, FP6, FP4 or NF4; the data format of the first data and the second data is F P16, the data format of the third data includes: FP8, int8, FP6, FP4 or NF4; the data format of the first data and the second data is int16, and the data format of the third data includes: FP8, int8, FP6, FP4 or NF4; the data format of the first data and the second data is FP8, and the data format of the third data includes: int8, FP6, FP4 or NF4; the data format of the first data and the second data is int8, and the data format of the third data includes: FP8, FP6, FP4 or NF4; or, the data format of the first data and the second data is FP6, and the data format of the third data includes: FP4 or NF4. Among them, the above-mentioned FP8 includes: 8-bit floating point number-4-bit exponent bits-3-bit mantissa bits (float 8-exponent4-mantissa3, FP8-E4M3) or 8-bit floating point number-5-bit exponent bits-2-bit mantissa bits (float 8-exponent5-mantissa2, FP8-E5M2), and the above-mentioned FP6 includes: 6-bit floating point number-2-bit exponent bits-3-bit mantissa bits (float 6-exponent2-mantissa3, FP6-E2M3) or 6-bit floating point number-3-bit exponent bits-2-bit mantissa bits (float 6-exponent3-mantissa2, FP6-E3M2).

[0020] In a possible implementation of the first aspect, the first device saves the third data and a first scaling factor, where the first scaling factor (scale) is a scaling factor when the second data is quantized, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

[0021] In the above implementation, the number of data elements in the third data corresponding to the first scaling factor is not limited. That is, in the above implementation, the granularity of the scale is not limited by the granularity of the scale supported by the chip computing unit, and the scale can be any granularity, for example, the scale can be a tensor granularity, a channel granularity, a token granularity or a group granularity. Therefore, in the above implementation, the granularity of the scale is relatively flexible and can be flexibly adjusted according to user needs or according to the current device status. For example, when it is necessary to save storage space, the granularity of the scale can be set to be relatively small.

[0022] Optionally, the number of bits of the third data saved in the present application is independent of the data format supported by the chip computing unit of the first device, that is, the second value is independent of the data format supported by the chip computing unit of the first device. For example, when the chip of the first device only supports high-precision formats such as FP16 and FP32, the second value can still be 4 or 8, that is, in this case, the number of bits of the third data can be 8 bits or 4 bits. It can be seen that in this application, the data is not quantized before model training or during model calculation, but the data obtained by forward calculation is quantized before saving. By quantizing the data before saving the data, that is, quantizing the activation tensor or feature tensor obtained by forward calculation, the amount of data of the activation tensor or feature tensor finally saved is reduced, thereby reducing the occupancy of dynamic memory and improving the performance of the model training process.

[0023] In a possible implementation of the first aspect, the first device obtains fourth data from the saved third data; the first device performs a dequantization operation on the fourth data to obtain fifth data, and the accuracy of the fifth data is the same as that of the first data; the first device performs a reverse calculation on the fifth data through the first model.

[0024] Based on the above implementation, before the reverse calculation, the saved data is obtained and dequantized so that the number of bits of the data finally used for the reverse calculation is the same as the number of bits of the first data and the second data in the forward calculation process, that is, in the above implementation, the precision of the data for the forward calculation and the reverse calculation is the same. For example, the precision of the data for the forward calculation and the reverse calculation can both be the precision supported by the chip computing unit of the first device, while the saved data obtained by the forward calculation may not be the precision supported by the chip computing unit of the first device, for example, it may be lower than the precision supported by the chip computing unit. Therefore, this solution can be applied to scenarios where the chip computing unit does not support low precision, and in this scenario it can still save storage space and improve the training performance of the model.

[0025] In a possible implementation of the first aspect, before the reverse calculation of the i-th layer neural network of the first model is completed, the first device obtains the fourth data, where the fourth data is obtained by quantizing the sixth data in the second data, and the sixth data is obtained by forward calculation through the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; the processing unit 901 is also used to: perform reverse calculation on the fifth data through the i+k-th layer neural network.

[0026] In the above implementation, the first device asynchronously obtains the saved fourth data. The so-called asynchronous acquisition means that the fourth data obtained by quantizing the sixth data obtained by the forward calculation of the i+k-th layer neural network is obtained before the reverse calculation of the i-th layer neural network is completed. Therefore, before the reverse calculation of the i+k-th layer neural network, the fourth data has been dequantized to the same accuracy as the data of the forward calculation. Therefore, during the reverse calculation of the i+k-th layer neural network, there is no need to wait for the dequantization of the fourth data, thereby shortening the time required for model training.

[0027] In a possible implementation manner of the first aspect, the fourth data is obtained when the i-th layer neural network performs reverse calculation.

[0028] In the above implementation, the first device obtains the fourth data and the reverse calculation of the fifth data is performed synchronously, that is, the pre-fetch pipeline and the calculation pipeline are parallel, thereby masking the delay caused by the process of obtaining the fourth data and the impact on performance, thereby improving the performance of model training.

[0029] Optionally, when the i-th layer neural network performs reverse calculation, the fourth data is dequantized to obtain the fifth data. That is, the dequantization process of the fourth data is performed simultaneously with the reverse calculation process of the i-th layer neural network.

[0030] In a possible implementation manner of the first aspect, the k is 1.

[0031] In the above implementation, the first device asynchronously obtains the fourth data of the previous layer of the l+1 layer, i.e., the previous layer, before the reverse calculation of the lth layer of the neural network is completed. In this case, since the reverse calculation of the lth layer of the neural network is completed, the reverse calculation of the l+1th layer of the neural network will start. Therefore, the fifth data obtained by dequantizing the fourth data can be used immediately in the process of reverse propagation, and there is no need to save the fifth data or the saving time of the fifth data is short, so the fifth data does not occupy the storage space or the storage space is occupied for a short time, so the storage space can be saved.

[0032] In a possible implementation manner of the first aspect, a data format of the third data is an 8-bit floating point number FP8.

[0033] Based on the above implementation method, when the data format of the third data is FP8, the data volume of the third data is small, and the storage space occupied by the saved third data is small, which can save storage space overhead and thus improve the performance of the model training process.

[0034] In a second aspect, the present application provides a data processing device, which includes: a processing module and a storage module, the processing module being used to perform forward calculation on first data through a first model to obtain second data, the first model being a neural network model, and the number of bits of the first data and the number of bits of the second data being a first value; quantizing the second data to obtain third data, the number of bits of the third data being a second value, and the second value being smaller than the first value; the storage module being used to save the third data, and the saved third data being used for reverse calculation of the first model.

[0035] In a possible implementation of the second aspect, the second data includes: an activation tensor or a feature tensor.

[0036] In a possible implementation of the second aspect, the saving module is specifically used to: save the third data and a first scaling factor, where the first scaling factor is a scaling factor when the second data is quantized, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

[0037] In a possible implementation of the second aspect, the device also includes: an acquisition module, which is used to: acquire fourth data from the saved third data; the processing module is also used to: perform a dequantization operation on the fourth data to obtain fifth data, the number of bits of the fifth data is the first value; and reversely calculate the fifth data through the first model.

[0038] In a possible implementation of the second aspect, the acquisition module is specifically used to: before the reverse calculation of the i-th layer neural network of the first model is completed, obtain the fourth data, the fourth data is obtained by quantizing the sixth data in the second data, and the sixth data is obtained by forward calculation through the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; the processing module is specifically used to: perform reverse calculation on the fifth data through the i+k-th layer neural network.

[0039] In a possible implementation of the second aspect, the fourth data is obtained when the i-th layer neural network performs reverse calculation.

[0040] In a possible implementation of the second aspect, the k is 1.

[0041] In a possible implementation of the second aspect, the data format of the third data is an 8-bit floating point number FP8.

[0042] In a third aspect, the present application provides a data processing device, comprising: a processor and a memory; the memory is used to store programs; the processor is used to execute the programs in the memory, so as to execute the above-mentioned first aspect and any optional method thereof.

[0043] In one possible design, the processor is coupled to the memory.

[0044] In one possible design, the device may also include a bus system.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof, and the above-mentioned first aspect and any optional method thereof.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0047] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, for example, sending or processing the data involved in the above methods; or, information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or training device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0048] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0050] Figure 2 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a neural network of a first model provided in this application;

[0053] Figure 5 A schematic diagram of the forward calculation process of a model provided in this application;

[0054] Figure 6 A schematic diagram of a quantization process provided for this application;

[0055] Figure 7 A schematic diagram of a reverse calculation provided for this application;

[0056] Figure 8 A schematic diagram of a model training process provided for this application;

[0057] Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0060] Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0062] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0063] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged when appropriate, which is just a way of distinguishing the objects of the same attributes when describing the embodiments of the present application. In addition, the terms "system" and "network" in the embodiments of the present application can be used interchangeably. "Including" means "including but not limited to". When A includes multiple elements or situations, A can be one or more of the elements or situations. For example, A includes: B or C, then A can be B, A can be C, and A can also be B and C. "At least one" means one or more, and "multiple" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the front and back associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC or ABC. Also, unless otherwise specified, ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects.

[0064] As used herein, the terms "substantially," "about," and the like are used as terms of approximation, not as terms of degree, and are intended to take into account the inherent deviations of measurements or calculations that one of ordinary skill in the art would know. In addition, the use of "may" when describing embodiments of the present invention means "one or more possible embodiments." As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. In addition, the term "exemplary" is intended to refer to an example or illustration.

[0065] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0066] (1) Infrastructure

[0067] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (such as central processing unit (CPU), neural processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC) or field-programmable gate array (FPGA) and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0068] (2) Data

[0069] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0070] (3) Data processing

[0071] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0072] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0073] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0074] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0075] (4) General ability

[0076] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0077] (5) Smart products and industry applications

[0078] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0079] The embodiments of the present application can be applied to natural language processing, image processing, computer vision, speech processing, industrial neural network, etc. A neural network model is trained to obtain a trained neural network model.

[0080] Task processing for several scenarios.

[0081] First, the application scenarios of the present application are introduced. The present application can be applied to, but not limited to, natural language processing, image processing, computer vision, speech processing, industrial neural networks, and other scenarios. Specifically, the embodiments of the present application can train a neural network model, and the trained neural network model can perform task processing in the above scenarios.

[0082] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the following first introduces the relevant concepts such as neural networks involved in the embodiments of the present application.

[0083] 1. Neural Network Model

[0084] A neural network model is a model composed of different layers, each of which performs a specific function, thereby realizing layer-by-layer abstraction and processing of input data. A neural network can usually include: an input layer, a hidden layer, or an output layer. A neural network can include multiple hidden layers, and the number of hidden layers depends on the network complexity of the neural network model. The following is a detailed introduction to the input layer, hidden layer, and output layer of a neural network:

[0085] ①The input layer of a neural network is the first layer of the neural network. In a neural network, each neuron in the input layer corresponds to a feature in the input data. The number of neurons in the input layer depends on the number of features in the input data. For example, in an image classification task, each pixel is a neuron in the input layer. The function of the input layer is to convert the input data into a format that can be processed by the neural network, such as converting image data into a vector form. The output of the input layer is the input passed to the next layer of neurons, which can be a hidden layer or an output layer.

[0086] ②The hidden layer of a neural network is one or more layers located between the input layer and the output layer. Its function is to convert the input data into a higher-level feature representation. The number of neurons and the way they are connected in different hidden layers can be different, depending on the specific neural network architecture. The number of hidden layers in a neural network can be 0 or any positive integer. And with the continuous development of the model, the number of layers and hidden layers in the neural network is also increasing. The introduction of hidden layers allows the network to represent complex nonlinear functions and solve problems that linear models cannot handle. Hidden layers include: Hidden layers mainly include: convolutional layers, fully connected layers, pooling layers, normalized exponential layers or activation layers. The input of the hidden layer is the output of the previous layer of neurons, where the previous layer can be the output of the input layer or the previous hidden layer. The output of the hidden layer is the input passed to the next layer of neurons, where the next layer can be the next hidden layer or the output layer.

[0087] ③The output layer of the neural network is the last layer in the neural network, and its output is the prediction or classification result of the neural network for the input data. The design of the output layer needs to be adjusted according to the specific task to adapt to different application scenarios.

[0088] 2. Forward calculation

[0089] Forward calculation refers to the propagation process of the neural network from the input layer to the output layer. During the forward propagation process, weighted and activated calculations are performed in each layer of nodes to obtain the output result of the current layer and pass it to the next layer. This process continues until the output layer of the neural network, and finally the loss under the current parameters is obtained. From the overall process of forward calculation, the neural network model uses the input data and the current model parameters to calculate and generate predicted results. The system will compare these predicted results with the labeled data to calculate the current predicted loss.

[0090] 3. Reverse calculation

[0091] Backward calculation refers to the calculation of the gradient of the loss function for each parameter from the output layer to the input layer of the neural network based on the loss obtained by the forward calculation, and the update of related parameters at a given learning rate. In the back propagation process, the error can be decomposed into the errors of different nodes, and the error is passed back layer by layer through the chain rule to calculate the gradient of weights and biases, and the parameters of the neural network are continuously adjusted with the gradient descent algorithm. Through the back propagation process, the neural network continuously adjusts its parameters to minimize the error of the objective function.

[0092] 4. Large Model

[0093] Large models refer to machine learning models with large-scale parameters and complex computational structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. Large models are designed to improve the expressiveness and predictive performance of the model and to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Large models learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data.

[0094] The following is an example of the system architecture on which the embodiments of the present application are based.

[0095] Figure 2 This is a schematic diagram of the system architecture provided in the embodiment of the present application. Figure 2 As shown, the system architecture 200 includes an execution device 210 , a training device 220 , a database 230 , a client device 240 , a data storage system 250 , and a data collection system 260 .

[0096] The execution device 210 includes a calculation module 211, an input / output (I / O) interface 212, a preprocessing unit 213 and a preprocessing unit 214. The calculation module 211 may include the target model / rule 201, and the preprocessing unit 213 and the preprocessing unit 214 are optional.

[0097] The data acquisition device 260 is used to collect training samples. The training samples may be image data or text data, etc. After collecting the training samples, the data acquisition device 260 stores the training samples in the database 230 .

[0098] The training device 220 can train the neural network model to be trained (such as the first model in the method 100 described below) based on the training samples maintained in the database 230 to obtain the target model / rule 201.

[0099] It should be understood that the training device 220 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 230, or fine-tune the model based on the pre-training.

[0100] It should be noted that, in actual applications, the training samples maintained in the database 230 may not all come from the data acquisition device 260, but may also be received from other devices. It should also be noted that the training device 220 may not train the target model / rule 201 entirely based on the training samples maintained in the database 230, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0101] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as Figure 2 The execution device 210 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle terminal, etc., or a server, etc.

[0102] Specifically, the training device 220 may transfer the trained model to the execution device 210 .

[0103] exist Figure 2 In the embodiment, the I / O interface 212 configured by the execution device 210 is used for data interaction with an external device, and a user can input data (such as image data or text data in the embodiment of the present application) into the I / O interface 212 through the client device 240.

[0104] The preprocessing unit 213 and the preprocessing unit 214 are used to preprocess the input data received by the I / O interface 212. It should be understood that there may be no preprocessing unit 213 and the preprocessing unit 214 or only one preprocessing unit. When there are no preprocessing unit 213 and the preprocessing unit 214, the computing module 211 may be directly used to process the input data.

[0105] When the execution device 210 preprocesses the input data, or when the computing module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0106] Finally, the I / O interface 212 provides the processing results to the client device 240 and thus to the user.

[0107] exist Figure 2 In the case shown, the user can manually give input data, and the "manually given input data" can be operated through the interface provided by the I / O interface 212. In another case, the client device 240 can automatically send input data to the I / O interface 212. If the client device 240 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as shown in the figure as new sample data, and store them in the database 230. Of course, it is also possible not to collect through the client device 240, but the I / O interface 212 directly stores the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as new sample data in the database 230.

[0108] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210. It should be understood that the execution device 210 may be deployed in the client device 240.

[0109] From the training side of the model:

[0110] In the embodiment of the present application, the training device 220 can obtain the memory ( Figure 2 Not shown in the figure, the code stored in the training device 220 (or deployed separately from the training device 220) can be integrated into the training device 220 to implement the steps related to model training in the embodiments of the present application.

[0111] In an embodiment of the present application, the training device 220 may include a hardware circuit (such as an ASIC, FPGA, a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 220 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0112] It should be understood that the training device 220 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 220 that does not have the function of executing instructions, which is not limited here.

[0113] As mentioned above, neural network models are used in more and more fields. In order to improve the accuracy and performance of neural network models, it is necessary to train the neural network models. The training process of neural network models usually includes: forward calculation and reverse calculation. For example, forward calculation can be the propagation process of the neural network model from the input layer to the output layer. During the forward calculation process, weighted and activated calculations are performed in each layer of the neural network to obtain the output result of the current layer and pass it to the next layer of the neural network. This process continues until the output layer of the neural network model, and finally the loss under the current parameters is obtained. For example, reverse calculation can be based on the loss obtained by forward calculation, and the gradient of the loss function for each parameter is calculated from the output layer to the input layer of the neural network model in sequence, and the relevant parameters are updated under a given learning rate. When training a neural network model, chips are usually required. However, the computing units of some chips do not support low-precision formats such as FP8, but only support high-precision formats such as FP16 or FP32. When using such chips for model training, the data in the model calculation process has high precision, for example, the data precision of forward calculation and reverse calculation is high. Compared with low-precision data, high-precision data occupies more storage space, which results in a large storage space overhead during model training. To this end, the embodiment of the present application provides a method 100, which can save the storage space occupied during model training and improve the performance of model training. The method can be applied to a wide range of chips, such as chips whose computing units do not support low-precision formats such as floating point 8 (FP8). When the method 100 is applied to such chips or modules or devices configured with such chips, dynamic memory can be saved.

[0114] Figure 3 is a schematic flow chart of the method 100 provided in the embodiment of the present application. Optionally, when the method 100 is applied to Figure 2 When the system shown in FIG. 1 is used, the method 100 may be performed by the training device 220. It should be noted that Figure 3The method is illustrated by taking the first device as the execution subject of the interaction diagram as an example, but the present application does not limit the execution subject of the interaction diagram. For example, the execution subject of S301-S303 and related implementation methods can be the first device, and optionally, the first device can be a server, an accelerator or a terminal. The first device in S301-S303 and related implementation methods can also be replaced by a chip, a chip system, or a processor that supports the first device to implement the method, and can also be replaced by a logic module or software that can implement all or part of the controller functions.

[0115] like Figure 3 As shown, the method 100 provided in the embodiment of the present application includes the following steps:

[0116] S301: The first device performs forward calculation on the first data by using the first model to obtain second data;

[0117] The first model is a neural network model, and the number of bits of the first data and the number of bits of the second data are first values.

[0118] Optionally, the above S301 can be replaced by: the first device performs forward calculation on the first data through the m+1th layer neural network of the first model to obtain second data, and the first data is obtained by forward calculation of the mth layer neural network of the first model; or, the first device performs forward calculation on the first model according to the first data to obtain the second data.

[0119] Optionally, the first model may also be replaced by the first rule.

[0120] Optionally, the first model includes: a large model, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short term memory (LSTM), a feedforward neural network (FNN), a feedback neural network, a fully connected neural network (FCN), a generative adversarial network (GANs), a text convolutional neural network (TextCNN), a residual network (ResNet), a deep residual network (DRN), a transformer model (transformer), a deep convolutional network (DCN), a deconvolutional network (DN), a liquid state machine (LSM), an extreme learning machine (ELM), or a deep belief nets (DBN). Among them, the large model is also called the foundation model, and the large model includes: a large language model (LLM), a Vincent-graph large model, a Vincent-video large model or a multimodal large model. With the continuous development of the large model, the embodiment of the present application can also be applied to the large model that will evolve in the future.

[0121] As mentioned above, the first model, as a neural network model, may include a multi-layer neural network. Figure 4 For example, Figure 4 The first model may include an input layer, N hidden layers, and an output layer, where N is a positive integer.

[0122] It should be noted that forward computing (forward) in this application is an exemplary name and can also be replaced by any possible name, such as forward propagation, forward algorithm or forward, etc.

[0123] The first data in this application is introduced below:

[0124] From the perspective of the neural network level, the first data may include at least one or more of the following data:

[0125] ① All or part of the data input in the input layer. In this case, it is described in terms of the overall dimension of the first model. Optionally, the data input in the input layer can be data obtained by the first device from the cloud, a local database, or a storage device for model training. The data input in the input layer can also be data sent to the first device by other devices. Figure 2 For example, the first device may be the training device 110, and the first data may be obtained by the first device from the database 130, or may be sent to the first device by the client device 140 or the data acquisition device 160; or,

[0126] ② All or part of the data output by the previous layer. In this case, it is described from the dimension of a neural network layer. Figure 4 For example, for Figure 4 In the hidden layer 1, the data output by the previous layer can be one or more of the data output by the input layer. Figure 4 In the hidden layer 2, the data output by the previous layer can be one or more of the data output by the hidden layer 1.

[0127] From the perspective of the type of the first data, optionally, the first data may be all or part of the data output by the input layer. Figure 4 For example, the data may be x1, x2, and x3. When the first data is all or part of the data output by the previous layer, the first data includes: an activation tensor output by the previous layer or a feature tensor output by the previous layer.

[0128] Optionally, the activation tensor in this application is an exemplary name and can also be replaced by any possible name, such as activation matrix, activation value, output value, intermediate result or discrimination result.

[0129] Optionally, the feature tensor in the present application is an exemplary name and can also be replaced by any possible name, such as a feature map, a feature value or a feature parameter.

[0130] The second data in this application is introduced below:

[0131] In an optional implementation, the second data includes: an activation tensor or a feature tensor.

[0132] Based on the above implementation, by quantizing the activation tensor or feature tensor, the size of the activation tensor or feature tensor can be compressed. Since the gradient needs to be calculated based on the activation tensor or feature tensor during the reverse calculation process, the activation tensor or feature tensor needs to be saved. Since the size of the activation tensor or feature tensor is compressed, the above implementation can reduce the dynamic memory occupied by saving the activation tensor or feature tensor.

[0133] Optionally, the second data may be all activation tensors or feature tensors obtained by forward calculation of the first model, or activation tensors or feature tensors obtained by forward calculation of all neural network layers of the first model; or partial activation tensors or feature tensors obtained by forward calculation of the first model, or output activation tensors or feature tensors obtained by forward calculation of some neural network layers of the first model, for example, it may be the output activation tensor or feature tensor of the second neural network layer of the first model; for example, it may be the output activation tensor or feature tensor of the third neural network layer and the fourth neural network layer of the first model.

[0134] The following describes the forward calculation process in this application:

[0135] Example a1: Use algebraic methods to describe the forward calculation process of the first model, as follows:

[0136] See also Figure 5 , Figure 5 The neural network shown is an example of a first model including one hidden layer. Figure 5 Assume that the input layer has 3 neurons, the hidden layer has 3 neurons, and the output layer has 2 neurons. Figure 5 The forward calculation process in this application is introduced by taking the neural network shown as an example.

[0137] assumed is the weight of the kth neuron in the l-1th layer to the jth neuron in the lth layer of the first model, is the bias of the jth neuron in the lth layer of the first model, is the activation tensor of the jth neuron in the lth layer of the first model.

[0138] Assume that the three neurons in the input layer of the first model output x1, x2 and x3 respectively, and the hidden layer of the first model has three neurons. Relative to the overall neural network of the first model, the hidden layer is the second layer (layer 2). Then the activation tensor output by the hidden layer includes: and Optionally, and It can be calculated by the following formula:

[0139]

[0140]

[0141]

[0142] Where f is the activation function, for example, f can be Sigmoid, ReLU, PReLU and other functions.

[0143] Assuming that the output layer of the first model has 2 neurons, relative to the overall neural network of the first model, the output layer is the third layer (layer3), then the activation tensor output by the output layer includes: and Specifically, and It can be calculated by the following formula:

[0144]

[0145]

[0146] Optionally, in this example, for the hidden layer, the first data includes: x1, x2 or x3; for the output layer, the first data includes: or

[0147] Optionally, in this example, for the hidden layer, the second data includes: or For the output layer, the second data includes: or

[0148] Example a2: Use the matrix method to describe the forward calculation process of each layer of the first model, as follows:

[0149] Assume W (l) is the weight of the first model's layer l, b (l) is the bias of the lth layer of the first model, a (l) is the activation tensor of the lth layer of the first model, a (l-1) is the activation tensor of the l-1th layer of the first model, and f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network. For example, f can be a function such as sigmoid, ReLU, PReLU, etc. For example, the forward calculation process of each layer is expressed as follows by matrix multiplication:

[0150] z (l) =W (l) a (l-1) +b (l)

[0151] a (l) =σ(z (l) )

[0152] Optionally, in this example, the first data is a (l-1), that is, the activation tensor output by the previous neural network layer of the l-th neural network of the first model, where l-1 can be the serial number of any one or more neural network layers of the first model, for example, 1 and / or 2, etc. Correspondingly, the first data can be a 1 and / or a 2 wait.

[0153] Optionally, in this example, the second data is a l , that is, the activation tensor output by the lth layer of the first model, or the second data is the activation tensor output by this layer. Wherein, l can be the serial number of any one or more neural network layers of the first model, for example, it can be 2 and / or 3, etc. The second data can be a 2 and / or a 3 wait.

[0154] S302: The first device quantizes the second data to obtain third data;

[0155] The number of bits of the third data is a second value, and the second value is smaller than the first value.

[0156] The data formats of the first data, the second data, and the third data in this application are introduced below:

[0157] As mentioned above, the number of bits of the first data and the number of bits of the second data are the first value, and the number of bits of the third data is the second value.

[0158] It should be noted that the number of digits is an exemplary name and can also be replaced by the number of bits, etc. The size of the number of digits is related to the precision of the data. The larger the number of digits, the higher the precision of the data, and the smaller the number of digits, the lower the precision of the data.

[0159] Optionally, the number of bits of the first data is the sum of the number of sign bits of the first data, the number of exponent bits of the first data, and the number of mantissa bits of the first data; the number of bits of the second data is the sum of the number of sign bits of the second data, the number of exponent bits of the second data, and the number of mantissa bits of the second data; the number of bits of the third data is the sum of the number of sign bits of the third data, the number of exponent bits of the third data, and the number of mantissa bits of the third data, where the mantissa bits may also be referred to as decimal bits.

[0160] Optionally, when the number of bits of the first data or the number of bits of the second data is a first value, the number of bits of the total storage space occupied by the first data, or the size of the first data may or may not be the first value. For example, when the number of bits of the first data is 16, i.e., 16 bits, the total storage space occupied by the first data may be 200 bits or 16 bits.

[0161] Optionally, the number of bits is related to a data format or a storage format. For example, the corresponding relationship between the number of bits and the data format or the storage format may be referred to one or more columns in Table 1 and / or Table 2:

[0162] Table 1

[0163] Data Format Sign bit Exponent Mantissa Number of digits FP6 1 11 52 64 FP32 1 8 23 32 TF32 1 8 10 19 BF16 1 8 7 16 FP16 1 5 10 16 FP8-E4M3 1 4 3 8 FP8-E5M2 1 5 2 8 FP6-E2M3 1 2 3 6 FP6-E3M2 1 3 2 6 FP4 1 2 1 4

[0164] Table 2

[0165] Data Format Number of digits int32 32 int16 16 int8 8 NF4 4

[0166] To help understand, several examples are given in conjunction with the above Table 1. For example, when the data format of the first data is FP16, the number of bits of the first data is 16, that is, the first value is 16. For example, when the data format of the second data is FP32, the number of bits of the first data is 32, that is, the first value is 32. For example, when the data format of the third data is FP8-E4M3, the number of bits of the third data is 8, that is, the second value is 8.

[0167] Optionally, in the present application, the number of bits of the data when the first model performs forward calculation is a first value, that is, the input data of the forward calculation process of the first model, and the number of bits of the activation tensor, feature tensor, weight and bias of the forward calculation process of the first model is the first value.

[0168] Optionally, the first value is usually related to the data format supported by the chip computing unit of the first device. For example, when the chip of the first device only supports high-precision formats such as FP16 and FP32, the first value can be 16 or 32, but not 8 or 4, that is, in this case, the number of bits of the first data and the second data can be 16 bits or 32 bits, but not 8 bits or 4 bits. It can be seen that the present application does not change the format of the data in the model calculation process, such as the format of the data in the forward calculation or reverse calculation process. The format of the data in the model calculation process can be a data format supported by the chip computing unit. Therefore, the present application can be applicable to chips or devices whose chip computing units only support high-precision data formats, such as chips or devices whose chip computing units do not support low-precision data formats such as FP8-E4M3, FP8-E4M3, FP4 or NF4, and can also be applicable to chips or devices whose chip computing units can support low-precision data.

[0169] In an optional implementation, the data format of the third data is 8-bit floating point number FP8.

[0170] Based on the above implementation method, when the data format of the third data is FP8, the data volume of the third data is small, and the storage space occupied by the saved third data is small, which can save storage space overhead and thus improve the performance of the model training process.

[0171] It should be noted that the above implementation is a typical implementation of the present application, but the third data in the present application may also be in other formats, and there are many possible formats of the first data and the second data. For example, the data formats of the first data, the second data, and the third data in the present application may include at least one or more of the following:

[0172] ① The data format of the first data and the second data is FP64, and the data format of the third data includes: FP32, int32, TF32, BF16, FP16, int16, 8-bit floating point number FP8, int8, 6-bit floating point number FP6, FP4 or NF4;

[0173] ② The data format of the first data and the second data is FP32, and the data format of the third data includes: TF32, BF16, FP16, int16, FP8, int8, FP6, FP4 or NF4;

[0174] ③ The data formats of the first data and the second data are TF32, and the data formats of the third data include: BF16, FP16, int16, FP8, int8, FP6, FP4 or NF4;

[0175] ④ The data formats of the first data and the second data are BF16, and the data formats of the third data include: FP8, int8, FP6, FP4 or NF4;

[0176] ⑤ The data formats of the first data and the second data are FP16, and the data formats of the third data include: FP8, int8, FP6, FP4 or NF4;

[0177] ⑥ The data format of the first data and the second data is int16, and the data format of the third data includes: FP8, int8, FP6, FP4 or NF4;

[0178] ⑦ The data formats of the first data and the second data are FP8, and the data formats of the third data include: int8, FP6, FP4 or NF4;

[0179] ⑧ The data formats of the first data and the second data are int8, and the data format of the third data includes: FP8, FP6, FP4 or NF4; or,

[0180] ⑨ The data format of the first data and the second data is FP6, and the data format of the third data includes: FP4 or NF4.

[0181] Among them, the above-mentioned FP8 includes: FP8-E4M3 or FP8-E5M2, and the above-mentioned FP6 includes: FP6-E2M3 or FP6-E3M2.

[0182] In the above ① to ⑨, only the data format of the first data, the second data, and the third data is introduced as a single format. The data format of the first data, the second data, and / or the third data may also be a mixture of multiple formats. For example, the format of the first data and the second data may be a mixture of FP32 and FP16, that is, the data format of a part of the first data and the second data is FP32, the data format of another part of the first data and the second data is FP16, and the third data is FP8. For example, the data format of the first data and the second data may be FP16, and the third data may be a mixture of FP8 and FP4, that is, the data format of a part of the third data is FP8, and the data format of a part of the third data is FP4.

[0183] The process of quantifying the second data in this application is described below:

[0184] As mentioned above, the second data is quantized to obtain the third data, wherein quantization is a data compression technology. By quantizing the second data into the third data, the data size can be greatly reduced, thereby reducing the size of the final saved data, that is, the saved data is changed from the original third data with a larger data volume to the second data with a smaller data volume, thereby saving dynamic memory.

[0185] In one example, the quantization can be layer-by-layer quantization, that is, a layer of neural network is used as a quantization unit, and each layer of the network has a set of quantization parameters. In this case, the granularity of the quantization process is the tensor granularity. For example, multiple neurons in the i-th layer of the neural network of the first model perform forward calculations to obtain multiple second data, and the multiple second data of the i-th layer of the neural network correspond to the same set of quantization parameters.

[0186] In another example, the quantization can be channel-by-channel quantization, that is, each quantization channel of a layer of the network is used as a unit, and each channel uses a set of quantization parameters separately. Channel-by-channel quantization can obtain higher quantization accuracy due to finer quantization granularity, but the calculation is also more complicated. Channel-by-channel quantization includes two cases. For example, when the granularity of the quantization process is the token granularity, then for the activation tensor, each row corresponds to a quantization parameter. For example, when the granularity of the quantization process is the channel granularity, then for the weight w, each column corresponds to a quantization parameter.

[0187] In another example, the above quantization can be group-by-group quantization, and the granularity of the quantization process is group granularity, and each group (for example, K rows (for activation tensors) or K columns (for weights)) uses a set of s and z. The group granularity is between tensor and channel, for example, group_size=128 corresponds to one quantization parameter. When group_size=1, group-by-group quantization is equivalent to layer-by-layer quantization; when group_size=num_filters (for example, dw (Depthwise) turns the convolution kernel into a single channel), group-by-group quantization is equivalent to channel-by-channel quantization.

[0188] Optionally, the above quantization parameter may be scale.

[0189] See also Figure 6 , Figure 6 Taking the above quantization process using scaling technology as an example, first, the scaling operation needs to be performed on the second data, that is, the second data needs to be multiplied by a scale, which can be a floating point number or an integer. Then, the data obtained after multiplying the second data by the scale is converted into a numerical format to obtain the third data. For example, when the numerical format of the second data is FP16 and the numerical format of the third data is FP8, the above numerical format conversion is converted from FP16 to FP8.

[0190] After the second data is quantized to obtain the third data, S303 may be performed, which is specifically as follows:

[0191] S303: The first device saves the third data.

[0192] The saved third data is used for reverse calculation of the first model.

[0193] Optionally, the above-mentioned "the saved third data is used for the reverse calculation of the first model" can also be replaced by "the saved third data is related to the reverse calculation of the first model" or "the saved third data is processed to obtain data A, and the data A is reverse calculated through the first model."

[0194] It should be noted that, as mentioned above, the second data obtained by the forward calculation of the first model, such as an activation tensor or a feature tensor, etc. Since the activation tensor or the feature tensor, etc., are also needed to calculate the gradient (dW) of the weight, it is usually necessary to save the second data. In order to save storage space, the present application adjusts the data format of the second data before saving, that is, the second data is quantized to obtain the third data, and the number of bits of the third data finally saved is less than the number of bits of the second data, that is, the third data has lower precision and occupies less memory.

[0195] It should be noted that “the saved third data is used for the reverse calculation of the first model” does not limit the saved third data to be directly used for the reverse calculation of the first model. Usually, the third data needs to be processed in a certain way before being used for the reverse calculation of the first model.

[0196] Optionally, the number of bits of the third data saved in the present application is independent of the data format supported by the chip computing unit of the first device, that is, the second value is independent of the data format supported by the chip computing unit of the first device. For example, when the chip of the first device only supports high-precision formats such as FP16 and FP32, the second value can still be 4 or 8, that is, in this case, the number of bits of the third data can be 8 bits or 4 bits. It can be seen that in this application, the data is not quantized before model training or during model calculation, but the data obtained by forward calculation is quantized before saving. By quantizing the data before saving the data, that is, quantizing the activation tensor or feature tensor obtained by forward calculation, the amount of data of the activation tensor or feature tensor finally saved is reduced, thereby reducing the occupancy of dynamic memory and improving the performance of the model training process.

[0197] Optionally, the first device may store the third data in a video memory, for example, in a high bandwidth memory (HBM), graphics double data rate version 5 (GDDR5), GDDR5X, graphics double data rate version 6 (GDDR6), or GDDR6X. GDDR5X is an upgraded version of GDDR5, and GDDR6X is an upgraded version of GDDR6.

[0198] In addition to storing the third data, the first device may also store other data, as follows:

[0199] In an optional implementation, the first data saves the third data and a first scaling factor, where the first scaling factor is a scaling factor when the second data is quantized, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

[0200] In the above implementation, the number of data elements in the third data corresponding to the first scaling factor is not limited. That is, in the above implementation, the granularity of the scale is not limited by the granularity of the scale supported by the chip computing unit, and the scale can be any granularity, for example, the scale can be tensor granularity, channel granularity, token granularity or group granularity. Therefore, in the above implementation, the granularity of the scale is relatively flexible and can be flexibly adjusted according to user needs or according to the current device status. For example, when it is necessary to save storage space, the granularity of the scale can be set to be relatively small.

[0201] It should be noted that since the scale in this solution is independent of the calculation process of the forward calculation or reverse calculation model, the scale is a scaling factor when the third data is quantized before saving, and is therefore not limited by the granularity of the scale supported by the chip computing unit.

[0202] The above introduces the forward calculation process of the first model and the process of saving the relevant data obtained by the forward calculation. After the above process is completed, the training of the first model will enter the next stage, that is, the reverse calculation stage. The reverse calculation process of the first model is introduced below:

[0203] In an optional implementation, the first device obtains fourth data from the saved third data; performs a dequantization operation on the fourth data to obtain fifth data, the accuracy of the fifth data is the same as that of the first data; and reversely calculates the fifth data through the first model.

[0204] Based on the above implementation, before the reverse calculation, the saved data is obtained and dequantized so that the number of bits of the data finally used for the reverse calculation is the same as the number of bits of the first data and the second data in the forward calculation process, that is, in the above implementation, the precision of the data for the forward calculation and the reverse calculation is the same. For example, the precision of the data for the forward calculation and the reverse calculation can both be the precision supported by the chip computing unit of the first device, while the saved data obtained by the forward calculation may not be the precision supported by the chip computing unit of the first device, for example, it may be lower than the precision supported by the chip computing unit. Therefore, this solution can be applied to scenarios where the chip computing unit does not support low precision, and in this scenario it can still save storage space and improve the training performance of the model.

[0205] It should be noted that in the present application, backward is an exemplary name and can also be replaced by any possible name, such as back propagation, reverse algorithm or reverse, etc.

[0206] To help understand the above implementation, an example is given: after the forward calculation process of each layer of the first model is completed, the activation tensor or feature tensor obtained by the forward calculation of each layer can be obtained, that is, the second data, and the second data is quantized to obtain the third data, and then the third data is saved. The reverse calculation process is also calculated layer by layer by neural network. Before the reverse calculation of the p-th layer neural network, the fourth data related to the p-th layer neural network can be obtained and saved, where p is a positive integer. The so-called "fourth data related to the p-th layer neural network" means: the fourth data is obtained by quantizing the data obtained by the forward calculation of the p-th layer neural network, such as the activation tensor or feature tensor, etc., wherein the second data may include the data obtained by the forward calculation of the multi-layer neural network, and the "data obtained by the forward calculation of the p-th layer neural network" is a part of the second data. Then the fourth data is dequantized to obtain the fifth data, the number of bits of the fifth data is the same as the number of bits of the second data, that is, the same as the precision of the second data. Then the first device calculates the fifth data through the p-th layer neural network of the first model.

[0207] Regarding the time for obtaining the fourth data, the above implementation only limits the time for obtaining the fourth data before the reverse calculation of the fifth data. The time for obtaining the fourth data may be further limited. For details, please refer to the following example:

[0208] Example b1: The first device obtains the fourth data before the reverse calculation of the i-th layer neural network of the first model is completed. The fourth data is obtained by quantizing the sixth data in the second data. The sixth data is obtained by forward calculation through the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; the first device performs reverse calculation on the fifth data through the i+k-th layer neural network.

[0209] In example b1, the first device asynchronously obtains the saved fourth data. The so-called asynchronous acquisition means that the fourth data obtained by quantizing the sixth data obtained by the forward calculation of the i+k-th layer neural network is obtained before the reverse calculation of the i-th layer neural network is completed. Therefore, before the reverse calculation of the i+k-th layer neural network, the fourth data has been dequantized to the same accuracy as the data of the forward calculation. Therefore, during the reverse calculation of the i+k-th layer neural network, there is no need to wait for the dequantization of the fourth data, thereby shortening the time required for model training.

[0210] It should be noted that it is assumed that the neural network has N layers (1, 2, ..., i, i+1, ..., N), N is a positive integer, the forward calculation order is i->i+1, and the reverse calculation order is i+1->i. The i+k and i in Example b1 are the network layer order defined by the reverse calculation. If the network layer order defined by the forward calculation is followed, the above example is replaced by: the first device obtains the fourth data before the reverse calculation of the i+k-th layer neural network of the first model is completed. The fourth data is obtained by quantizing the sixth data in the second data. The sixth data is obtained by forward calculation of the i-th layer neural network of the first model. i is a positive integer and k is a positive integer; then the first device performs reverse calculation on the fifth data through the i-th layer neural network.

[0211] Optionally, in example b1, based on asynchronously acquiring the fourth data, the time for acquiring the fourth data may be further limited:

[0212] In one example, the fourth data is obtained when the i-th layer neural network performs reverse calculation.

[0213] In the above example, the first device obtains the fourth data and performs reverse calculations of the fifth data synchronously, that is, the pre-fetch pipeline and the calculation pipeline are parallel, thereby masking the delay caused by the process of obtaining the fourth data and the impact on performance, thereby improving the performance of model training.

[0214] In another example, when the i-th layer neural network performs reverse calculation, the fourth data is dequantized to obtain the fifth data. That is, the dequantization process of the fourth data is performed simultaneously with the reverse calculation process of the i-th layer neural network.

[0215] In the above example, since the dequantization process and the reverse calculation process are performed simultaneously, the time required for the dequantization and reverse calculation as a whole can be shortened, thereby reducing the delay of the model training process.

[0216] In yet another example, k is 1.

[0217] In the above example, the first device asynchronously obtains the fourth data of the previous layer of the l+1 layer, i.e., the previous layer, before the reverse calculation of the lth layer of the neural network is completed. In this case, since the reverse calculation of the lth layer of the neural network is completed, the reverse calculation of the l+1th layer of the neural network will start. Therefore, the fifth data obtained by dequantizing the fourth data can be used immediately in the process of reverse propagation, and there is no need to save the fifth data or the saving time of the fifth data is short. Therefore, the fifth data does not occupy storage space or occupies storage space for a short time, so storage space can be saved.

[0218] It should be noted that the above four examples can cooperate with each other. For example, when the fourth data can be obtained asynchronously, the reverse calculation and the acquisition of the fourth data can be performed simultaneously, and / or the reverse calculation and the inverse quantization can be performed simultaneously. Figure 7 For example, Figure 7 is a typical example of this application. Figure 7 The scheme combining each of the above four examples is as follows:

[0219] See also Figure 7 The upper part of Figure 7 The upper part is a processing flow chart of asynchronous pre-dequantization of a data in the reverse calculation stage. When the first device performs reverse calculation on the i-th linear layer (linear_i), that is, the i-th layer neural network of the first model, the a data of the i+1-th linear layer (linear_i+1) is obtained in advance, that is, the a data of the i+1-th layer neural network of the first model. The a data here is an example of the fourth data. The a data represents the data of quantizing the activation tensor obtained by forward calculation of the linear_i+1 neural network. The first device saves the a data of linear_i+1. Then, the a data of linear_i+1 is dequantized. The dequantization process can be performed simultaneously with the reverse calculation process of linear_i. The dequantized data can be used for the reverse calculation of linear_i+1. In the reverse calculation process of linear_i+1, the a data of the i+2th linear layer (linear_i+2) is saved in advance, that is, the a data of the i+1th layer of the neural network of the first model, and then the first device can also dequantize the a data of (linear_i+2). The dequantization process can be performed simultaneously with the reverse calculation process of linear_i+1, and the dequantized data can be used for the reverse calculation of linear_i+2. By analogy, in the reverse calculation process of each layer of the neural network, the a data of the next layer can be pre-fetched and saved, and then the a data is dequantized. The dequantization process and the reverse calculation of the neural network can be performed simultaneously, and the dequantized data can be used for the reverse calculation of the next layer.

[0220] See also Figure 7 The lower part of Figure 7 The lower part is the data flow diagram of asynchronous pre-dequantization of a data in the reverse calculation stage, where Figure 7The lower half of includes two dotted boxes. The upper dotted box represents the reverse calculation of linear_i, that is, the calculation pipeline, and the lower dotted box represents the pre-fetching of the a data of the linear_i+1 layer, that is, the pre-fetch pipeline. The upper and lower dotted boxes are performed synchronously, that is, the calculation pipeline and the pre-fetch pipeline are concurrent, thereby concealing the delay caused by the acquisition of the a data and the dequantization process of the a data. In the upper dotted box, the first device obtains the saved W from the HBM, where W is the weight of linear_i, and W can be used for the reverse calculation process of linear i. In the upper dotted box, the first device obtains the saved a data of linear_i+1 from the HBM, and then dequantizes the a data into a vector (vector, vec), which can be used for the reverse calculation of linear_i+1. Specifically, the vec can be used as the a data of the reverse calculation process of linear_i+1.

[0221] Optionally, when performing the reverse calculation of the i-th layer neural network, the fourth data of the i+1-th layer neural network is obtained in advance and the fourth data is dequantized. Compared with the process of the i-th layer neural network reverse calculation, the time for obtaining the fourth data and dequantizing the fourth data is usually shorter. Therefore, it can be ensured that before the reverse calculation of the i+1-th layer neural network starts, the fourth data has been dequantized and can be directly used for the reverse calculation of the i+1-th layer neural network.

[0222] Example b2: The first device may also obtain the fourth data after the reverse calculation of the i-th layer neural network of the first model is completed and before the reverse calculation of the i+1-th layer neural network.

[0223] In example b2, the first device does not asynchronously obtain the stored fourth data, that is, it does not obtain the fourth data before the reverse calculation process of the previous layer is completed, but obtains it after the reverse calculation process of the previous layer is completed. The advantage of this method is that after the fourth data is dequantized into the fifth data, the fifth data does not need to be saved or the time for saving the fifth data is short, so the fifth data does not occupy storage space or occupies storage space for a short time, which can save storage space.

[0224] The process of dequantizing the second data in this application is described below:

[0225] In one example, the quantization can be layer-by-layer inverse quantization, that is, a layer of neural network is used as a quantization unit, and each layer of the network has a set of inverse quantization parameters. At this time, the granularity of the quantization process is the tensor granularity. For example, the fourth data of the i-th layer of the neural network of the first model can correspond to the same set of quantization parameters.

[0226] In another example, the quantization may be channel-by-channel inverse quantization. For details, please refer to the description of the group-by-group quantization, which will not be repeated here.

[0227] In another example, the quantization may be group-by-group inverse quantization, and the granularity of the inverse quantization process is the group granularity. For details, please refer to the description of the group-by-group quantization, which will not be repeated here.

[0228] Optionally, the above inverse quantization parameter may be scale.

[0229] For example, the above-mentioned inverse quantization process may include: the first data performs a numerical format conversion on the fourth data, for example, when the numerical format of the fourth data is FP8 and the numerical format of the third data is FP16, the above-mentioned numerical format is converted from FP8 to FP16. The data obtained by the numerical format conversion of the fourth data is then divided by scale to obtain the fifth data.

[0230] The following is an example of the reverse calculation process in this application:

[0231] Exemplarily, the first device performs reverse calculation on the output layer, and the reverse calculation calculates the gradient by the chain rule. Assuming that the loss function is the mean square error (MSE), assuming that the reverse calculation of the i-th layer neural network is being performed, the loss function E can be expressed as:

[0232]

[0233] Among them, the output layer has n neurons, E is the loss function, y k is the target value of the kth neuron in the output layer, is the fifth data. Optionally, the fourth data is obtained by quantizing the activation tensor obtained by forward calculation of the output layer, and the fifth data is obtained by dequantizing the fourth data. Assume that the above reverse calculation is applied to the aforementioned Figure 5 The output layer of the neural network shown in , then n = 2, for and Quantize the fourth data to obtain the fourth data, and quantize the fourth data to obtain the fifth data, that is.

[0234] Exemplarily, after the loss function E is calculated, the gradient is calculated based on the loss function using the back propagation algorithm. Then the gradient descent algorithm or other optimization algorithm is used to update the parameters of the neural network of the first model so that the performance of the neural network gradually reaches the optimal level, and the training process of the first model is completed.

[0235] To help understand, the following Figure 8 The method 100 in the present application is introduced as an example, wherein: Figure 8The FWD in the figure represents forward calculation, and the BWD represents reverse calculation. The A data, i.e., the activation tensor, is obtained through the forward calculation of the i+1th layer of the first model. The A data is quantized, and it is assumed that the quantized data is the B data. Figure 8 In the example, the A data is quantized to FP8-E5M2, and then the B data is saved. The B data is obtained and saved when the reverse calculation of the i-th layer neural network is performed. The B data is dequantized, and it is assumed that the quantized data is C data. The C data can be used for the reverse calculation of the i-th layer neural network to calculate the gradient of the weight, that is, dW.

[0236] Optionally, the data obtained by the reverse calculation may be dequantized. For example, the reverse calculated data may be dequantized by an unscaling technique. For example, if the data format of the dequantized data is FP16, it may be dequantized into a mixed format of FP16 and FP32, or into a format of FP32.

[0237] After training the first model based on method 100 and related implementations or examples, a target model / rule is obtained, and the target model / rule can be used to perform a specific task. For example, natural language processing, image processing, etc. Optionally, the target model / rule can be run in the first device or configured in other devices.

[0238] From the above introduction, it can be seen that this application has the following technical effects:

[0239] First of all, many chip computing units currently do not support low-precision data formats, that is, when such chips calculate data, the data format can only be a high-precision format, such as FP16, FP32 and other formats. Due to the forward calculation and reverse calculation and other calculation processes in this application, the number of bits of data can be a data format supported by the chip, for example, it can be a high-precision format such as FP16, FP32. This is because the process of quantizing the data in this application is after the forward calculation, and the data is dequantized before the reverse calculation, so the data in the forward calculation and reverse calculation processes can be in a high-precision format. It can be seen that the scope of application of this application is relatively wide. This application can be applied to modules or devices that do not support low-precision formats in the configured chip computing unit, and can also be applied to modules or devices that support low-precision formats in the configured chip computing unit. The present application can be used for the model training process of modules or devices that do not support low-precision formats in the configured chip computing unit. Before the module or device saves part of the data, the data is quantized, which can save the storage space of such modules or devices, thereby improving the performance of model training.

[0240] Furthermore, before the activation tensors and other data obtained by the forward calculation are saved, the present application quantifies such data, which can reduce the dynamic storage space occupied by such data, thereby saving the dynamic memory of the model training scenario. After measurement, the use of this solution during model training can save 90% of dynamic memory. Because the present application saves dynamic memory, the present application can increase the micro-batch size (mbs) while keeping the global batch size (GBS) unchanged, improve the chip MAC (such as multiplication and accumulation) utilization (such as computing power utilization), and improve training throughput. And compared with the recalculation solution, the present application can reduce the recalculation overhead; in addition, it reduces the tensor parallelism (TP) domain of distributed training, the pipeline parallelism PP (PP) domain, etc., and reduces the collective communication overhead.

[0241] In addition, in the present application, when saving data, i.e., the activation tensor and other data obtained by the previous calculation, the present application adopts an asynchronous acquisition method, that is, when performing reverse calculation on the current layer of neural network, the data of the next layer or the next two layers of neural network are asynchronously acquired and dequantized, so that when performing reverse calculation on the next layer or the next two layers of neural network, the data format of these data has been adjusted to a data format that can be supported by the chip computing unit, so that these data can be used directly, thereby covering up the delay caused by acquiring the saved data and dequantizing the data. And in this scheme, the reverse calculation and, and the acquisition and dequantization of the saved data of the next layer or the next two layers of neural network are performed synchronously, which can further reduce the time consumed by the reverse calculation process of training, thereby shortening the time required for model training and improving the performance of model training.

[0242] Please refer to Fig. 9 , Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, an embodiment of the present application provides a data processing device 900, wherein the device 900 includes:

[0243] The processing module 901 is used to perform forward calculation on the first data through a first model to obtain second data, the first model is a neural network model, and the number of bits of the first data and the number of bits of the second data are a first value; quantize the second data to obtain third data, the number of bits of the third data is a second value, and the second value is less than the first value;

[0244] The saving module 902 is used to save the third data, and the saved third data is used for reverse calculation of the first model.

[0245] In a possible implementation, the second data includes: an activation tensor or a feature tensor.

[0246] In a possible implementation, the saving module 902 is specifically used to: save the third data and a first scaling factor, where the first scaling factor is a scaling factor when quantizing the second data, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

[0247] In a possible implementation, the device further includes: an acquisition module ( Fig. 9 (not shown), the acquisition module is used to: obtain the fourth data from the saved third data; the processing module 901 is also used to: perform a dequantization operation on the fourth data to obtain fifth data, the number of bits of the fifth data is the first value; and reversely calculate the fifth data through the first model.

[0248] In a possible implementation, the acquisition module is specifically used to: before the reverse calculation of the i-th layer neural network of the first model is completed, obtain the fourth data, the fourth data is obtained by quantizing the sixth data in the second data, and the sixth data is obtained by forward calculation through the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; the processing module 901 is specifically used to: reversely calculate the fifth data through the i+k-th layer neural network.

[0249] In a possible implementation, the fourth data is obtained when the i-th layer neural network performs reverse calculation.

[0250] In one possible implementation, k is 1.

[0251] In a possible implementation, the data format of the third data is 8-bit floating point number FP8.

[0252] Next, a terminal device provided by an embodiment of the present application is introduced. Fig.10 , Fig.10 This is a schematic diagram of a terminal device provided in an embodiment of the present application. The terminal device 1000 can be a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Specifically, the terminal device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003 and a memory 1004 (wherein the number of processors 1003 in the terminal device 1000 can be one or more, Fig.10In the example of FIG. 1003 , the processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of the present application, the receiver 1001, the transmitter 1002, the processor 1003 and the memory 1004 may be connected via a bus or other means.

[0253] The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1003. A portion of the memory 1004 may also include a non-volatile random access memory (NVRAM). The memory 1004 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0254] The processor 1003 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0255] The method disclosed in the above embodiment of the present application can be applied to the processor 1003, or implemented by the processor 1003. The processor 1003 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1003. The above processor 1003 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include ASIC, FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1003 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1004, and the processor 1003 reads the information in the memory 1004 and combines its hardware to complete the steps of the above method involving the model training or model reasoning process.

[0256] The receiver 1001 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1002 can be used to output digital or character information through the first interface; the transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1002 can also include a display device such as a display screen.

[0257] The present application also provides a server. Fig.11 , Fig.11 This is a structural diagram of a server provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1112 (for example, one or more processors) and memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1112 can be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0258] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158; or, one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0259] In the embodiment of the present application, the central processing unit 1112 is used to execute actions related to model training or model reasoning in the above embodiments.

[0260] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0261] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0262] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0263] For details, please refer to Fig.12 , Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1200, NPU 1200 is mounted on the main CPU (host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0264] In some implementations, the operation circuit 1203 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1203 is a two-dimensional systolic array. The operation circuit 1203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1203 is a general-purpose matrix processor.

[0265] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1201 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1208.

[0266] The unified memory 1206 is used to store input data and output data. The weight data is directly transferred to the weight memory 1202 through the direct memory access controller (DMAC) 1205. The input data is also transferred to the unified memory 1206 through the DMAC.

[0267] The bus interface unit (BIU) 1210 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1209 .

[0268] The bus interface unit 1210 is also used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0269] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or to transfer weight data to the weight memory 1202 or to transfer input data to the input memory 1201.

[0270] The vector calculation unit 1207 includes multiple operation processing units, and further processes the output of the operation circuit 1203 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0271] In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector calculation unit 1207 can apply a linear function; or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, for example, for use in a subsequent layer in a neural network.

[0272] An instruction fetch buffer 1209 connected to the controller 1204 is used to store instructions used by the controller 1204;

[0273] Unified memory 1206, input memory 1201, weight memory 1202 and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0274] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

[0275] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0276] Through the description of the above implementation, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation method in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0277] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0278] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0279] Through the description of the above implementation method, technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working processes of the systems, devices, units, and computer program products described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0280] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited to this. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Performing forward calculation on first data through a first model to obtain second data, wherein the first model is a neural network model, and the number of bits of the first data and the number of bits of the second data are first values; quantizing the second data to obtain third data, where the number of bits of the third data is a second value, and the second value is smaller than the first value; The third data is saved, and the saved third data is used for reverse calculation of the first model.

2. The data processing method according to claim 1, characterized in that: The second data includes: an activation tensor or a feature tensor.

3. The data processing method according to claim 1 or 2, characterized in that: The storing of the third data comprises: The third data and a first scaling factor are saved, where the first scaling factor is a scaling factor when the second data is quantized, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

4. The data processing method according to any one of claims 1 to 3, characterized in that: After saving the third data, the method further includes: Acquire fourth data from the saved third data; Performing a dequantization operation on the fourth data to obtain fifth data, where the number of bits of the fifth data is the first value; The fifth data is reversely calculated using the first model.

5. The data processing method according to claim 4, characterized in that: The acquiring and saving fourth data from the third data includes: Before the reverse calculation of the i-th layer neural network of the first model is completed, the fourth data is obtained, where the fourth data is obtained by quantizing the sixth data in the second data, and the sixth data is obtained by forward calculation of the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; The reverse calculation of the fifth data by using the first model includes: The fifth data is reversely calculated through the i+k-th layer neural network.

6. The data processing method according to claim 5, characterized in that: The fourth data is obtained when the i-th layer neural network performs reverse calculation.

7. The data processing method according to claim 5 or 6, characterized in that: The k is 1.

8. The data processing method according to any one of claims 1 to 7, characterized in that: The data format of the third data is 8-bit floating point number FP8.

9. A data processing device, characterized in that: The device comprises: a processing module, configured to perform forward calculation on first data through a first model to obtain second data, wherein the first model is a neural network model, and the number of bits of the first data and the number of bits of the second data are first values; and quantize the second data to obtain third data, wherein the number of bits of the third data is a second value, and the second value is smaller than the first value; A saving module is used to save the third data, and the saved third data is used for reverse calculation of the first model.

10. The device according to claim 9, characterized in that The second data includes: an activation tensor or a feature tensor.

11. The device according to claim 9 or 10, characterized in that The storage module is specifically used for: The third data and a first scaling factor are saved, where the first scaling factor is a scaling factor when the second data is quantized, and one of the first scaling factors is associated with m data elements in the third data, where m is an arbitrary positive integer.

12. The device according to any one of claims 9 to 11, characterized in that The device further includes: an acquisition module, the acquisition module being used to: acquire fourth data from the stored third data; The processing module is also used for: Performing a dequantization operation on the fourth data to obtain fifth data, where the number of bits of the fifth data is the first value; The fifth data is reversely calculated using the first model.

13. The device according to claim 12, characterized in that The acquisition module is specifically used to: before the reverse calculation of the i-th layer neural network of the first model is completed, acquire the fourth data, the fourth data is obtained by quantizing the sixth data in the second data, and the sixth data is obtained by forward calculation of the i+k-th layer neural network of the first model, where i is a positive integer and k is a positive integer; The processing module is specifically used to: perform reverse calculation on the fifth data through the i+kth layer neural network.

14. The device according to claim 13, characterized in that The fourth data is obtained when the i-th layer neural network performs reverse calculation.

15. The device according to claim 13 or 14, characterized in that The k is 1.

16. The device according to any one of claims 9 to 15, characterized in that The data format of the third data is 8-bit floating point number FP8.

17. A data processing device, characterized in that: The device comprises: a processor and a memory; the memory is used to store a program; the processor is used to execute the program in the memory so as to execute the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: The medium stores instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

19. A computer program product, characterized in that The method comprises instructions, and when the instructions are executed on a processor, the method according to any one of claims 1 to 8 is executed.

20. A chip, characterized in that: The method comprises at least one processing unit and an interface circuit, wherein the interface circuit is used to provide program instructions or data to the at least one processing unit, and the at least one processing unit is used to execute the program instructions to implement the method according to any one of claims 1 to 8.

Images