Data quantification method and device

Abstract

The invention provides a data quantization method and device in the field of artificial intelligence, which can quantify data needing to be transmitted in a parallel training scene, do not need to transmit a quantization scaler during data transmission, and reduce the communication overhead in data transmission. The method can be applied to a distributed system, a plurality of devices in the system are used for training a first model by using training data, in one time of training, the first device receives at least one kind of quantized data, and one kind of quantized data can be obtained after the first model is trained in a second device. Quantizing the data of the trained first model to obtain the first model; the first device performs aggregation based on the received at least one kind of quantized data to obtain aggregated data; the first device inversely quantizes the aggregated data according to the locally stored data of the first model to obtain inversely quantized data, and at the moment, a quantization scaler does not need to be received; and then the first device updates the first model according to the inverse quantization data to obtain an updated first model.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data quantization method and device. Background Art

[0002] In model training, data parallelism is a common training strategy. For example, the training data can be divided into multiple batches and trained in parallel on multiple computing devices. In the data parallel process, each computing device has a complete copy of the model and executes the stochastic gradient descent (SGD) algorithm on each batch of data to obtain local gradients. In each iteration, the computing device aggregates all local gradients to obtain the global gradient and uses the global gradient to update the model.

[0003] When transmitting updated model data, communication transmission between devices is usually required. The larger the model scale, the more data needs to be transmitted. For example, in some scenarios, the gradient can be quantized. Specifically, the gradient is discretized and represented as a smaller number of bits. At the same time, the gradient quantization scaler is calculated, and the quantized gradient and the quantization scaler are sent to the receiver at the same time. The receiver restores the gradient based on the received quantization codeword and quantization scaler. However, each quantization scaler usually requires a higher bit transmission, such as 32-bit transmission. The communication overhead required for transmission is large, and the contradiction between model performance and communication overhead cannot be resolved. And the coarser the quantization scaler, the more serious the loss of model performance. The finer the quantization scaler, the more additional communication overhead is required. In this way, how to reduce the amount of data transmitted between devices has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data quantization method and device, which can quantize the data that needs to be transmitted in a parallel training scenario. When transmitting data, there is no need to transmit a quantization scaler, thereby reducing the communication overhead in data transmission.

[0005] In view of this, the present application provides a data quantization method, which is applied to a distributed system. The distributed system includes multiple devices, and the multiple devices are used to train a first model using training data respectively. Taking one of the training processes performed by one of the multiple devices as an example, for the convenience of distinction, it is called the first device, and the at least one second device mentioned below is a device different from the first device in the distributed system. The method includes: first, the first device receives at least one quantized data, and the at least one quantized data is obtained by at least one second device among the aforementioned multiple devices respectively quantizing the data of the first model after their respective local training, and the at least one quantized data corresponds to at least one second device, that is, one of the multiple devices The quantized data can be obtained by quantizing the data of the trained first model (such as the weight parameters or gradients of the trained first model) after one of the second devices trains its local first model; then, the first device aggregates based on at least one received quantized data to obtain aggregated data; then, the first device dequantizes the aggregated data according to the relevant data of the first model stored locally to obtain dequantized data, where the relevant data of the first model is the data stored locally by the first device, such as the data obtained by the first device in updating the first model in the previous one or more iterations; then, the first device updates the first model according to the dequantized data to obtain an updated first model.

[0006] In the implementation mode of the present application, during the parallel training process, each training device can quantize the data of the trained model and transmit the quantized data to the corresponding receiving end, thereby reducing the communication overhead of the data transmission by quantization. The receiving end can use the relevant data of the locally stored model to dequantize the received data without receiving the quantization scaler, so there is no need to transmit the quantization scaler between devices, thereby reducing the communication overhead in the parallel training scenario. And the data stored locally on the device is used for dequantization, and the quantization loss is not affected by the bits of data transmission, which can reduce the data loss of dequantization and achieve more accurate dequantization.

[0007] In a possible implementation, the aforementioned first device dequantizes the aggregate data according to the relevant data of the first model stored locally to obtain dequantized data, which may include: the first device calculates a quantization scaler according to the relevant data of the first model, and the quantization scaler represents the range of quantization bits of at least one quantized data; the first device dequantizes the aggregate data according to the quantization scaler to obtain dequantized data.

[0008] In the implementation manner of the present application, the receiving end can use the locally stored data to calculate the quantization scaler required for dequantization, and use the quantization scaler calculated using the local data to perform dequantization, so there is no need to transmit the quantization scaler between devices, thereby reducing the communication overhead in the parallel training scenario. And when the data stored locally on the device is used for dequantization, the quantization loss is not affected by the bits of data transmission, which can reduce the data loss of dequantization and achieve more accurate dequantization.

[0009] In a possible implementation, the relevant data of the first model stored locally by the aforementioned first device may specifically include: optimizer data or parameters of the first model stored locally by the first device, the optimizer is the data used when updating the first model, the optimizer may include data used or generated in the process of updating the model, such as statistical parameters of the gradient or parameters of the loss function, the parameters of the first model such as model weight parameters, and the specific calculation method may include calculating the second-order momentum of the gradient included in the optimizer data or calculating the quantization scaler based on the model weight and a pre-set scaling factor.

[0010] Therefore, in the implementation of the present application, the quantization scaler can be calculated based on the data of the model itself or the data generated in the process of updating the model, so that the quantization scaler does not need to be transmitted, reducing the communication overhead. And the data stored locally on the device is used for dequantization, and the quantization loss is not affected by the bits of data transmission, which can reduce the data loss of dequantization and achieve more accurate dequantization.

[0011] In a possible implementation, the process of model training may be an iterative training process. In order to improve the calculation accuracy of the quantization scaler, the data for calculating the quantization scaler stored locally in each device may be updated after each preset number of iterations. For example, after each preset number of iterations, unquantized gradients may be transmitted between devices so that dequantization may be performed based on accurate data stored locally, thereby improving the accuracy of the dequantization result.

[0012] For example, in a round of iteration, the first device can receive unquantized data of at least one first model, and the first device can use the received unquantized model data to update relevant data of the locally stored model, such as updating the state data of the optimizer, or directly use the gradient to update the model to obtain new relevant data of the first model, etc., so that in subsequent iterations, more accurate local data can be used to calculate the quantization scaler, thereby improving the accuracy of the calculated quantization scaler.

[0013] In a possible implementation, each second device may train the locally stored first model to obtain a gradient, and the data of the first model mentioned above may include a gradient, that is, the data of the trained first model locally stored by each second device may include a gradient, and the at least one quantization data mentioned above may be obtained by quantizing the local gradients of at least one second device respectively, that is, the quantization data mentioned above may be obtained by quantizing the local gradients of a second device, and the dequantization data mentioned above may be used to represent the value after the gradients of one or more second devices are aggregated. When the first device updates the first model, the aggregated value can be used as the gradient to update the first model to obtain the updated first model.

[0014] In the implementation of the present application, in the scenario of parallel training, quantized gradients can be transmitted between devices, so that the device performing aggregation can update the model based on the values ​​after the gradients of each device are aggregated. Therefore, only a small amount of gradients need to be transmitted to achieve parallel training of the model, which only requires low communication overhead.

[0015] In a possible implementation, the aforementioned first device aggregating based on at least one quantitative data to obtain aggregated data may include: the first device aggregating data based on at least one quantitative data and the first device training a first model to obtain aggregated data.

[0016] In the implementation manner of the present application, when the first device performs data aggregation, it can also aggregate the locally stored model training data, that is, the device performing data aggregation itself can also perform model training, thereby making full use of available resources and improving model training efficiency.

[0017] In a possible implementation, the aforementioned first model is divided into multiple regions, multiple devices respectively update the parameters of the multiple regions, and the first device updates the first model according to the inverse quantization data to obtain the updated first model. It can also include: the first device updates the parameters of the first region according to the inverse quantization data to obtain the updated first model, and the first region is one of the multiple regions. Therefore, in the implementation of the present application, each device can update the model in a partitioned manner, so that each model can locally update the data of the region that needs to be updated, and can make full use of the training resources of each device and balance the load of each device.

[0018] In a second aspect, the present application provides a data quantization device, which is applied to a distributed system, wherein the distributed system includes multiple devices, and the multiple devices are used to train a first model using training data respectively. The data quantization device is one of the multiple devices, and the second device mentioned below is a device different from the data quantization device in the multiple devices, and the device includes:

[0019] A transceiver module, configured to receive at least one type of quantized data, the at least one type of quantized data corresponding to at least one second device, wherein the at least one type of quantized data is obtained by quantizing data of a trained first model after one of the second devices trains the first model;

[0020] An aggregation module, used for performing aggregation based on at least one quantitative data to obtain aggregated data;

[0021] A dequantization module, used for dequantizing the aggregated data according to the relevant data of the first model stored locally to obtain dequantized data;

[0022] The updating module is used to update the first model according to the dequantized data to obtain an updated first model.

[0023] In one possible implementation, the inverse quantization module is specifically used to: calculate a quantization scaler based on relevant data of the first model stored locally, where the quantization scaler represents a range of quantization bits of at least one quantized data; and inverse quantize the aggregated data based on the quantization scaler to obtain inverse quantized data.

[0024] In a possible implementation, the relevant data of the first model includes: optimizer data or parameters of the first model stored locally by the first device, the optimizer data includes statistical parameters of the gradient or parameters in the loss function, and the parameters of the first model include weight parameters of the first model.

[0025] In a possible implementation, the data of the aforementioned first model may include gradients, and the aforementioned quantization data may be obtained by quantizing the local gradients by the second device, and the local gradients of each second device are obtained after each second device trains the locally stored first model, and the aforementioned dequantization data may be used to represent the value after aggregating the gradients of one or more second devices.

[0026] In a possible implementation, the aggregation module is specifically configured to aggregate data based on at least one quantized data and data obtained by locally training the first model to obtain aggregated data.

[0027] In a possible implementation, the first model is divided into multiple regions, and multiple devices update parameters of the multiple regions respectively.

[0028] The updating module is specifically used to update the parameters of the first region according to the inverse quantization data to obtain an updated first model, where the first region is one of the multiple regions.

[0029] In a possible implementation, the plurality of devices respectively iterate the first model multiple times using the training data, and the method further includes:

[0030] In one of the iterations, the transceiver module is further used to receive at least one type of training data, where the training data is unquantized data;

[0031] The dequantization module is further used to update relevant data of the first model according to at least one training data, and the updated relevant data of the first model is used to dequantize the quantized data in a subsequent iterative training process.

[0032] In a third aspect, an embodiment of the present application provides a data quantization device, which has the function of implementing the data quantization method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0033] In a fourth aspect, an embodiment of the present application provides a data quantization device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the data quantization method shown in any one of the first aspects above. Optionally, the data quantization device can be a chip.

[0034] In a fifth aspect, an embodiment of the present application provides a distributed system, which may include multiple devices, and the multiple devices are respectively used to train their respective local first models using training data, wherein the multiple devices can be used to execute the steps in the data quantization method shown in any one of the above-mentioned first aspects.

[0035] In the sixth aspect, an embodiment of the present application provides a digital processing chip or chip, the chip includes a processing unit and a communication interface, the processing unit obtains program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0036] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or any optional implementation of the first aspect.

[0037] In an eighth aspect, an embodiment of the present application provides a computer program product comprising a computer program / instructions, which, when executed by a processor, enables the processor to execute the method in the above-mentioned first aspect or any optional implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the architecture of a distributed system provided for this application;

[0039] Figure 2 A schematic diagram of another distributed system architecture provided for this application;

[0040] Figure 3 A schematic diagram of another distributed system architecture provided for this application;

[0041] Figure 4 A flow chart of a data quantification method provided in this application;

[0042] Figure 5 A schematic diagram of another data quantification method provided for this application;

[0043] Figure 6 A schematic diagram of another data quantification method provided for this application;

[0044] Figure 7 A flowchart of a data aggregation method provided for this application;

[0045] Figure 8 A schematic diagram of the structure of a data quantization device provided in this application;

[0046] Fig. 9 A schematic diagram of the structure of another data quantization device provided by the present application;

[0047] Fig.10 A schematic diagram of a chip structure provided for this application. DETAILED DESCRIPTION

[0048] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0049] The recommendation method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0050] First, the overall workflow of the artificial intelligence system is described. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain") and "IT value chain". Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensed process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

[0051] (1) Infrastructure

[0052] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes the distributed computing framework (or distributed system) and network and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0053] (2) Data

[0054] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0055] (3) Data processing

[0056] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0057] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0058] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0059] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0060] (4) General ability

[0061] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0062] (5) Smart products and industry applications

[0063] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0064] The embodiments of the present application involve related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0065] (1) Neural Network

[0066] A neural network may be composed of neural units. A neural unit may refer to an operation unit with xs and intercept 1 as input. The output of the operation unit may be as shown in formula (1-1):

[0067]

[0068] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0069] (2) Deep Neural Networks

[0070] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0071] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0072] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0073] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0074] (3) Convolutional Neural Network

[0075] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract features is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0076] (4) Graph Convolutional Network (GCN)

[0077] Graph neural network is a deep learning model that models and processes non-Euclidean spatial data (such as graph data). Its principle is to use pairwise message passing so that graph nodes iteratively update their corresponding representations by exchanging information with their neighbors.

[0078] GCN is similar to CNN, but the difference is that the input of CNN is usually two-dimensional structured data, while the input of GCN is usually graph structured data. GCN has cleverly designed a method to extract features from graph data, so that these features can be used to perform node classification, graph classification, link prediction, and graph embedding.

[0079] (5) Loss Function

[0080] In the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as squared error, cross entropy, logarithm, exponential, etc. For example, the squared error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0081] (6) Back propagation algorithm

[0082] An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters. The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0083] (7) Optimizer (also called optimization algorithm or optimization model, etc.)

[0084] During the model training process, the loss function can be used to measure the difference between the model's output value and the expected value, and the model can be updated in the opposite direction using this difference. When the loss value (i.e., the output value of the loss function) decreases during training, it indicates that the model optimization is positive optimization. Optimization is used to describe the reduction of loss value, and the optimizer is the data that reduces the loss value during the model training process.

[0085] The optimizer data mentioned in this application may specifically include data used or generated in the process of training the model, such as model parameters in the last iterative training process, statistical parameters of the gradient, parameters in the update function, or dynamic statistical parameters for maintaining the gradient.

[0086] (8) Quantification

[0087] It is a compression method that converts high bits into low bits. For example, the compression of converting regular 32-bit floating-point operations into low-bit integer operations is called quantization. For example, when the low bit is quantized to 8 bits, it can be called int8 quantization, that is, a weight originally needs to be represented by float32, but after quantization, it only needs to be represented by int8. In theory, it can achieve 4 times network acceleration. At the same time, 8 bits can reduce 4 times the storage space compared to 32 bits, reducing storage space and operation time, thereby achieving the purpose of compressing the model and accelerating.

[0088] (9) Federal training

[0089] Federated training (or federated learning) is a distributed machine learning technology. It is a distributed machine learning paradigm in which multiple parties use their own data to collaboratively train artificial intelligence models without aggregating data from multiple parties. The traditional machine learning paradigm requires the aggregation of large amounts of data for model training, and the data used for training is likely to come from multiple different organizations or users or participants or clients. Bringing together data from multiple parties is very likely to cause the risk of data leakage, which will expose information assets to organizations and may leak personal privacy to individual users. Federated training allows multi-party data to be retained locally without multi-party data aggregation, and multiple parties can jointly train AI models by collaboratively computing (securely) interacting with intermediate calculation results. Through federated learning technology, the business data of multiple parties is protected, and the data of multiple parties can be fully utilized to collaboratively train models, thereby obtaining a more powerful model. The communication system provided in this application can be used as Figure 1 As shown, the communication system may also be referred to as a federated learning system, a federated learning network, or a communication network, etc. In the present application, the federated learning process is exemplarily divided into steps executed by the client and the server. The client uses local data to train the model, and the server can aggregate and fuse the data trained by multiple clients to obtain a federated model.

[0090] (10) Data parallelism (DP) training

[0091] Data parallelism is a way to accelerate model training. The training data is divided into multiple batches and calculated in parallel on multiple devices (GPU, NPU, CPU, and federated training is also theoretically feasible). Each device has a complete copy of the model, and the gradients are calculated separately and then the models are updated synchronously.

[0092] (11) Gradient

[0093] That is, the derivative vector of the loss function with respect to the parameters. It can be used to measure the change in the parameters of the model.

[0094] (12) Gradient Descent

[0095] Using calculus, gradient descent iteratively adjusts parameter values ​​to minimize the output value of a provided loss function after defining initial parameter values.

[0096] (13) Stochastic Gradient

[0097] The number of samples in machine learning is very large, so the loss function is calculated each time based on randomly sampled data, and the corresponding gradient is called stochastic gradient.

[0098] (14) Stochastic gradient descent (SGD)

[0099] By loading the entire dataset of n points at once to calculate the derivative of the loss function, SGD does not use the entire dataset in each iteration, but calculates the derivative by randomly selecting a small number of samples, thereby reducing the computational intensity.

[0100] (15) Gradient polymerization

[0101] In data parallel training, the gradients computed on each device are summed and synchronized across all devices.

[0102] (16) Gradient Quantization

[0103] Gradients are usually stored as 32-bit or 16-bit floating point numbers. In order to save communication overhead, the gradients are quantized to a lower number of bits.

[0104] (17) Quantization scaler:

[0105] The quantization scaler is used to determine the quantization range when quantization is performed, such as the range of quantization codeword values ​​or the range of quantization bits, etc. The receiving end can perform inverse quantization based on the quantization scaler. Taking gradient quantization as an example, the quantization scaler can determine the quantization range of the gradient (value or value range, such as the maximum value of the quantization codeword). After the gradient is quantized, the receiving end restores the gradient according to the quantization codeword and the scaler.

[0106] (18) Pre-trained language model (PLM)

[0107] It is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. PLM training consists of two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, the model is trained on language model tasks on large-scale unsupervised text to learn word representations. In the fine-tuning stage, the model is initialized using the parameters learned in the pre-training stage, and is trained on downstream tasks such as text classification or sequence labeling with fewer steps, so that the semantic information obtained from pre-training can be successfully transferred to downstream tasks.

[0108] (19)Epoch

[0109] Defines the number of times the learning algorithm works on the entire training set. An epoch can be considered the number of times the neural network is trained using the entire training set.

[0110] (20) batch;

[0111] Closely related to the definition of epoch, an epoch involves training the neural network using the entire dataset, and a batch represents one of the batches of data in an epoch, specifically expressed as batch_size*batchs=epoch. It can be understood that each epoch is divided into one or more batches, and each batch can use part of the data in the training set to train the neural network.

[0112] Among them, data parallelism is a commonly used training strategy in the model training process. For example, data parallelism accelerates the training process by dividing the training data into multiple batches and processing them in parallel on multiple computing devices. In the data parallel process, each computing device has a complete copy of the model and executes the stochastic gradient descent (SGD) algorithm on each batch of data to obtain local gradients. In each iteration, the computing device aggregates all local gradients to obtain the global gradient and uses the global gradient to update the model. When the number of nodes is large and the degree of data parallelism is large, the communication ratio of gradient aggregation is very high.

[0113] Gradient compression is a technology used in deep learning training. Gradients are usually stored as 32-bit or 16-bit floating-point numbers. In order to save communication overhead, compressing the gradient to a lower number of bits can save the communication overhead of gradient aggregation. However, gradient quantization will reduce the accuracy of model training. How to reduce the communication overhead of gradient aggregation while ensuring training accuracy is the difficulty of gradient quantization technology.

[0114] In some schemes, gradient quantization can be performed to discretize the gradient and represent it as a smaller number of bits (such as 2 bits or 4 bits) to reduce computational and communication costs. The specific process may include: calculating a gradient quantization scaler. In general, the maximum value of the absolute value of the gradient can be selected as the quantization scaler. The discrete quantization value is determined according to the quantization scaler and the number of quantization bits, the gradient is quantized to the quantization value closest to its original value, and the quantization codeword is represented by bits. Sending quantized gradient: The device sends the codeword and quantization scaler of the quantization gradient, and the quantization scaler is represented by a 32-bit floating point number. Dequantizing gradient: The receiving end restores the gradient according to the received quantization codeword and quantization scaler. However, the gradient of each parameter of the model is usually not independent and identically distributed, which will cause the coarse-grained quantization scaler to not match the fluctuation range of certain gradient values. For example, the fluctuation range of a gradient value is very small, but its quantization scaler is very large, resulting in a large restored gradient value, causing a large relative error, which may cause the model to not converge. Gradient quantization schemes all require additional transmission of quantization scalers. Each scaler in a fine-grained quantization scaler usually requires 32-bit transmission, which cannot solve the contradiction between model performance and communication overhead. The coarser the quantization scaler, the more serious the loss of model performance. The finer the quantization scaler, the more additional communication overhead is required.

[0115] Therefore, the present application provides a data quantization method, which can quantize the data that needs to be transmitted in a parallel training scenario and reduce the communication overhead in data transmission.

[0116] First, the system architecture used by the method provided in this application can be called a distributed system, and the distributed system can include multiple devices, each of which is used for model training. That is, the method provided in this application can be applied to scenarios where multiple devices are trained in parallel, such as federated training or data parallel training.

[0117] For example, Figure 1 As shown, taking federated training as an example, the present application provides a schematic diagram of the architecture of a federated learning system (i.e., a distributed system). The system (or cluster for short) may include multiple servers, which can establish connections with each other, i.e., the servers can communicate with each other. Each server can communicate with one or more clients, and the clients can be deployed in various devices, such as mobile terminals or servers, etc. Figure 1 The client 1, client 2, ..., client N-1 and client N etc. are shown in FIG.

[0118] Specifically, the servers or the servers and the clients can interact through any communication mechanism / communication standard communication network, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes, but is not limited to, a fifth generation mobile communication technology (5th-Generation, 5G) system, a long term evolution (long term evolution, LTE) system, a global system for mobile communication (global system for mobilecommunication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wirelessfidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near fieldcommunication, NFC) in any one or more combinations. The wired network may include a fiber optic communication network or a network composed of coaxial cables, etc.

[0119] Generally, the client can be deployed in various servers or terminals. The client mentioned below may also refer to a server or terminal on which the client software program is deployed. The terminal may include a mobile terminal or a fixedly installed terminal, etc. For example, the terminal may specifically include a mobile phone, a tablet, a personal computer (PC), a smart bracelet, a speaker, a television, a smart watch or other terminals, etc.

[0120] When conducting federated learning, each server can send the model to be trained to the client that has established a connection with it. The client can train the model using the training samples stored locally and feed back the data of the trained model to the server. After receiving the trained model or models fed back by one or more clients, the server can aggregate the data of the received one or more models to obtain the aggregated data, which is equivalent to the aggregated model. After the stop condition is met, the final model can be output to complete the federated learning.

[0121] For example, Figure 2 As shown, based on a finer-grained distributed system architecture. The distributed system may include one or more CPUs ( Figure 2 Only one CPU), multiple GPUs, memory and other components are shown in the figure. The CPU and GPU are connected via PCIe (PCIe), and the GPUs are connected via NCCL (GPU Communication Library). Data parallel model training is a major application scenario, which is used to accelerate the training speed of the model. Among them, the GPU can also be replaced by NPU or other processors. Here, only the GPU is used as an example for exemplary introduction.

[0122] Take data parallelism with a data parallelism degree of 4 as an example. In the initialization phase, the CPU divides the training data into 4 minibatches and stores the 4 minibatches in the video memory of 4 GPUs. The model parameters are copied four times and stored in the video memory of 4 GPUs. In the model training phase, each GPU calculates the gradient using the SGD algorithm; then GPU 0 collects the gradients on the 4 GPUs and calculates the mean of the gradients; finally, GPU 0 broadcasts the mean of the gradients to each GPU and updates the model parameters. After multiple rounds of iterations, the model converges.

[0123] For example, the specific steps performed by each GPU can be found in Figure 3 . Take 4 GPUs as an example, Figure 3 As shown in the figure, GPU0, GPU1, GPU2 and GPU3, exemplarily, GPU1, GPU2 and GPU3 can respectively perform gradient calculation, that is, use the training data as the input of the model, calculate the loss value according to the output of the model, and calculate the parameter gradient of the model based on the loss value, and then quantize the gradient respectively, and send the quantized gradient to GPU0 through NCCL communication. GPU0 performs scaler calculation based on the locally maintained data, that is, calculates the quantization value range, and then dequantizes (or dequantizes) the received data based on the quantization value range, and updates the model after obtaining the aggregated gradient.

[0124] The method flow provided by this application is introduced in detail below.

[0125] It should be noted that the method provided in the present application can be deployed in a training architecture of a distributed system, such as in a federated learning or data parallel scenario, and the architecture may include multiple devices, each of which can perform model training separately. The following is an exemplary introduction using the steps performed by one of the devices as an example, that is, the first device mentioned below can be any one of the multiple devices, and the second device mentioned below is other than the first device among the multiple devices in the distributed system, that is, the first device and the second device mentioned below are different devices in the distributed system, and the number of the second devices is one or more.

[0126] See also Figure 4 , a flow chart of a data quantification method provided in this application is as follows.

[0127] 401. A first device receives at least one type of quantized data.

[0128] Specifically, the first device may receive quantized data sent by at least one second device, and the at least one quantized data may be obtained by the at least one second device training the first model stored locally, and then quantizing the data of the trained first model. One type of quantized data may include one of the second devices training its local first model, and then quantizing the data of the trained first model.

[0129] In the distributed system training architecture, multiple devices can perform model training separately, quantize the data related to the trained model, and then transmit it to the aggregated device (i.e., the first device) for aggregation and model update. For example, at least one second device can perform training on the first model separately, and quantize the data of the first model obtained after training, and then transmit it to the first device.

[0130] Optionally, when performing quantization, each second device may quantize the parameters of the updated model (such as weight parameters), or may quantize the gradients used to update the model parameters, that is, the data of the first model mentioned above may include gradients and parameters, which may be selected according to the actual application scenario, and this application does not limit this. This application exemplarily introduces the example of each second device quantizing the gradient calculated after training, and the gradients mentioned below may also be replaced by other data related to the training model, such as the weight parameters or structural parameters of the model.

[0131] 402. The first device performs aggregation based on at least one type of quantitative data to obtain aggregated data.

[0132] After the first device receives at least one type of quantized data, it may aggregate the at least one type of quantized data to obtain aggregated data.

[0133] In one possible scenario, the first device may also train the first model stored locally, and may also perform quantization after obtaining the gradient. In this scenario, the first device may aggregate the quantized data obtained by itself (i.e., the data obtained by the first device training the first model locally) with at least one received quantized data to obtain aggregated data. That is, the first device itself also participates in the training of the first model. In this scenario, when the first device aggregates the quantized data again, it may aggregate the data obtained by local training to obtain more comprehensive aggregated data.

[0134] 403. The first device dequantizes the aggregated data according to the locally stored relevant data of the first model to obtain dequantized data.

[0135] After the first device obtains the aggregated data, it can dequantize the aggregated data based on the locally stored relevant data of the first model to obtain dequantized data.

[0136] In one possible scenario, if the first device trains the first model stored locally, it is also possible not to perform quantization after obtaining the data of the first model of the parameter part that needs to be updated by the first device. In this scenario, when the first device aggregates (i.e., in step 402), it is not necessary to aggregate the data obtained by the first device training the first model locally. After step 403, the dequantized data obtained by dequantization and the data of the first model trained locally can be aggregated to obtain more comprehensive dequantized data after re-aggregation.

[0137] Optionally, the device can calculate a quantization scaler based on relevant data of the first model stored locally, where the quantization scaler represents the range of quantization bits corresponding to at least one of the aforementioned quantization data, and then perform inverse quantization based on the quantization scaler to restore the quantized data and obtain high-bit inverse quantized data.

[0138] Optionally, the data related to the first model stored locally by the first device may specifically include, but are not limited to: parameters of the first model or data of an optimizer, etc. The optimizer is an optimizer used when training the first model. The data of the optimizer may specifically include statistical parameters of gradients or parameters of loss functions, etc. The parameters of the first model may include weight parameters or structural parameters of the first model, etc. Therefore, the first device can calculate the quantization scaler based on the locally stored data, thereby performing inverse quantization, and the sending end does not need to send the quantization scaler, which greatly reduces the communication overhead.

[0139] 404. The first device updates the first model according to the dequantized data to obtain an updated first model.

[0140] After the first device obtains the dequantized data, it can use the dequantized data to update the parameters of the local first model to obtain an updated first model.

[0141] When quantizing the data of the trained first model, each second device can quantize the gradient of the trained first model saved locally. In this scenario, the dequantized data can represent the value after the gradient of each second device is aggregated, or called the aggregated gradient. The parameters of the local first model can be updated based on the aggregated value to obtain the updated first model.

[0142] Therefore, in the implementation of the present application, in the distributed training scenario, each device can only transmit quantization codewords, that is, quantized data, without transmitting the quantization scaler, and the device can use the locally stored relevant data of the first model to perform dequantization. This can reduce the data that needs to be transmitted between devices, reduce communication overhead, and reduce model training costs.

[0143] In a possible implementation, in a distributed training architecture, the aforementioned first model can be divided into multiple areas, such as partitioning according to the model structure or partitioning according to the parameter type of the model. Each device updates the parameters of at least one of the areas respectively. For example, the first device can update the parameters of the first area, and the quantized data it receives is the parameters of the first area calculated by other devices when training the first model. Therefore, the first device can aggregate and update the parameters of the first area obtained by training multiple devices, so that each device can update the model parameters, making the load of each device more balanced, thereby improving the model training efficiency.

[0144] The following is a further introduction to the method flow provided by this application in conjunction with a specific distributed architecture. The distributed architecture may include 4 GPUs, and the 4 GPUs here may also be replaced by other numbers or other devices, such as a CPU or NPU, etc. This application is only an exemplary introduction and is not intended to be limiting.

[0145] See also Figure 5 , a flow chart of another data quantification method provided in this application is as follows.

[0146] Among them, the method provided by this application can be divided into multiple stages, such as Figure 5 The model training, gradient quantization, data transmission, calculation of quantization scaler, dequantization, model update, etc. shown in are introduced below respectively.

[0147] 501. Model training.

[0148] Among them, multiple devices (including the aforementioned first device and second device) can perform model training separately. Each device can update the entire model and then transmit the updated parameters of the model, or after calculating the gradient of the model, transmit the gradient, thereby reducing the transmission amount. The following is an example of transmitting the gradient of the model.

[0149] For example, take data parallel training as an example, where the data parallelism is 4, that is, it is divided into 4 parallel processing channels, and the training set is divided into 4 subsets and assigned to 4 GPUs. During the model training process, the GPU can use the data in the subset as the model input, calculate the loss value of the loss function through the output of the model, and calculate the gradient of the model based on the loss value, or the gradient corresponding to each parameter in the model, which can be used to measure the change of each parameter in the model.

[0150] The training strategy for the model, i.e., the selection of the optimizer, can be determined based on the actual application scenario. For example, the optimizers that can be used include but are not limited to: Adam optimizer, stochastic gradient descent SGD, batch stochastic gradient descent, momentum optimizer, etc.

[0151] For example, the Adam optimizer can be used to update the model. The optimizer parallel algorithm is used, that is, the model parameters are sharded, each GPU is only responsible for the gradient aggregation and update of a part of the model, and the optimizer state corresponding to each device can be retained. For example, each device can maintain the locally saved optimizer state, which may include but is not limited to: the model parameters that need to be updated, the gradient statistical parameters, or the parameters in the update function, etc.

[0152] 502. Gradient quantization.

[0153] After each GPU calculates the gradient corresponding to the model parameters, each GPU can quantize its local gradient to obtain lower-bit quantized data to reduce communication overhead.

[0154] Typically, the same quantization algorithm can be used on each GPU so that the GPU that performs the aggregation can perform dequantization more efficiently.

[0155] Optionally, the specific quantization algorithm may adopt a TernGrad quantization algorithm (a ternary quantization algorithm) or a 1-bit algorithm, etc., which may be determined according to the actual application scenario, and the present application does not limit this.

[0156] For example, 1-bit quantization may be used, and only the positive and negative signs of the gradient are retained as quantization codewords.

[0157] For example, the TernGrad quantization algorithm can be used for quantization. TernGrad is a gradient compression method based on three-value quantization, which can compress the gradient value into three values: -1, 0, and 1. Each GPUi calculates the gradient g through the SGD algorithm i , the gradient is quantized into -1, 0 and 1 according to its size. The quantized gradient can be further reduced by lossless source coding, such as Huffman coding into 0, 1 bits.

[0158] 503. Data transmission.

[0159] After each GPU quantizes the gradient to obtain a quantization codeword, the required quantization codeword can be transmitted to each GPU.

[0160] Typically, each GPU can send the quantization codeword obtained by local quantization to all other GPUs, or send it to the GPU designated to perform gradient aggregation, or send different quantization codewords separately to GPUs that need to use the quantization codeword to update the model, etc. The appropriate data transmission method can be selected according to the actual application scenario.

[0161] 504. Calculate a quantized scaler.

[0162] Each GPU can use locally stored data to calculate a quantization scaler, which can represent the range of values ​​represented by the quantization codewords of each GPU when performing quantization, such as the quantized value, the range of the value, or the quantization maximum value.

[0163] Specifically, there are multiple ways to calculate the quantization scaler, such as calculating the quantization scaler based on the gradient second-order momentum or calculating the quantization scaler based on the weight parameters of the model, etc. The specific selection can be based on the actual application scenario, and this application does not limit this.

[0164] For example, the second-order momentum of the gradient can be used to calculate the gradient quantization scaler, which can be understood as using the second-order momentum of the gradient value obtained from the historical iteration to calculate the gradient quantization scaler of the current iteration. Generally, the second-order momentum of the gradient can be used to represent the variance of the dynamic statistics of the gradient, that is, the range of the quantization value corresponding to the gradient can be calculated based on the variance. For example, first, the second-order momentum of the gradient can accurately characterize the fluctuation range of the gradient amplitude, and the element-wise square root of the second-order momentum can be calculated as a quantization scaler, that is, v represents the gradient variance, the fluctuation range of the gradient. In addition, since the second-order momentum of the gradient changes continuously and smoothly, the second-order momentum of the gradient can be updated at a low frequency during the iterative training process of the model.

[0165] For another example, you can choose to use a fixed ratio of the current model weights to calculate the element-wise quantized scaler. In the initialization phase of model training, a scaler factor ρ is set, and the quantized scaler is the absolute value of the model weight w multiplied by the scaler factor, that is, ρ|w|. Using this scaler to restore the gradient can exponentially amplify or reduce the model weights, thereby accelerating the model convergence.

[0166] It can be understood that, in the implementation manner of the present application, each GPU can use the data stored locally to calculate the finest-grained quantization scaler without introducing additional communication overhead.

[0167] In addition, in a possible implementation, during the model update process, in order to improve the accuracy of the data for calculating the quantization scaler stored locally by each device, the data for calculating the quantization scaler stored locally by each device can be periodically updated. For example, during one of the iterations, the first device can receive unquantized data of at least one first model, and the first device can use the received unquantized model data to update the data of the locally stored model, such as updating the state data of the optimizer, or directly using the gradient to perform model update to obtain new data of the first model, etc., so that in subsequent iterations, more accurate local data can be used to calculate the quantization scaler, thereby improving the accuracy of the calculated quantization scaler.

[0168] For example, the optimizer state can be updated periodically using 32-bit transmission of accurate gradients, such as an update cycle of 50 training steps, which can effectively improve the accuracy of the optimizer state. Usually, in fewer iterations, the statistical characteristics of the gradient change little, so the maintenance of an accurate optimizer state can achieve accurate calculation of multiple rounds of quantized scalers and reduce the amount of communication required to maintain the optimizer state.

[0169] 505. Dequantization.

[0170] After the quantization scaler is calculated, the received quantization codeword may be dequantized based on the quantization scaler.

[0171] Usually, for the same model parameters, the GPU may obtain multiple quantization codewords after GPU quantization. Before dequantization, the multiple quantization codewords obtained can be aggregated to obtain aggregated data. Then, the aggregated quantization codewords can be dequantized based on the quantization scaler, that is, the quantization codewords can be converted into high-bit data.

[0172] 506. Model update.

[0173] After obtaining the available gradient, the model parameters can be updated based on the gradient value to obtain the updated model for the current iteration.

[0174] For example, when using 1-bit quantization, the steps performed by each GPU can be as follows: Figure 6 As shown, each GPU can calculate the gradient g through the SGD algorithm i , quantize the gradient into positive and negative signs sign(g i ), that is, if the gradient is positive, it is encoded as 1, and if it is negative, it is 0. Usually, the quantization codeword of the gradient is a discrete positive and negative sign, which is different from the gradient aggregation represented by traditional floating point numbers and cannot be directly aggregated, which may cause data overflow. The specific method of the quantization codeword in the embodiment of the present application can be as follows Figure 7 As shown in the figure, each row represents the gradient codeword stored in each GPU. Due to the use of parallel optimizer, the model is divided into 4 pieces. Each column represents the gradient codeword of each piece of the model. First, the four gradient codewords of the same piece of model are transferred to the same GPU through all-to-all. Then, each GPU locally aggregates the gradient codewords to obtain ∑sign(g i ). Calculate the element-wise square root of the second-order momentum as a quantized scaler, i.e. The GPU obtains the aggregated gradient codeword ∑sign(g i ) and the gradient quantization scaler The quantized gradient can be recovered later by element-wise multiplication, as expressed as

[0175]

[0176] Then the Adam optimizer is used Update model parameters.

[0177] For example, each GPU can also use the TernGrad quantization algorithm, such as each GPU can use the SGD algorithm to calculate the gradient g i , based on the magnitude of the gradient, the gradient is quantized to -1, 0, or 1, expressed as tern(g i ). Select to use a fixed ratio of the current model weights to calculate the element-wise quantized scaler. Set a scaler factor ρ in the initialization phase of model training. The quantized scaler is the absolute value of the model weight w multiplied by the scaler factor, i.e. ρ|w|. Using this scaler to restore the gradient can exponentially amplify or reduce the model weights, thereby accelerating the model convergence. The GPU obtains the aggregated gradient codeword ∑tern(g i ) and the gradient quantization scaler ρ|w| can later be used to recover the quantized gradient by element-wise multiplication

[0178]

[0179] Then the LAMB optimizer (an optimizer for distributed learning) is used Update the model parameters. In this implementation, the granularity of the scaler can be refined to each dimension of the gradient, reducing the relative error of quantization and avoiding additional communication overhead.

[0180] Obviously, in the aforementioned quantization methods or optimization methods, there is no need to transmit quantization scalers between GPUs, thereby reducing the communication overhead of data transmission.

[0181] In the implementation mode of the present application, there is no need to transmit the quantization scaler, and the quantization granularity is refined to each parameter, which reduces the error of gradient quantization and does not introduce additional communication overhead. For example, after pre-training and comparison tests on Fine-tuning of the BERT model, 1-bit quantization does not reduce the accuracy of model training. The gradient originally represented by a 32-bit floating point number is quantized, and the quantized codeword is only 1 bit, and the communication overhead gain can reach 32 times.

[0182] The above describes the process of the method provided in the present application. The following describes the device structure provided in the present application in combination with the above method process.

[0183] See also Figure 8 , a structural schematic diagram of a data quantization device provided in the present application is described as follows.

[0184] The data quantization device may be any one of the devices in the aforementioned distributed system. The second device mentioned below is a device in the distributed system different from the data quantization device, which may be used to perform the aforementioned Figure 4-Figure 7 Corresponding method steps, the data quantization device may include:

[0185] The transceiver module 801 is used to receive at least one kind of quantized data, where the at least one kind of quantized data corresponds to at least one second device, that is, one kind of quantized data is obtained by quantizing the data of the trained first model after one of the second devices trains the first model;

[0186] An aggregation module 802, configured to perform aggregation based on at least one quantitative data to obtain aggregated data;

[0187] A dequantization module 803, configured to dequantize the aggregated data according to the locally stored relevant data of the first model to obtain dequantized data;

[0188] The updating module 804 is used to update the first model according to the dequantized data to obtain an updated first model.

[0189] In one possible implementation, the dequantization module 803 is specifically used to: calculate a quantization scaler based on relevant data of the first model stored locally, where the quantization scaler represents a range of quantization bits of at least one quantized data; and dequantize the aggregated data based on the quantization scaler to obtain dequantized data.

[0190] In a possible implementation, in the dequantization module 803, the relevant data of the first model include: optimizer data or parameters of the first model stored locally on the first device, the optimizer data include statistical parameters of the gradient or parameters in the loss function, and the parameters of the first model include weight parameters of the first model.

[0191] In a possible implementation, the data of the aforementioned first model may include gradients, the aforementioned quantized data may be obtained by a second device quantizing a local gradient, the gradient being obtained after each second device trains the first model respectively, and the aforementioned dequantized data may be used to represent the value after aggregating the gradient of at least one of the second devices.

[0192] In a possible implementation, the aggregation module 802 is specifically configured to aggregate data based on at least one quantized data and data obtained by local training of the first model to obtain aggregated data.

[0193] In a possible implementation, the first model is divided into multiple regions, and multiple devices update parameters of the multiple regions respectively.

[0194] The updating module 804 is specifically configured to update the parameters of the first region according to the inverse quantization data to obtain an updated first model, where the first region is one of the multiple regions.

[0195] In a possible implementation, multiple devices respectively use training data to iterate the first model multiple times.

[0196] In one of the iterations, the transceiver module 801 is further used to receive at least one type of training data, where the training data is unquantized data;

[0197] The dequantization module 803 is further used to update relevant data of the first model according to at least one training data, and the updated relevant data of the first model is used to dequantize the quantized data in a subsequent iterative training process.

[0198] like Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of a data quantization device 90 provided in an embodiment of the present application. The data quantization device 90 can be used to implement the aforementioned Figures 3 to 7 The steps of the method.

[0199] Fig. 9The data quantization device 90 shown may include: a processor 901 , a memory 902 , a communication interface 903 , and a bus 904 . The processor 901 , the memory 902 , and the communication interface 903 may be connected via a bus 904 .

[0200] The processor 901 is the control center of the data quantization device 90, and can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adaptively set according to the actual application scenario.

[0201] As an example, the processor 901 may include one or more CPUs, and may also include other processors, such as Fig. 9 The CPU, NPU or GPU shown in

[0202] The memory 902 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0203] In a possible implementation, the memory 902 may exist independently of the processor 901. The memory 902 may be connected to the processor 901 via a bus 904 and used to store data, instructions, or program codes. When the processor 901 calls and executes the instructions or program codes stored in the memory 902, the method provided in the embodiment of the present application can be implemented, for example, Figures 4 to 7 The data quantification method shown.

[0204] In another possible implementation, the memory 902 may also be integrated with the processor 901 .

[0205] The communication interface 903 is used for connecting the data quantization device 90 with other devices through a communication network, and the communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 903 may include a receiving unit for receiving data and a sending unit for sending data.

[0206] The bus 904 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0207] It should be pointed out that Fig. 9 The structure shown in the figure does not constitute a limitation on the data quantization device 90, except Fig. 9 In addition to the components shown, the data quantization device 90 may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0208] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a data quantization device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0209] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0210] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for training a model or performing an inference task is stored. When the program is run on a computer, the computer performs the above-mentioned Figures 4 to 7 All or part of the steps in the method described in the embodiment shown in FIG.

[0211] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor or the function of the processor. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more embodiments in the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the method steps of any one or more embodiments in the above-mentioned embodiments according to the program code stored in the external memory.

[0212] A computer program product is also provided in the embodiment of the present application, and the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0213] The target sensing device or target sensing device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be a processor, and the communication unit may be an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figure 4-Figure 7 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0214] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0215] For example, see Fig.10 , Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 100, NPU 100 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0216] In some implementations, the operation circuit 1003 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1003 is a two-dimensional systolic array. The operation circuit 1003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1003 is a general-purpose matrix processor.

[0217] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1001 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1008.

[0218] The unified memory 1006 is used to store input data and output data. The weight data is directly transferred to the weight memory 1002 through the direct memory access controller (DMAC) 1005. The input data is also transferred to the unified memory 1006 through the DMAC.

[0219] The bus interface unit (BIU) 1010 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1009 .

[0220] The bus interface unit 1010 (BIU) is used for the instruction fetch memory 1009 to obtain instructions from the external memory, and is also used for the storage unit access controller 1005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0221] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1006 or to transfer weight data to the weight memory 1002 or to transfer input data to the input memory 1001.

[0222] The vector calculation unit 1007 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0223] In some implementations, the vector calculation unit 1007 can store the processed output vector to the unified memory 1006. For example, the vector calculation unit 1007 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1003, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1003, for example, for use in a subsequent layer in a neural network.

[0224] An instruction fetch buffer 1009 connected to the controller 1004 is used to store instructions used by the controller 1004;

[0225] Unified memory 1006, input memory 1001, weight memory 1002 and instruction fetch memory 1009 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0226] The operations of each layer in the recurrent neural network may be performed by the operation circuit 1003 or the vector calculation unit 1007 .

[0227] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the above Figure 4-Figure 7 The method of procedure.

[0228] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0229] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0230] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0231] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0232] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. The term "and / or" in the present application is only a kind of association relationship describing the associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can be represented: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some ports, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the present application solution.

Claims

1. A data quantization method, characterized in that: Applied to a distributed system, the distributed system includes a plurality of devices, the plurality of devices are used to respectively train a first model using training data, the method includes: A first device receives at least one type of quantized data, where the at least one type of quantized data corresponds to at least one second device, wherein the one type of quantized data is obtained by quantizing data of a trained first model after the second device trains the first model, and the first device and at least one of the second devices are different devices among the multiple devices; The first device performs aggregation based on the at least one quantified data to obtain aggregated data; The first device dequantizes the aggregated data according to the locally stored relevant data of the first model to obtain dequantized data; The first device updates the first model according to the dequantized data to obtain an updated first model.

2. The method according to claim 1, characterized in that The first device dequantizes the aggregated data according to the locally stored relevant data of the first model to obtain dequantized data, including: The first device calculates a quantization scaler according to locally stored relevant data of the first model, the quantization scaler representing a range of quantization bits of the at least one quantization data; The first device dequantizes the aggregated data according to the quantization scaler to obtain the dequantized data.

3. The method according to claim 2, characterized in that The relevant data of the first model include: the optimizer data or the parameters of the first model saved locally by the first device, the optimizer data include the statistical parameters of the gradient or the parameters in the loss function, and the parameters of the first model include the weight parameters of the first model.

4. The method according to any one of claims 1 to 3, characterized in that The data of the first model includes gradients, and the quantized data is obtained by quantizing a local gradient by the second device; The dequantized data is used to represent a value after aggregating the gradient of at least one of the second devices.

5. The method according to any one of claims 1 to 4, characterized in that The first device performs aggregation based on the at least one quantified data to obtain aggregated data, including: The first device aggregates the at least one quantized data and the data obtained by the first device training the first model locally to obtain the aggregated data.

6. The method according to any one of claims 1 to 5, characterized in that The first model is divided into a plurality of regions, the plurality of devices respectively update parameters of the plurality of regions, the first device updates the first model according to the dequantized data to obtain an updated first model, and further includes: The first device updates parameters of a first region according to the dequantized data to obtain an updated first model, where the first region is one of the multiple regions.

7. The method according to any one of claims 1 to 6, characterized in that The multiple devices respectively use the training data to iterate the first model multiple times, and the method further includes: In one of the iterations, the first device receives at least one type of training data, where the training data is unquantized data; The first device updates relevant data of the first model according to the at least one training data, and the updated relevant data of the first model is used to dequantize the quantized data.

8. A data quantization device, characterized in that: Applied to a distributed system, the distributed system includes a plurality of devices, the plurality of devices are used to respectively train a first model using training data, the apparatus is one of the plurality of devices, and the apparatus includes: a transceiver module, configured to receive at least one type of quantized data, wherein the at least one type of quantized data corresponds to at least one second device, wherein the at least one type of quantized data is obtained by quantizing data of a trained first model after a second device trains a first model, and the at least one second device includes a device other than the apparatus among the plurality of devices; an aggregation module, configured to perform aggregation based on the at least one quantitative data to obtain aggregated data; A dequantization module, configured to dequantize the aggregated data according to the locally stored relevant data of the first model to obtain dequantized data; An updating module is used to update the first model according to the dequantized data to obtain an updated first model.

9. The device according to claim 8, characterized in that The dequantization module is specifically used for: Calculate a quantization scaler according to the locally stored relevant data of the first model, the quantization scaler representing a range of quantization bits of the at least one quantization data; The aggregated data is dequantized according to the quantization scaler to obtain the dequantized data.

10. The device according to claim 9, characterized in that The relevant data of the first model include: the optimizer data or the parameters of the first model saved locally by the first device, the optimizer data include the statistical parameters of the gradient or the parameters in the loss function, and the parameters of the first model include the weight parameters of the first model.

11. The device according to any one of claims 8 to 10, characterized in that The data of the first model includes gradients, and the quantized data is obtained by quantizing a local gradient by the second device; The dequantized data is used to represent a value after aggregating the gradient of at least one of the second devices.

12. The device according to any one of claims 8 to 11, characterized in that The aggregation module is specifically used to aggregate data obtained by training the first model locally based on the at least one quantitative data to obtain the aggregated data.

13. The device according to any one of claims 8 to 12, characterized in that The first model is divided into multiple regions, and the multiple devices update parameters of the multiple regions respectively. The updating module is specifically used to update the parameters of the first region according to the inverse quantization data to obtain an updated first model, where the first region is one of the multiple regions.

14. The device according to any one of claims 8 to 13, characterized in that The multiple devices respectively use the training data to iterate the first model multiple times; In one of the iterations, the transceiver module is further used to receive at least one type of training data, where the training data is unquantized data; The dequantization module is further used to update the relevant data of the first model according to the at least one training data, and the updated relevant data of the first model is used to dequantize the quantized data.

15. A data quantization device, characterized in that: include: A memory storing executable program instructions; and A processor, wherein the processor is used to be coupled with the memory, read and execute instructions in the memory, and trigger the client to implement the method according to any one of claims 1 to 7.

16. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

18. A distributed system, characterized in that: The method comprises a plurality of devices, wherein the plurality of devices comprises a device for performing the method as claimed in any one of claims 1 to 7.

Images