A method and device for data processing

Abstract

The present application discloses a method for data processing, which can be used in the field of artificial intelligence (AI). The method includes: the terminal device divides the gradient data of the model to be trained into gradient segments, and then performs quantization conversion on the gradient segments according to the codebook, the number of elements in the quantization representation obtained after the quantization conversion is less than the number of elements in the gradient segments, and then sends the quantization representation to the cloud device. Since the number of elements in the quantization representation of the present application is less than the number of elements in the gradient segments, that is, compared with the element-by-element quantization technology, the compression ratio of the gradient data is improved. Because the compression ratio of the quantization representation is improved, the communication overhead when the terminal device transmits the quantization representation to the cloud device is also reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] In recent years, with the improvement of hardware computing power and the emergence of big data, deep learning models are being developed and applied at an unprecedented speed and enthusiasm. Deep learning models need to be trained before they can be applied. Federated learning is a model training solution. Cloud devices can work with multiple terminal devices to jointly train models and solve privacy issues at the same time. Federated learning has multiple rounds of training processes. In each round, multiple terminal devices are selected according to certain rules to participate in the training, which solves the problem of small amount of sample data on the terminal side to a certain extent. During the federated learning process, user privacy data is used locally on the terminal side and does not need to be uploaded to the cloud side, which can solve the problem of privacy leakage. The gradient obtained from each terminal training must be uploaded to the cloud side. The cloud side aggregates the gradient data uploaded by multiple terminals and then sends the aggregated gradient data to the terminal side.

[0003] A large amount of gradient data is transmitted between the client and the cloud, causing federated learning to face a huge communication bottleneck problem that needs to be solved urgently. Summary of the invention

[0004] The embodiment of the present application provides a data processing method, which can improve the compression ratio of gradient data and effectively reduce the communication overhead of gradient data transmission. The embodiment of the present application also provides a corresponding device.

[0005] The first aspect of the present application provides a method for data processing, which can be applied to a terminal device, such as a terminal device. The method may include: obtaining A gradient data of a model to be trained, where A is a positive integer. For each gradient data of the B gradient data (or referred to as the first gradient data), the following operations are performed: the gradient data is divided into at least two gradient segments, the gradient data includes d elements, the B gradient data are included in the A gradient data, B is a positive integer and B≤A, each gradient segment includes d′ elements, d′ is a positive integer greater than 2, and d is divisible by d′; each gradient segment of the at least two gradient segments is quantized using the code book (or first code book) of the gradient data to obtain the quantization representation corresponding to each gradient segment, the number of elements in the quantization representation is less than d′, the code book is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, m is a positive integer greater than 2, and m≥d′. B quantization sets corresponding to the B gradient data are sent to a cloud device, wherein a first quantization set corresponding to the first gradient data includes a quantization representation corresponding to each gradient segment.

[0006] It should be noted that, for ease of reference, in this application, "first gradient data" may be used to refer to any one of the B gradient data. This application does not limit the source of the model to be trained and the A gradient data.

[0007] Here, "sending the B quantization sets corresponding to the B gradient data" can mean sending the B gradient data uniformly after all quantization conversions are completed, or sending them in multiple times or sending them after some (including one) are completed without waiting for all to be completed.

[0008] In the first aspect, there may be A weight-related operators in the model to be trained, each operator will have a corresponding gradient data, each gradient data will include at least one element, and each element can represent a gradient. B gradient data can be part of or all of A gradient data, for example: when B=A, it means that each of A gradient data needs to be divided into gradient segments and quantized, when B<A, it means that part of them needs to be divided into gradient segments and quantized, and the remaining (AB) gradient data do not need to be divided into gradient segments and quantized. When dividing the gradient segments, d' can be determined according to the number of d, and d can be divided by d', for example: d=1200, d'=16, d / d'=75, which means that the first gradient data can be divided into 75 gradient segments. Each first gradient data will correspond to a codebook, and the d' of different gradient data may be different, and the d' and m in different codebooks may also be different. m≥d' can ensure that the first matrix or the second matrix is ​​in a full rank state, so that a column or row with higher matching accuracy can be found for each gradient segment. During model training, it is usually specified to use the first matrix or the second matrix. The number of elements in the quantized representation is less than d′, which further reduces the amount of quantized data on the basis of quantization compression, thereby improving the compression ratio. In addition, because the compression ratio of the quantized representation is improved, the communication overhead when the terminal device transmits the quantized representation to the cloud device is also reduced.

[0009] In a possible implementation manner of the first aspect, when A>B, the method may further include:

[0010] The (AB) gradient data among the A gradient data except the B gradient data are sent to the cloud device.

[0011] In this possible implementation, the (AB) gradient data may have very few elements, and the quantization conversion process will consume computing resources. From the perspective of comprehensive benefits, the (AB) gradient data can be directly sent without compression processing, or other compression processing methods can be used to compress the (AB) gradient data and then send them.

[0012] In a possible implementation manner of the first aspect, the method may further include:

[0013] According to the data amount of each gradient data, determining that B gradient data among the A gradient data need to be quantized;

[0014] For each first gradient data in the B gradient data, determine d′ and m according to the number of elements d of the first gradient data, where d′ is 2 to the pth power, m is 2 to the qth power, p and q are both positive integers, and q≥p;

[0015] The first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows is randomly generated for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0016] In this possible implementation, a data volume threshold that needs to be quantized can be determined from the perspective of comprehensive benefits. For example, the data volume threshold is set to 512 elements. If the number of elements in the gradient data is greater than or equal to 512, it means that quantization conversion is required. If the number of elements in the gradient data is less than 512, it means that quantization conversion is not required. The terminal device only needs to generate a codebook for the gradient data that needs to be quantized. This can reduce the computational overhead caused by generating the codebook and improve the comprehensive benefits of computational overhead and communication overhead. In order to ensure that the cloud device can restore the gradient data normally, the random seed used by the terminal device to generate the codebook is the same as that of the cloud device. The random seed can be a numerical value. By inputting the random seed into the random function, a codebook for gradient data can be randomly generated.

[0017] In a possible implementation manner of the first aspect, the method may further include:

[0018] Receive codebooks corresponding to the B gradient data respectively sent by the cloud device, wherein the codebooks corresponding to the B gradient data respectively include the first codebook.

[0019] In this possible implementation, the terminal device does not need to generate a codebook by itself. The codebook can be generated by the cloud device and then sent to the terminal device, which can reduce the computing overhead of the terminal device.

[0020] In a possible implementation manner of the first aspect, the step of: using the first codebook of the first gradient data to perform quantization conversion on each of the at least two gradient segments to obtain a quantized representation corresponding to each gradient segment may include:

[0021] Determine a target code for a first gradient segment from the first codebook, wherein the target code is a column in the first matrix or a row in the second matrix, and the first gradient segment is any one of the at least two gradient segments;

[0022] Determining a pseudo norm and a code index of the target code;

[0023] The pseudo modulus length and the code index are determined as a quantized representation of the first gradient segment.

[0024] In this possible implementation, the pseudo-modulus is relative to the modulus, and the pseudo-modulus is not obtained by the mathematical calculation method of the vector modulus, but is calculated through the code book and the gradient fragment. The code index represents the index of a column of the first matrix, for example: index(5) represents the fifth column. Or, it represents the index of a row of the second matrix, for example: index(6) represents the sixth row. If the modulus is represented by u and the code index is represented by index(c), the quantized representation can be (u, index(c)). It can be seen from this possible implementation that a gradient fragment is converted into only two elements including the pseudo-modulus and the code index, which greatly reduces the amount of data, improves the compression ratio of the gradient data, and reduces the communication overhead of transmitting the gradient data.

[0025] In a possible implementation manner of the first aspect, the step of determining a target code for the first gradient segment from the first codebook may include:

[0026] Determining a vector norm of the first gradient segment;

[0027] If the vector modulus is equal to 0, any column in the first matrix or any row in the second matrix is ​​determined as the target code.

[0028] In this possible implementation, the vector modulus is the sum of the squares of the elements on the first gradient segment and then the square root is calculated. If the vector modulus is equal to 0, it means that all elements are 0, so you only need to randomly select a column or a row to form a format that conforms to the quantization representation, for example, you can select the first column or the first row.

[0029] In a possible implementation manner of the first aspect, the method may further include:

[0030] If the vector modulus is not equal to 0, determining a first coefficient vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the first coefficient vector includes m elements;

[0031] Normalizing each element in the first coefficient vector to obtain a second coefficient vector, wherein the second coefficient vector includes m normalized elements;

[0032] A roulette wheel selection strategy is adopted for the normalized m elements to determine that the i-th column in the first matrix or the i-th row in the second matrix corresponding to the i-th element is the target code, and the i-th element is the element selected by the roulette wheel selection strategy.

[0033] In this possible implementation, if the first codebook is represented by C, the first gradient segment is represented by g, and the first coefficient vector is represented by p, p=C T (CC T ) -1 g, where CT represents the transposed matrix of C and the second coefficient vector is express, It includes m normalized elements, and element normalization can be understood as: in represents the bth element among m normalized elements, |p b | represents the absolute value of the bth element in the first coefficient vector p, and ||p||1 represents the sum of the absolute values ​​of the elements in the m elements of the first coefficient vector p. In this scheme, the roulette selection strategy can be considered that each normalized element has its own probability, and the probability of each element is large or small. The sum of all probabilities is equal to 1. The probability of being selected with a large probability is also large, and the probability of being selected with a small probability is also small. However, the final selection result is still based on the element selected after the roulette wheel is turned. If the selected element is Then the code of the column or row corresponding to the bth element is selected as the target code. Of course, if the i-th normalized element is selected, the code of the column or row corresponding to the i-th element is selected as the target code.

[0034] In a possible implementation manner of the first aspect, the step of determining a target code for the first gradient segment from the first codebook includes:

[0035] Determine a projection vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the projection vector includes m elements;

[0036] Determine that the i-th column in the first codebook or the i-th row in the second matrix corresponding to the i-th element is a target code, and the absolute value of the i-th element is the largest among the absolute values ​​of the m elements.

[0037] In this possible implementation, if the first codebook is represented by C, the first gradient segment is represented by g, and the projection vector is represented by p, then p=C T g, where CT represents the transposed matrix of C. The projection vector p includes m elements; |p i | represents the absolute value of the i-th element in the m elements. If i=5, it means that the 5th column in the first codebook or the 5th row in the second matrix is ​​selected as the target code.

[0038] In a possible implementation manner of the first aspect, the method may further include:

[0039] Receive respective quantized sets of B gradient aggregate data corresponding to the B gradient data sent by the cloud device, wherein the quantized set of each gradient aggregate data includes a quantized representation corresponding to each gradient segment of the gradient aggregate data.

[0040] In this possible implementation, after the cloud device performs gradient aggregation, the gradient aggregation data obtained after aggregation can also be quantized using the quantization conversion method of the above-mentioned terminal device, which can further reduce the communication overhead between the cloud device and the terminal device.

[0041] The second aspect of the present application provides a method for data processing, which can be applied to cloud-side devices, such as cloud devices. The cloud device can be a server, or it can be other devices or a virtual resource. The method may include: receiving B quantization sets corresponding to B gradient data sent by a terminal device, wherein each quantization set includes a quantization representation corresponding to each gradient segment of the corresponding gradient data; for each first quantization set in the B quantization sets, using the first codebook corresponding to the first quantization set to dequantize each quantization representation in the first quantization set to obtain the first gradient segment corresponding to each quantization representation, the first gradient segment includes d′ elements, the number of elements in the quantization representation is less than d′, the first codebook is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, and d and m are A positive integer greater than 2, and m≥d′; concatenating the first gradient fragments corresponding to each quantization representation in the first quantization set to obtain first gradient data corresponding to the first quantization set, the first gradient data including elements, d being a positive integer, and d being divisible by d′; for a first weight among B weights corresponding to the B gradient data, performing gradient aggregation on the N first gradient data of the first weight to obtain gradient aggregation data of the first weight, the multiple first gradient data of the first weight corresponding to N terminal devices, and N being an integer greater than 1; and sending the gradient aggregation data of each of the B weights to the terminal device.

[0042] It should be noted that, for the convenience of reference, in this application, "first quantization set" may be used to refer to any one of the B quantization sets. "First gradient data" may be used to refer to any one of the B gradient data.

[0043] Here, “receiving B quantization sets corresponding to B gradient data” may be received in a unified manner or in multiple times.

[0044] Here, "sending the gradient aggregation data of each of the B weights" can be sending them all at once after the gradient aggregation is completed, or can be sending them in multiple times, or can be sending them after some (including one) are completed without waiting for all to be completed.

[0045] In the second aspect, after receiving the quantized representation, the cloud device will use the same codebook as the terminal device during the quantization conversion to perform inverse quantization, that is, quantization recovery, and then perform gradient aggregation. The process of gradient aggregation refers to the cloud device aggregating the gradients of the same weight received from multiple terminal devices. For example, if 500 gradients are received for weight A, these 500 gradients can be added and averaged, and the average value is used as the aggregated gradient of weight A. After the cloud device sends the aggregated gradient aggregation data to the terminal device, the terminal device can update each weight, so that the model to be trained can be further converged. The weight update process can be to subtract the gradient aggregation data calculated in this round from the current weight of this round to obtain the updated weight, and the updated weight is used for the next round of training. In the second aspect, because the cloud device receives the quantized representation after the quantization conversion of the terminal device from the terminal device, and because the number of elements in the quantized representation is less than the number of elements d′ in each gradient segment, only a very small communication overhead is required to achieve the reception of gradient data.

[0046] In a possible implementation manner of the second aspect, the method may further include:

[0047] For each first gradient data among B gradient data, the first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows is randomly generated for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0048] In this possible implementation, the random seed may be a numerical value, and a codebook of gradient data may be randomly generated by inputting the random seed into the random function.

[0049] In a possible implementation manner of the second aspect, the method may further include:

[0050] Sending codebooks corresponding to the B gradient data to the terminal device.

[0051] In this possible implementation, the cloud device sends the generated codebook to the terminal device, which can reduce the computing overhead of the terminal device.

[0052] In a possible implementation manner of the second aspect, the step of: using the first codebook corresponding to the first quantization set to dequantize each quantization representation in the first quantization set to obtain a first gradient segment corresponding to each quantization representation may include:

[0053] For each first quantization representation in the first quantization set, determine a target code of the first quantization representation from the first codebook according to a code index of the first quantization representation;

[0054] A first gradient segment corresponding to the first quantized representation is restored according to the pseudo modulus length in the first quantized representation and the target code.

[0055] In this possible implementation, the cloud device can find the target code from the first codebook according to the code index, and then use the pseudo modulus length to multiply the elements in the target code to restore the corresponding gradient segment.

[0056] In a possible implementation manner of the second aspect, the method may further include:

[0057] Performing quantization conversion on the B weighted gradient aggregation data respectively to obtain respective quantization sets of the B gradient aggregation data;

[0058] The sending the gradient aggregation data of each of the B weights to the terminal device includes:

[0059] Sending respective quantization sets of the B gradient aggregated data to the terminal device, wherein the quantization set of each gradient aggregated data includes a quantization representation corresponding to each gradient segment of the gradient aggregated data.

[0060] In this possible implementation, the cloud device may perform a quantization conversion on the gradient aggregation data to be sent to the terminal device similar to the quantization conversion performed by the terminal device on the gradient data, which may further reduce the communication overhead between the cloud device and the terminal device.

[0061] The third aspect of the present application provides a terminal device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: a receiving unit, a processing unit, and a sending unit.

[0062] In a fourth aspect, the present application provides a cloud device having the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: a receiving unit, a processing unit, and a sending unit. The cloud device in the present application can be any computer device deployed on the network side or the cloud side.

[0063] A fifth aspect of the present application provides a terminal device, which includes at least one processor, a memory, a transceiver, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0064] In a sixth aspect, the present application provides a cloud device, which includes at least one processor, a memory, a communication port, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes the method described in the second aspect or any possible implementation of the second aspect.

[0065] In a seventh aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0066] In an eighth aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the second aspect or any possible implementation of the second aspect.

[0067] The ninth aspect of the present application provides a computer program product (or computer program) storing one or more computer execution instructions. When the computer execution instructions are executed by the processor, the processor executes the method of the above-mentioned first aspect or any possible implementation of the first aspect.

[0068] The tenth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method of the second aspect or any possible implementation of the second aspect.

[0069] In the eleventh aspect of the present application, a chip system is provided, which includes a processor for supporting a terminal device to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory for storing program instructions and data necessary for the terminal device. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0070] In the twelfth aspect of the present application, a chip system is provided, which includes a processor for supporting a cloud device to implement the functions involved in the second aspect or any possible implementation of the second aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the cloud device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0071] Among them, the technical effects brought about by the third, fifth, seventh, ninth and eleventh aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0072] Among them, the technical effects brought about by the fourth, sixth, eighth, tenth and twelfth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the second aspect or different possible implementation methods of the second aspect, and will not be repeated here.

[0073] The embodiment of the present application divides the gradient data to be transmitted into gradient segments, and then performs quantization conversion according to the code book to obtain a quantized representation, and the number of elements in the quantized representation is less than the number of elements in each gradient segment. In this way, the amount of quantized data is further reduced on the basis of quantization compression, thereby improving the compression ratio. In addition, because the compression ratio of the quantized representation is improved, the communication overhead when the terminal device transmits the quantized representation to the cloud device is also reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A schematic diagram of an artificial intelligence main framework provided for an embodiment of the present application;

[0075] Figure 2 A schematic diagram of an application environment provided for an embodiment of the present application;

[0076] Figure 3 A schematic diagram of a convolutional neural network structure provided in an embodiment of the present application;

[0077] Figure 4 A schematic diagram of a convolutional neural network structure provided in an embodiment of the present application;

[0078] Figure 5A schematic diagram of the structure of a neural network processor provided in an embodiment of the present application;

[0079] Figure 6 A schematic diagram of a terminal-cloud combined model training system provided in an embodiment of the present application;

[0080] Figure 7 A schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;

[0081] Figure 8 A schematic diagram of another embodiment of the data processing method provided in an embodiment of the present application;

[0082] Fig. 9 A schematic diagram of an embodiment of a terminal device provided in an embodiment of the present application;

[0083] Fig.10 A schematic diagram of an embodiment of a cloud device provided in an embodiment of the present application;

[0084] Fig.11 A schematic diagram of an embodiment of a terminal device provided in an embodiment of the present application;

[0085] Fig.12 A schematic diagram of an embodiment of a cloud device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0087] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0088] The embodiment of the present application provides a data processing method, which can improve the compression ratio of gradient data in the gradient transmission process between the terminal side and the cloud side, and effectively reduce the communication overhead of gradient data transmission. The embodiment of the present application also provides a corresponding device. The following are detailed descriptions.

[0089] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0090] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0091] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0092] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0093] (1) Infrastructure:

[0094] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (central processing unit (CPU), neural network processor (NPU), graphic processing unit (GPU), application specific integrated circuit (ASIC), field-programmable gate array (FPGA) and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0095] (2) Data

[0096] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0097] (3) Data processing

[0098] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0099] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0100] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0101] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0102] (4) General ability

[0103] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0104] (5) Smart products and industry applications

[0105] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0106] See attached Figure 2 , the embodiment of the present application provides a system architecture 200. The data acquisition device 260 is used to collect training data and store it in the database 230, and the training device 220 generates a target model / rule 201 based on the training data maintained in the database 230. The following will describe in more detail how the training device 220 obtains the target model / rule 201 based on the training data. The target model / rule 201 can be used in application scenarios such as image recognition, video classification, speech recognition, and language translation.

[0107] The target model / rule 201 may be obtained based on a deep neural network or a convolutional neural network (CNN). The deep neural network or the convolutional neural network will be introduced below.

[0108] The work of each layer in a deep neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0109] Because we want the output of the deep neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0110] The target model / rule obtained by the training device 220 can be applied to different systems or devices. Figure 2In the embodiment, the execution device 210 is configured with an I / O interface 212 for data interaction with external devices, and a “user” can input data into the I / O interface 212 through a client device 240 .

[0111] The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0112] The calculation module 211 processes the input data using the target model / rule 201. Taking text-type language translation as an example, the calculation module 211 can parse the sentences in the text of the first language to obtain the subject, predicate, object and other words in each sentence.

[0113] The association function module 213 can translate the subject, predicate, object and other words in the first sentence in the calculation module 211 into a second language, and then organize the sentence in combination with the grammatical logic of the second language.

[0114] The association function module 214 can translate the subject, predicate, object and other words in the second sentence in the calculation module 211 into the second language, and then organize the sentence in combination with the grammatical logic of the second language.

[0115] Finally, the I / O interface 212 returns the processing result to the client device 240 and provides it to the user.

[0116] More deeply, the training device 220 can generate corresponding target models / rules 201 based on different data for different goals to provide users with better results.

[0117] In the attached Figure 2 In the case shown in , the user can manually specify the data input into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data into the I / O interface 212 and obtain the result. If the client device 240 needs to obtain the user's authorization to automatically input data, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also serve as a data collection terminal to store the collected training data into the database 230.

[0118] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0119] A convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron responds to overlapping areas in the image input to it.

[0120] like Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0121] Convolutional layer / pooling layer 120:

[0122] Convolutional Layer:

[0123] like Figure 3 The convolution layer / pooling layer 120 shown may include layers 121-126 as shown in the examples. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0124] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0125] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0126] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0127] Pooling layer:

[0128] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 3The layers 121-126 illustrated in 120 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value. The maximum pooling operator may take the pixel with the largest value within a specific range as the result of the maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0129] Neural network layer 130:

[0130] After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 3 As shown in 131, 132 to 13n) and the output layer 140, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0131] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 3 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 3 The propagation from 140 to 110 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0132] It should be noted that if Figure 3The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 4 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0133] Figure 3 and Figure 4 The convolutional neural network based algorithm shown can be found in Figure 5 The NPU chip shown is implemented in the figure.

[0134] Figure 5 This is a chip hardware structure diagram provided in an embodiment of the present application.

[0135] The neural network processor NPU 50NPU is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core part of the NPU is the operation circuit 50, which controls the operation circuit 503 through the controller 504 to extract the matrix data in the memory and perform multiplication operations.

[0136] In some implementations, the operation circuit 503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0137] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 508.

[0138] The unified memory 506 is used to store input data and output data. The weight data is directly transferred to the weight memory 502 through the storage unit access controller 505 Direct Memory Access Controller, DMAC. The input data is also transferred to the unified memory 506 through the DMAC.

[0139] BIU stands for Bus Interface Unit, i.e., bus interface unit 510, which is used for the interaction between AXI bus, DMAC and instruction fetch buffer 509.

[0140] The bus interface unit 510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 509 to obtain instructions from the external memory, and is also used for the storage unit access controller 505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0141] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 506 or to transfer weight data to the weight memory 502 or to transfer input data to the input memory 501.

[0142] The vector calculation unit 507 has multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / FC layer network calculations in neural networks, such as Pooling, Batch Normalization, Local Response Normalization, etc.

[0143] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0144] An instruction fetch buffer 509 connected to the controller 504, for storing instructions used by the controller 504;

[0145] Unified memory 506, input memory 501, weight memory 502 and instruction fetch memory 509 are all On-Chip memories. External memories are private to the NPU hardware architecture.

[0146] in, Figure 3 and Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the matrix calculation unit 212 or the vector calculation unit 507.

[0147] Above Figures 1 to 5 The relevant contents of artificial intelligence are described, and the embodiment of the present application provides a method for data processing. The method for data processing provided by the embodiment of the present application can be based on, for example, Figure 2The training device 220 in the embodiment of the present invention is implemented, and the training device 220 may be equivalent to the model training system of the present application. It should be noted that the expression form of the model training system provided in the embodiment of the present application may be different from the above-mentioned Figure 2 The training device 220 is different from that in the embodiment of the present application, but the model trained in the embodiment of the present application can be applied to Figure 1 In the various scenarios described, the model can be used Figure 3 to Figure 4 Any possible neural network structure in .

[0148] The model training system of the embodiment of the present application may be a terminal-cloud combined model training system. Figure 6 , introducing an embodiment of the present application to provide a model training system that combines end and cloud.

[0149] The model training system includes a cloud device and multiple terminal devices, and the cloud device and the multiple terminal devices are connected to each other through a communication network.

[0150] A cloud device can be a collection of resources with computing and data transmission and reception functions. It can be an independent computer device or a cluster of multiple independent computer devices. It can also be a virtual machine (VM).

[0151] Terminal equipment (also known as user equipment (UE)) is a device with wireless transceiver function, which can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on the water (such as ships, etc.); it can also be deployed in the air (such as airplanes, balloons and satellites, etc.). The terminal can be a mobile phone, a tablet computer (pad), a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0152] In the end-cloud combined model training system, a copy of the model to be trained is stored on the cloud device and each terminal device. Of course, the model to be trained can be expressed in the form of a computational graph containing operators and edges or in other forms of files on the cloud device and each terminal device.

[0153] As mentioned above Figure 2 The corresponding content describes the deep neural network model, and the model training process is the process of updating weights through continuous training. Because the weights set for the model to be trained are usually large when the weights are initialized, the weight update process is usually to subtract the aggregate gradient of the weight in this round from the current weight in this round to obtain the updated weight, which is used for the next round of training.

[0154] The process of model training is usually the process of calculating the gradient of each weight, and the gradient is usually the derivative of the weight. Because each terminal device will calculate a gradient for the same weight (for example, weight A), and each terminal device usually uses different training data during model training, the gradient calculated by each training node for the same weight is usually different, so the cloud device is required to aggregate the gradients calculated by each terminal device to obtain an aggregated gradient. The process of gradient aggregation is usually the process of adding the gradients calculated by each terminal device and then averaging them. For example: for weight A, the gradient calculated by terminal device 0 is a0, the gradient calculated by terminal device 1 is a1, the gradient calculated by terminal device 2 is a2, and the gradient calculated by terminal device 3 is a3, then the aggregated gradient of weight A can be (a0+a1+a2+a3) / 4. Of course, the method of gradient aggregation is not limited to this one, and other applicable gradient aggregation methods are also applicable to this application, which will not be introduced further here.

[0155] In the embodiment of the present application, the terminal device needs to transmit gradient data to the cloud device, which will generate communication overhead between the terminal device and the cloud device. In order to minimize the communication overhead, the embodiment of the present application provides a method for data processing.

[0156] See also Figure 7 , an embodiment of the data processing method provided in the embodiment of the present application may include:

[0157] 601. The terminal device obtains A gradient data of the model to be trained.

[0158] A is a positive integer. The number of A in the embodiment of the present application is related to the structure of the model to be trained. Taking the model to be trained as a convolutional neural network as an example, the structure of the model to be trained is convolution (Conv) 1-Conv2-full connection (FC) 1-FC2), wherein Conv1 and Conv2 respectively include two logical units, filter and bias, so A=6 in the model. It can also be understood that the model includes 6 weight-related operators. This is just an example of a model of this structure. The value of A is related to the specific model structure, and this application does not limit it. Each gradient data will include at least one element, and each element can represent a gradient.

[0159] 602. The terminal device divides each first gradient data in B gradient data into at least two gradient segments.

[0160] It should be noted that, for ease of reference, in this application, "first gradient data" may be used to refer to any one of the B gradient data. This application does not limit the source of the model to be trained and the A gradient data.

[0161] The first gradient data includes d elements, the B gradient data are included in the A gradient data, B is a positive integer and B≤A, each gradient segment includes d′ elements, d′ is a positive integer greater than 2, and d is divisible by d′.

[0162] The B gradient data may be part of or all of the A gradient data. For example, when B=A, it means that each of the A gradient data needs to be divided into gradient segments. When B<A, it means that part of the A gradient data needs to be divided into gradient segments, and the remaining (AB) gradient data do not need to be divided into gradient segments. When dividing the gradient segments, d' can be determined according to the number of d, and d can be divided by d'. For example, when d=1200, d'=16, d / d'=75, it means that the first gradient data can be divided into 75 gradient segments.

[0163] 603. The terminal device uses the first codebook of the first gradient data to perform quantization conversion on each gradient segment of the at least two gradient segments to obtain a quantization representation corresponding to each gradient segment.

[0164] The number of elements in the quantized representation is less than d′, and the first codebook is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, where m is a positive integer greater than 2, and m≥d′.

[0165] Each first gradient data will correspond to a codebook, and the d' of different gradient data may be different, and the d' and m in different codebooks may also be different. m ≥ d' can ensure that a matching column or row is found for each gradient segment in the first matrix or the second matrix. In the model training process, it is usually specified to use the first matrix or the second matrix.

[0166] Regarding which gradient data the terminal device performs quantization conversion on, and the first matrix or the second matrix used, please refer to the following Table 1 for understanding.

[0167] Table 1: Relationship between gradient data and codebook

[0168]

[0169] As can be seen from Table 1, for the model to be trained of the convolutional neural network structure described in step 601, there can be 6 gradient data, each of which has multiple elements, such as: the gradient data of Con1-filter has 150 elements, the gradient data of Con1-bias has 6 elements, the gradient data of Con2-filter has 2400 elements, the gradient data of Con2-bias has 96 elements, the gradient data of FC1-weight has 48000 elements, and the gradient data of FC2-weight has 1200 elements. Among them, whether to quantize a line, N means no quantization is required, and Y means quantization is required. If the set data volume threshold for whether quantization is required is 512, then the gradient data of Con1-filter, Con1-bias and Con2-bias are all less than 512, no quantization is required, and no codebook is required. In addition, the gradient data of Con2-filter, FC1-weight and FC2-weight all exceed 512, which means that they all need to be quantized, so each gradient data needs a corresponding codebook. For example: the gradient data of Con2-filter has 2400 elements. If d′=32, it can be divided into 75 gradient segments, and m=32, which means that the codebook is a 32*32 matrix. Similarly, the codebook corresponding to the gradient data of FC1-weight is a 64*64 matrix. The codebook corresponding to the gradient data of FC2-weight is a 16*32 matrix.

[0170] 604. The terminal device sends B quantization sets corresponding to the B gradient data to the cloud device, wherein the first quantization set corresponding to the first gradient data includes the quantization representation corresponding to each gradient segment.

[0171] Here, "sending the B quantization sets corresponding to the B gradient data" can mean sending the B gradient data uniformly after all quantization conversions are completed, or sending them in multiple times or sending them after some (including one) are completed without waiting for all to be completed.

[0172] Each quantization set corresponds to a gradient data. If B=3, gradient data 1 has 75 gradient segments, then there are 75 quantization representations in the quantization set corresponding to gradient data 1. Gradient data 2 has 120 gradient segments, then there are 120 quantization representations in the quantization set corresponding to gradient data 2. Gradient data 3 has 150 gradient segments, then there are 150 quantization representations in the quantization set corresponding to gradient data 3.

[0173] When A>B, the terminal device also sends (AB) gradient data among the A gradient data except the B gradient data to the cloud device.

[0174] The (AB) gradient data may have very few elements, and the quantization conversion process will consume computing resources. From the perspective of comprehensive benefits, the (AB) gradient data can be sent directly without compression processing, or other compression processing methods can be used to compress the (AB) gradient data and then send them.

[0175] 605. After the cloud device receives B quantization sets corresponding to B gradient data sent by the terminal device, for each first quantization set in the B quantization sets, use the first codebook corresponding to the first quantization set to dequantize each quantization representation in the first quantization set to obtain a first gradient segment corresponding to each quantization representation.

[0176] The first gradient segment includes d′ elements, the number of elements in the quantized representation is less than d′, the first codebook is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, d′ and m are positive integers greater than 2, and m≥d′.

[0177] After receiving the quantized representation, the cloud device will use the same codebook as the terminal device during the quantization conversion to perform dequantization, that is, quantization recovery.

[0178] 606. The cloud device concatenates the first gradient segments corresponding to each quantization representation in the first quantization set to obtain first gradient data corresponding to the first quantization set.

[0179] The first gradient data includes d elements, d is a positive integer, and d is divisible by d′.

[0180] The splicing process is a sorting process, in which each gradient segment is arranged in the order in which the terminal device divides the gradient segment, thereby restoring the first gradient data.

[0181] 607. The cloud device performs gradient aggregation on the N first gradient data of the first weight for a first weight among the B weights corresponding to the B gradient data to obtain gradient aggregation data of the first weight.

[0182] The plurality of first gradient data of the first weights correspond to N terminal devices, where N is an integer greater than 1.

[0183] Gradient polymerization can be understood by referring to the aforementioned description of the gradient polymerization process, which will not be repeated here.

[0184] 608. The cloud device sends the gradient aggregation data of each of the B weights to the terminal device.

[0185] 609. The terminal device updates the weight according to the gradient aggregation data.

[0186] The updating weights can be understood by referring to the above description of the updating weights process, which will not be repeated here.

[0187] The embodiment of the present application divides the gradient data to be transmitted into gradient segments, and then performs quantization conversion according to the code book to obtain a quantized representation, and the number of elements in the quantized representation is less than the number of elements in each gradient segment. In this way, the amount of quantized data is further reduced on the basis of quantization compression, thereby improving the compression ratio. In addition, because the compression ratio of the quantized representation is improved, the communication overhead when the terminal device transmits the quantized representation to the cloud device is also reduced.

[0188] In the embodiment of the present application, a codebook is used both in the quantization conversion and in the inverse quantization process. The codebook can be generated by the terminal device and the cloud device respectively according to the same random seed, or it can be generated by the cloud device and then sent to the terminal device. These two situations are introduced respectively below.

[0189] 1. The terminal device and cloud device generate codebooks by themselves.

[0190] The terminal device determines, according to the data amount of each of the gradient data, that B gradient data among the A gradient data need to be quantized;

[0191] The terminal device determines d′ and m for each first gradient data in the B gradient data according to the number of elements d of the first gradient data, where d′ is 2 to the pth power, m is 2 to the qth power, p and q are both positive integers, and q≥p;

[0192] The terminal device randomly generates the first matrix with d′ rows and m columns or the second matrix with d columns and m rows for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0193] For each first gradient data among the B gradient data, the cloud device randomly generates the first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook for the first gradient data, where each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0194] The terminal device can determine a data volume threshold that needs to be quantized from the perspective of comprehensive benefits. For example, if the data volume threshold is set to 512 elements, if the number of elements in the gradient data is greater than or equal to 512, it means that quantization conversion is required. If the number of elements in the gradient data is less than 512, it means that quantization conversion is not required. The terminal device only needs to generate a codebook for the gradient data that needs to be quantized. This can reduce the computational overhead caused by generating the codebook and improve the comprehensive benefits of computational overhead and communication overhead. In order to ensure that the cloud device can restore the gradient data normally, the random seed used by the terminal device to generate the codebook is the same as that of the cloud device. The random seed can be a numerical value, for example: random seed s = 798, which is usually set by the cloud device and then synchronized to each terminal for merging. The codebook of the gradient data can be randomly generated by inputting the random seed into the random function. Because the random seeds are the same, the codebook generated by the terminal device and the codebook generated by the cloud device are the same.

[0195] 2. The cloud device generates a codebook and then sends it to the terminal device.

[0196] For each first gradient data among the B gradient data, the cloud device randomly generates the first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook for the first gradient data, where each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0197] The cloud device sends the codebooks corresponding to the B gradient data to the terminal device. Correspondingly, the terminal device receives the codebooks corresponding to the B gradient data sent by the cloud device, wherein the codebooks corresponding to the B gradient data include the first codebook.

[0198] In this implementation, the cloud device sends the generated codebook to the terminal device, which can reduce the computing overhead of the terminal device.

[0199] In the embodiment of the present application, the above step 603: the terminal device uses the first codebook of the first gradient data to perform quantization conversion on each gradient segment of the at least two gradient segments to obtain a quantized representation corresponding to each gradient segment, may include:

[0200] The terminal device determines a target code for a first gradient segment from the first codebook, where the target code is a column in the first matrix or a row in the second matrix, and the first gradient segment is any one of the at least two gradient segments;

[0201] The terminal device determines the pseudo modulus length and code index of the target code;

[0202] The terminal device determines the pseudo modulus length and the code index as a quantized representation of the first gradient segment.

[0203] In the embodiment of the present application, the pseudo-modulus is relative to the modulus, and the pseudo-modulus is not obtained according to the usual mathematical calculation method of the vector modulus, but is calculated through the code book and the gradient segment. The code index represents the index of a column of the first matrix, for example: index(5) represents the fifth column. If there are multiple code books, the code book mark is also required, such as index(3,5), which represents the fifth column of code book 3. Alternatively, the code index represents the index of a row of the second matrix, for example: index(3,6) represents the sixth row of code book 3. If the modulus is represented by u and the code index index(c), the quantized representation can be (u, index(c)). In the embodiment of the present application, a gradient segment is converted into only two elements including the pseudo-modulus and the code index, which greatly reduces the amount of data, improves the compression ratio of the gradient data, and reduces the communication overhead of transmitting the gradient data.

[0204] There are two implementation methods for the step of: determining a target code for the first gradient segment from the first codebook.

[0205] See also Figure 8 , Method 1 may include:

[0206] 701. Determine a vector norm of the first gradient segment.

[0207] 702. Determine whether the vector modulus is equal to 0. If it is equal to 0, execute step 703; if it is not equal to 0, execute step 704.

[0208] The vector magnitude is the sum of the squares of the elements on the first gradient segment and then the square root.

[0209] 703. If the vector modulus is equal to 0, determine any column in the first matrix or any row in the second matrix as the target code.

[0210] If the vector modulus is equal to 0, it means that all elements are 0, so you only need to randomly select a column or a row to form a format that conforms to the quantitative representation. For example, you can select the first column or the first row.

[0211] If the first gradient segment is represented by g, then If||g||2=0, then return(0,c1). ||g||2 represents the sum of the squares of the elements in the first gradient segment and then the square root, which is the vector modulus of the first gradient segment g. For the first gradient segment g, if the vector modulus is equal to 0, the vector representation (0, index(x,1)) can be output, indicating that the vector modulus = 0, the code index is the first row or column in the xth codebook, and x represents the index of the codebook, which can be the identifier or serial number of the codebook. Of course, you don't need to select the first column or the first row, just randomly select a column or a row.

[0212] 704. If the vector modulus is not equal to 0, determine a first coefficient vector of the first gradient segment according to the first codebook and the first gradient segment.

[0213] The first coefficient vector includes m elements.

[0214] If the first codebook is represented by C, the first gradient segment is represented by g, and the first coefficient vector is represented by p, then p = C T (CC T ) -1 g, where C T Represents the transposed matrix of C. P contains m elements.

[0215] 705. Normalize each element in the first coefficient vector to obtain a second coefficient vector.

[0216] The second coefficient vector includes m normalized elements.

[0217] The second coefficient vector is express, It includes m normalized elements, and element normalization can be understood as: in represents the bth element among m normalized elements, |p b | represents the absolute value of the bth element in the first coefficient vector p, and ||p||1 represents the sum of the absolute values ​​of each element in the m elements of the first coefficient vector p.

[0218] 706. A roulette wheel group selection strategy is used for the normalized m elements to determine that the i-th column in the first matrix or the i-th row in the second matrix corresponding to the i-th element is a target code.

[0219] The i-th element is an element selected using the selection strategy of the roulette wheel group.

[0220] The selection strategy of the roulette group can be considered that each normalized element has its own probability. The probability of each element is large or small. The sum of all probabilities is equal to 1. The probability of being selected is also large for the element with a large probability, and the probability of being selected is also small for the element with a small probability. However, the final selection result is still based on the element selected after the roulette group rotates. If the selected element is Then the code of the column or row corresponding to the bth element is selected as the target code. Of course, if the i-th normalized element is selected, the code of the column or row corresponding to the i-th element is selected as the target code.

[0221] If the target code selected in this step is c b : Then the pseudo-module length Where ||p||1 represents the sum of the absolute values ​​of the elements in the first coefficient. The Sign function is a symbol. If the target code is c b , then the pseudo modulus represents the product of the normalized bth element and the sum of the absolute values ​​of the elements in the first coefficient.

[0222] The code index is index(x, b), return(u, index(x, b)), which means the output quantized representation (u, index(x, b)).

[0223] Method 2 may include:

[0224] Determine a projection vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the projection vector includes m elements;

[0225] Determine that the i-th column in the first codebook or the i-th row in the second matrix corresponding to the i-th element is a target code, and the absolute value of the i-th element is the largest among the absolute values ​​of the m elements.

[0226] In this possible implementation, if the first codebook is represented by C, the first gradient segment is represented by g, and the projection vector is represented by p, then p=C T g, where CT represents the transposed matrix of C. The projection vector p includes m elements; |p i | represents the absolute value of the i-th element in the m elements. If i=5, it means that the 5th column in the first codebook or the 5th row in the second matrix is ​​selected as the target code.

[0227] If the target code c selected in this step i , then the pseudo-module length u=p i , that is, the pseudo-modulus length is equal to the i-th element in the projection vector p.

[0228] The code index is index(x, i), return(u, index(x, i)), which means the output quantized representation (u, index(x, i)).

[0229] If there is only one codebook, the code index may not include the codebook index x.

[0230] When dequantizing, the cloud device can: for each first quantization representation in the first quantization set, determine the target code of the first quantization representation from the first codebook according to the code index of the first quantization representation; and recover the first gradient segment corresponding to the first quantization representation according to the pseudo modulus length in the first quantization representation and the target code.

[0231] That is, the cloud device can determine the target code c through index(i) i , and then use the pseudo modulus length u to multiply each element in the target code to recover the corresponding gradient fragment.

[0232] The cloud device receives the quantized representation sent by the terminal device, performs a reverse operation on the quantized representation, and obtains the approximate value of the gradient data. Taking the gradient of FC2-weight as an example, the gradient has 75 quantized gradient segments. For each gradient segment, its quantized representation is set to (0.523641, (3, 25)), which means that the 25th code c is selected from codebook 3. The approximate value of the gradient segment is 0.523641*c. The approximate representation of the FC2-weight gradient data can be obtained by splicing the approximate values ​​of the 75 gradient segments.

[0233] It should be noted that the codebook in the above example is illustrated by taking the structure of the first matrix with d′ rows and m columns as an example. If the codebook is the second matrix with d′ columns and m rows, it is necessary to first transpose C in the above formula, convert it from the form of the second matrix to the form of the first matrix, and then calculate it through the above corresponding formula.

[0234] To further save communication overhead, cloud devices can also:

[0235] Performing quantization conversion on the B weighted gradient aggregation data respectively to obtain respective quantization sets of the B gradient aggregation data;

[0236] Sending respective quantization sets of the B gradient aggregated data to the terminal device, wherein the quantization set of each gradient aggregated data includes a quantization representation corresponding to each gradient segment of the gradient aggregated data.

[0237] In this possible implementation, the cloud device may perform a quantization conversion on the gradient aggregation data to be sent to the terminal device similar to the quantization conversion performed by the terminal device on the gradient data, which may further reduce the communication overhead between the cloud device and the terminal device.

[0238] After the terminal device receives the quantized sets of the B gradient aggregate data corresponding to the B gradient data sent by the cloud device, it can use the above-mentioned cloud device inverse quantization process to restore the B gradient aggregate data, and then perform weight update.

[0239] At present, there are several mainstream compression methods in the process of gradient data transmission, namely Stochastic Gradient Descent (SGD), Quantized SGD (QSGD), Sign SGD (SignSGD) and Ternary Gradients (TernGrad). The quantization conversion of the embodiment of the present application is also a gradient compression method, which can be called the gradient compression method of Hyper-Sphere Quantization (HSQ).

[0240] In the process of developing HSQ, engineers tested three popular deep learning models, VGG19, ResNet50, and ResNet101, on the same dataset in a simulated environment for the above compression methods. Table 2 lists the compression rate (pure SGD as the baseline) and the convergence accuracy of the algorithm, where d′ represents the gradient fragment size.

[0241] Table 2: Compression ratio and convergence accuracy of several compression methods

[0242]

[0243] As can be seen from Table 2 above, when d′=8 and d′=16, HSQ can achieve higher convergence accuracy than SGD, with compression ratios of 18.3 and 36.6 times, respectively. When d′=64, the compression ratio of HSQ is significantly higher than that of other algorithms, and the reduction in convergence accuracy is very small. Compared with several existing gradient compression methods, the method of this application has more advantages in gradient compression ratio and convergence accuracy.

[0244] The above describes the model training system based on terminal-cloud combination and the method of data processing in model training. The terminal device and cloud device provided in the embodiment of the present application are introduced below in conjunction with the accompanying drawings.

[0245] like Fig. 9 As shown, an embodiment of the terminal device 80 provided in the embodiment of the present application may include:

[0246] The processing unit 801 is used for:

[0247] Get A gradient data of the model to be trained, where A is a positive integer;

[0248] For each first gradient data in the B gradient data, the first gradient data is divided into at least two gradient segments, the first gradient data includes d elements, the B gradient data are included in the A gradient data, B is a positive integer and B≤A, each gradient segment includes d′ elements, d′ is a positive integer greater than 2, and d is divisible by d′;

[0249] Using a first codebook of the first gradient data, quantize and transform each gradient segment of the at least two gradient segments to obtain a quantized representation corresponding to each gradient segment, wherein the number of elements in the quantized representation is less than d′, and the first codebook is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, where m is a positive integer greater than 2, and m≥d′;

[0250] The sending unit 802 is used to send B quantization sets corresponding to the B gradient data to the cloud device, wherein the first quantization set corresponding to the first gradient data includes the quantization representation corresponding to each gradient segment.

[0251] The embodiment of the present application divides the gradient data to be transmitted into gradient segments, and then performs quantization conversion according to the code book to obtain a quantized representation, and the number of elements in the quantized representation is less than the number of elements in each gradient segment. In this way, the amount of quantized data is further reduced on the basis of quantization compression, thereby improving the compression ratio. In addition, because the compression ratio of the quantized representation is improved, the communication overhead when the terminal device transmits the quantized representation to the cloud device is also reduced.

[0252] In a possible embodiment, the sending unit 802 is further configured to send (AB) gradient data among the A gradient data except the B gradient data to the cloud device.

[0253] In this possible implementation, the (AB) gradient data may have very few elements, and the quantization conversion process will consume computing resources. From the perspective of comprehensive benefits, the (AB) gradient data can be directly sent without compression processing, or other compression processing methods can be used to compress the (AB) gradient data and then send them.

[0254] In a possible embodiment, the processing unit 801 is further configured to:

[0255] According to the data amount of each gradient data, determining that B gradient data among the A gradient data need to be quantized;

[0256] For each first gradient data in the B gradient data, determine d′ and m according to the number of elements d of the first gradient data, where d′ is 2 to the pth power, m is 2 to the qth power, p and q are both positive integers, and q≥p;

[0257] The first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows is randomly generated for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0258] In this possible implementation, a data volume threshold that needs to be quantized can be determined from the perspective of comprehensive benefits. For example, the data volume threshold is set to 512 elements. If the number of elements in the gradient data is greater than or equal to 512, it means that quantization conversion is required. If the number of elements in the gradient data is less than 512, it means that quantization conversion is not required. The terminal device only needs to generate a codebook for the gradient data that needs to be quantized. This can reduce the computational overhead caused by generating the codebook and improve the comprehensive benefits of computational overhead and communication overhead. In order to ensure that the cloud device can restore the gradient data normally, the random seed used by the terminal device to generate the codebook is the same as that of the cloud device. The random seed can be a numerical value. By inputting the random seed into the random function, a codebook for gradient data can be randomly generated.

[0259] In a possible embodiment, the receiving unit 803 is configured to receive a codebook corresponding to each of the B gradient data sent by the cloud device, wherein the codebook corresponding to each of the B gradient data includes the first codebook.

[0260] In this possible implementation, the terminal device does not need to generate a codebook by itself. The codebook can be generated by the cloud device and then sent to the terminal device, which can reduce the computing overhead of the terminal device.

[0261] In a possible embodiment, the processing unit 801 is used to:

[0262] Determine a target code for a first gradient segment from the first codebook, the target code being a column in the first matrix or a row in the second matrix, the first gradient segment being any one of the at least two gradient segments;

[0263] Determining a pseudo-modulus length and a code index of the target code;

[0264] The pseudo modulus length and the code index are determined as a quantized representation of the first gradient segment.

[0265] In a possible embodiment, the processing unit 801 is used to:

[0266] Determining a vector modulus of the first gradient segment;

[0267] If the vector modulus is equal to 0, any column in the first matrix or any row in the second matrix is ​​determined as the target code.

[0268] In a possible embodiment, the processing unit 801 is further configured to:

[0269] If the vector modulus is not equal to 0, determining a first coefficient vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the first coefficient vector includes m elements;

[0270] Normalizing each element in the first coefficient vector to obtain a second coefficient vector, wherein the second coefficient vector includes m normalized elements;

[0271] A roulette wheel group selection strategy is adopted for the normalized m elements to determine that the i-th column in the first matrix or the i-th row in the second matrix corresponding to the i-th element is the target code, and the i-th element is the element selected by the roulette wheel group selection strategy.

[0272] In a possible embodiment, the processing unit 801 is used to:

[0273] Determine a projection vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the projection vector includes m elements;

[0274] Determine that the i-th column in the first codebook or the i-th row in the second matrix corresponding to the i-th element is a target code, and the absolute value of the i-th element is the largest among the absolute values ​​of the m elements.

[0275] In a possible embodiment, the receiving unit 803 is used to receive respective quantized sets of B gradient aggregate data corresponding to the B gradient data sent by the cloud device, wherein the quantized set of each gradient aggregate data includes a quantized representation corresponding to each gradient segment of the gradient aggregate data.

[0276] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned terminal device 80 are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous description of the present application, and will not be repeated here.

[0277] See also Fig.10 An embodiment of the cloud device 90 provided in the embodiment of the present application may include:

[0278] A receiving unit 901 is configured to receive B quantization sets corresponding to B gradient data sent by a terminal device, wherein each quantization set includes a quantization representation corresponding to each gradient segment of the corresponding gradient data;

[0279] The processing unit 902 is used for:

[0280] For each first quantization set in the B quantization sets, use a first codebook corresponding to the first quantization set to dequantize each quantization representation in the first quantization set to obtain a first gradient segment corresponding to each quantization representation, the first gradient segment includes d′ elements, the number of elements in the quantization representation is less than d′, the first codebook is a first matrix with d′ rows and m columns or a second matrix with d′ columns and m rows, d′ and m are positive integers greater than 2, and m≥d′;

[0281] splicing the first gradient segments corresponding to each quantization representation in the first quantization set to obtain first gradient data corresponding to the first quantization set, wherein the first gradient data includes d elements, d is a positive integer, and d is divisible by d′;

[0282] For a first weight among B weights corresponding to B gradient data, performing gradient aggregation on N first gradient data of the first weight to obtain gradient aggregated data of the first weight, wherein the plurality of first gradient data of the first weight corresponds to N terminal devices, where N is an integer greater than 1;

[0283] The sending unit 903 is used to send the gradient aggregation data of each of the B weights to the terminal device.

[0284] In an embodiment of the present application, after receiving the quantized representation, the cloud device will use the same codebook as the terminal device during the quantization conversion to perform inverse quantization, that is, quantization recovery, and then perform gradient aggregation. The process of gradient aggregation refers to the cloud device aggregating the gradients of the same weight received from multiple terminal devices. After the cloud device sends the aggregated gradient aggregation data to the terminal device, the terminal device can update the weights, so that the model to be trained can be further converged. The weight update process can be to subtract the gradient aggregation data calculated in this round from the current weight of this round to obtain the updated weight, and the updated weight is used for the next round of training. Because the cloud device receives the quantized representation after the quantization conversion of the terminal device from the terminal device, and because the number of elements in the quantized representation is less than the number of elements d′ in each gradient segment, only a very small communication overhead is required to achieve the reception of gradient data.

[0285] In a possible embodiment, the processing unit 902 is further used to: for each first gradient data in the B gradient data, randomly generate the first matrix with d′ rows and m columns or the second matrix with d′ columns and m rows for the first gradient data according to a random seed, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

[0286] In a possible embodiment, the sending unit 903 is further configured to send a codebook corresponding to each of the B gradient data to the terminal device.

[0287] In a possible embodiment, the processing unit 902 is configured to:

[0288] For each first quantization representation in the first quantization set, determine a target code of the first quantization representation from the first codebook according to a code index of the first quantization representation;

[0289] A first gradient segment corresponding to the first quantized representation is restored according to the pseudo modulus length in the first quantized representation and the target code.

[0290] In a possible embodiment, the processing unit 902 is further configured to: perform quantization conversion on the B weighted gradient aggregation data respectively to obtain respective quantization sets of the B gradient aggregation data;

[0291] The sending unit 903 is configured to send the quantization sets of the B gradient aggregated data to the terminal device, wherein the quantization set of each gradient aggregated data includes a quantization representation corresponding to each gradient segment of the gradient aggregated data.

[0292] In this possible embodiment, the cloud device may perform a quantization conversion on the gradient aggregation data to be sent to the terminal device similar to the quantization conversion performed by the terminal device on the gradient data, which can further reduce the communication overhead between the cloud device and the terminal device.

[0293] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned cloud device 90 are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous description of the present application, and will not be repeated here.

[0294] like Fig.11 As shown, it is a structural diagram of another device of an embodiment of the present application, and the device is a terminal device, and the terminal device may include: a processor 1001 (such as a CPU), a memory 1002, a transmitter 1004 and a receiver 1003; the transmitter 1004 and the receiver 1003 are coupled to the processor 1001, and the processor 1001 controls the sending action of the transmitter 1004 and the receiving action of the receiver 1003. The memory 1002 may include a high-speed RAM memory, and may also include a non-volatile memory NVM, such as at least one disk memory. Various instructions may be stored in the memory 1002 to complete various processing functions and implement the method steps of the embodiment of the present application. Optionally, the terminal device involved in the embodiment of the present application may also include: a power supply 1005, and one or more of the communication ports 1006, Fig.11 The devices described in the embodiment may be connected via a communication bus or other connection methods, which is not limited in the embodiments of the present application. The receiver 1003 and the transmitter 1004 may be integrated in the transceiver of the terminal device, or may be independent receiving and transmitting antennas on the terminal device. The communication bus is used to realize the communication connection between the components. The above-mentioned communication port 1006 is used to realize the connection and communication between the terminal device and other peripherals.

[0295] In some embodiments, the processor 1001 in the terminal device may execute Fig. 9 The actions performed by the processing unit 801 in the terminal device can be performed by the receiver 1003 in the terminal device Fig. 9 The actions performed by the receiving unit 803 in the terminal device can be performed by the transmitter 1004 in the terminal device Fig. 9 The implementation principle and technical effect of the action performed by the sending unit 802 are similar and will not be repeated here.

[0296] The present application also provides a chip system, which includes a processor for supporting the above-mentioned terminal device to implement the functions involved, such as receiving or processing the data involved in the above-mentioned method embodiment. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the terminal device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0297] Fig.12 1 is a possible logical structure diagram of the cloud device 110 involved in the above-mentioned embodiment provided in the embodiment of the present application. The cloud device 110 includes: a processor 1101, a communication port 1102, a memory 1103 and a bus 1104. The processor 1101, the communication port 1102 and the memory 1103 are interconnected via the bus 1104. In the embodiment of the present application, the processor 1101 is used to control and manage the actions of the cloud device 110. For example, the processor 1101 is used to execute Fig.10 The functions performed by the processing unit 902 in the cloud device 110 are as follows. The communication port 1102 is used to support the cloud device 110 to communicate. The memory 1103 is used to store the program code and data of the cloud device 110.

[0298] Among them, the processor 1101 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the contents disclosed in this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 1104 can be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0299] The present application also provides a chip system, which includes a processor for supporting the above-mentioned cloud device to implement its functions, such as receiving or processing the data involved in the above-mentioned method embodiment. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the terminal device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0300] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When at least one processor of the device executes the computer-executable instructions, the device executes the above Figures 6 to 8 Some of the methods described in the examples.

[0301] In another embodiment of the present application, a computer program product is also provided. The computer program product includes computer-executable instructions, which are stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and at least one processor executes the computer-executable instructions so that the device performs the above Figures 6 to 8 Some of the methods described in the examples.

[0302] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0303] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0304] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0305] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0306] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0307] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0308] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for data processing, characterized in that: include: Get A gradient data of the model to be trained, where A is a positive integer; For each first gradient data in the B gradient data, the first gradient data is divided into at least two gradient segments, the first gradient data includes d elements, the B gradient data are included in the A gradient data, B is a positive integer and B≤A, and each gradient segment includes d ′ elements, d ′ is a positive integer greater than 2, and d can be d ′ divisibility; Using the first codebook of the first gradient data, each gradient segment of the at least two gradient segments is quantized and converted to obtain a quantized representation corresponding to each gradient segment, wherein the number of elements in the quantized representation is less than d ′ , the first codebook is d ′ The first matrix with m rows and columns or d ′ The second matrix with m columns and m rows, where m is a positive integer greater than 2 and m≥d'; B quantization sets corresponding to the B gradient data are sent to a cloud device, wherein a first quantization set corresponding to the first gradient data includes a quantization representation corresponding to each gradient segment.

2. The method according to claim 1, characterized in that The method further comprises: According to the data amount of each gradient data, determining that B gradient data among the A gradient data need to be quantized; For each first gradient data in the B gradient data, d is determined according to the number of elements d of the first gradient data. ′ and m, d ′ is 2 to the power of p, m is 2 to the power of q, p and q are both positive integers, and q ≥ p; According to the random seed, d is randomly generated for the first gradient data ′ The first matrix of m rows and columns or d ′ The second matrix has m columns and m rows, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

3. The method according to claim 1, characterized in that The method further comprises: Receive codebooks corresponding to the B gradient data respectively sent by the cloud device, wherein the codebooks corresponding to the B gradient data respectively include the first codebook.

4. The method according to any one of claims 1 to 3, characterized in that: The using the first codebook of the first gradient data to perform quantization conversion on each of the at least two gradient segments to obtain a quantization representation corresponding to each of the gradient segments includes: Determine a target code for a first gradient segment from the first codebook, the target code being a column in the first matrix or a row in the second matrix, the first gradient segment being any one of the at least two gradient segments; Determining a pseudo-modulus length and a code index of the target code; The pseudo modulus length and the code index are determined as a quantized representation of the first gradient segment.

5. The method according to claim 4, characterized in that The determining a target code for a first gradient segment from the first codebook comprises: Determining a vector modulus of the first gradient segment; If the vector modulus is equal to 0, any column in the first matrix or any row in the second matrix is ​​determined as the target code.

6. The method according to claim 5, characterized in that The method further comprises: If the vector modulus is not equal to 0, determining a first coefficient vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the first coefficient vector includes m elements; Normalizing each element in the first coefficient vector to obtain a second coefficient vector, wherein the second coefficient vector includes m normalized elements; A roulette wheel selection strategy is adopted for the normalized m elements to determine that the i-th column in the first matrix or the i-th row in the second matrix corresponding to the i-th element is the target code, and the i-th element is the element selected by the roulette wheel selection strategy.

7. The method according to claim 4, characterized in that The determining a target code for a first gradient segment from the first codebook comprises: Determine a projection vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the projection vector includes m elements; Determine that the i-th column in the first codebook or the i-th row in the second matrix corresponding to the i-th element is a target code, and the absolute value of the i-th element is the largest among the absolute values ​​of the m elements.

8. The method according to any one of claims 1 to 3, characterized in that: Also includes: Receive respective quantized sets of B gradient aggregate data corresponding to the B gradient data sent by the cloud device, wherein the quantized set of each gradient aggregate data includes a quantized representation corresponding to each gradient segment of the gradient aggregate data.

9. A method of data processing, characterized in that: include: Receiving B quantization sets corresponding to the B gradient data sent by the terminal device, wherein each quantization set includes a quantization representation corresponding to each gradient segment of the corresponding gradient data; For each first quantization set in the B quantization sets, dequantize each quantization representation in the first quantization set using a first codebook corresponding to the first quantization set to obtain a first gradient segment corresponding to each quantization representation, wherein the first gradient segment includes d ′ elements, the number of elements in the quantized representation is less than d ′ , the first codebook is d ′ The first matrix with m rows and columns or d ′ The second matrix with m columns and m rows, d ′ and m is a positive integer greater than 2, and m≥d'; The first gradient segments corresponding to each quantization representation in the first quantization set are spliced ​​to obtain first gradient data corresponding to the first quantization set, wherein the first gradient data includes d elements, d is a positive integer, and d can be divided by d ′ divisibility; For a first weight among B weights corresponding to B gradient data, performing gradient aggregation on N first gradient data of the first weight to obtain gradient aggregated data of the first weight, wherein the plurality of first gradient data of the first weight corresponds to N terminal devices, where N is an integer greater than 1; Sending gradient aggregation data of each of the B weights to the terminal device.

10. The method according to claim 9, characterized in that The method further comprises: For each first gradient data in the B gradient data, randomly generate d for the first gradient data according to a random seed ′ The first matrix of m rows and columns or d ′ The second matrix has m columns and m rows, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

11. The method according to claim 10, characterized in that The method further comprises: Sending codebooks corresponding to the B gradient data to the terminal device.

12. The method according to any one of claims 9 to 11, characterized in that: The dequantizing each quantized representation in the first quantized set using the first codebook corresponding to the first quantized set to obtain a first gradient segment corresponding to each quantized representation includes: For each first quantization representation in the first quantization set, determine a target code of the first quantization representation from the first codebook according to a code index of the first quantization representation; A first gradient segment corresponding to the first quantized representation is restored according to the pseudo modulus length in the first quantized representation and the target code.

13. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Performing quantization conversion on the gradient aggregation data of the B weights respectively to obtain respective quantization sets of the gradient aggregation data of the B weights; The sending the gradient aggregation data of each of the B weights to the terminal device includes: Sending respective quantized sets of the B weighted gradient aggregation data to the terminal device, wherein each quantized set of gradient aggregation data includes a quantized representation corresponding to each gradient segment of the gradient aggregation data.

14. A terminal device, comprising a transceiver, a processor and a memory, wherein the processor is coupled to the memory, characterized in that: The memory is used to store programs; The processor is used to: Get A gradient data of the model to be trained, where A is a positive integer; For each first gradient data in the B gradient data, the first gradient data is divided into at least two gradient segments, the first gradient data includes d elements, the B gradient data are included in the A gradient data, B is a positive integer and B≤A, and each gradient segment includes d ′ elements, d ′ is a positive integer greater than 2, and d can be d ′ divisibility; Using the first codebook of the first gradient data, each gradient segment of the at least two gradient segments is quantized and converted to obtain a quantized representation corresponding to each gradient segment, wherein the number of elements in the quantized representation is less than d ′ , the first codebook is d ′ The first matrix with m rows and columns or d ′ The second matrix with m columns and m rows, where m is a positive integer greater than 2 and m≥d'; The transceiver is used to send B quantization sets corresponding to the B gradient data to the cloud device, wherein the first quantization set corresponding to the first gradient data includes the quantization representation corresponding to each gradient segment.

15. The terminal device according to claim 14, characterized in that: The processor is further configured to: According to the data amount of each gradient data, determining that B gradient data among the A gradient data need to be quantized; For each first gradient data in the B gradient data, d is determined according to the number of elements d of the first gradient data. ′ and m, d ′ is 2 to the power of p, m is 2 to the power of q, p and q are both positive integers, and q ≥ p; According to the random seed, d is randomly generated for the first gradient data ′ The first matrix of m rows and columns or d ′ The second matrix has m columns and m rows, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

16. The terminal device according to claim 14, characterized in that: The transceiver is used to receive the codebooks corresponding to the B gradient data respectively sent by the cloud device, and the codebooks corresponding to the B gradient data respectively include the first codebook.

17. The terminal device according to any one of claims 14 to 16, characterized in that: The processor is used to: Determine a target code for a first gradient segment from the first codebook, the target code being a column in the first matrix or a row in the second matrix, the first gradient segment being any one of the at least two gradient segments; Determining a pseudo-modulus length and a code index of the target code; The pseudo modulus length and the code index are determined as a quantized representation of the first gradient segment.

18. The terminal device according to claim 17, characterized in that: The processor is used to: Determining a vector modulus of the first gradient segment; If the vector modulus is equal to 0, any column in the first matrix or any row in the second matrix is ​​determined as the target code.

19. The terminal device according to claim 18, characterized in that: The processor is further configured to: If the vector modulus is not equal to 0, determining a first coefficient vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the first coefficient vector includes m elements; Normalizing each element in the first coefficient vector to obtain a second coefficient vector, wherein the second coefficient vector includes m normalized elements; A roulette wheel selection strategy is adopted for the normalized m elements to determine that the i-th column in the first matrix or the i-th row in the second matrix corresponding to the i-th element is the target code, and the i-th element is the element selected by the roulette wheel selection strategy.

20. The terminal device according to claim 17, characterized in that: The processor is used to: Determine a projection vector of the first gradient segment according to the first codebook and the first gradient segment, wherein the projection vector includes m elements; Determine that the i-th column in the first codebook or the i-th row in the second matrix corresponding to the i-th element is a target code, and the absolute value of the i-th element is the largest among the absolute values ​​of the m elements.

21. The terminal device according to any one of claims 14 to 16, characterized in that: The transceiver is used to receive respective quantized sets of B gradient aggregate data corresponding to the B gradient data sent by the cloud device, wherein the quantized set of each gradient aggregate data includes a quantized representation corresponding to each gradient segment of the gradient aggregate data.

22. A cloud device comprising a communication port, a processor and a memory, wherein the processor is coupled to the memory, wherein: The memory is used to store programs; The communication port is used to receive B quantization sets corresponding to B gradient data sent by the terminal device, wherein each quantization set includes a quantization representation corresponding to each gradient segment of the corresponding gradient data; The processor is used to: For each first quantization set in the B quantization sets, dequantize each quantization representation in the first quantization set using a first codebook corresponding to the first quantization set to obtain a first gradient segment corresponding to each quantization representation, wherein the first gradient segment includes d ′ elements, the number of elements in the quantized representation is less than d ′ , the first codebook is d ′ The first matrix with m rows and columns or d ′ The second matrix with m columns and m rows, d ′ and m is a positive integer greater than 2, and m≥d'; The first gradient segments corresponding to each quantization representation in the first quantization set are spliced ​​to obtain first gradient data corresponding to the first quantization set, wherein the first gradient data includes d elements, d is a positive integer, and d can be divided by d ′ divisibility; For a first weight among B weights corresponding to B gradient data, performing gradient aggregation on N first gradient data of the first weight to obtain gradient aggregated data of the first weight, wherein the plurality of first gradient data of the first weight corresponds to N terminal devices, where N is an integer greater than 1; The communication port is used to send the gradient aggregation data of each of the B weights to the terminal device.

23. The cloud device according to claim 22, characterized in that: The processor is further configured to: for each first gradient data in the B gradient data, randomly generate d for the first gradient data according to a random seed ′ The first matrix of m rows and columns or d ′ The second matrix has m columns and m rows, and each Gaussian vector in the first matrix or the second matrix is ​​scaled to a modulus of 1 to obtain a first codebook of the first gradient data, each column in the first matrix is ​​a Gaussian vector, and each row in the second matrix is ​​a Gaussian vector.

24. The cloud device according to claim 23, characterized in that: The communication port is further used to send the codebooks corresponding to the B gradient data to the terminal device.

25. The cloud device according to any one of claims 22 to 24, characterized in that: The processor is used to: For each first quantization representation in the first quantization set, determine a target code of the first quantization representation from the first codebook according to a code index of the first quantization representation; A first gradient segment corresponding to the first quantized representation is restored according to the pseudo modulus length in the first quantized representation and the target code.

26. The cloud device according to any one of claims 22 to 24, characterized in that: The processor is further configured to perform quantization conversion on the gradient aggregation data of the B weights respectively to obtain respective quantization sets of the gradient aggregation data of the B weights; The communication port is used to send the quantized sets of the gradient aggregation data of the B weights to the terminal device, wherein each quantized set of the gradient aggregation data includes a quantized representation corresponding to each gradient segment of the gradient aggregation data.

27. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 8, or causes the computer to execute the method according to any one of claims 9 to 13.

Images