Data processing device and method, and related equipment

Abstract

The invention discloses a data processing device and method and related equipment. The device comprises a first state machine, a second state machine, a third state machine and a multiplier-adder, wherein the first state machine determines target computing node information and a storage address based on a weight parameter updating instruction; gradient data returned by the second state machine; the multiply-adder performs dimension reduction on the gradient data to obtain a dimension reduction result, and the third state machine wraps the dimension reduction result and writes the dimension reduction result into a storage position corresponding to the target computing node so that the target computing node can call the dimension reduction result. By means of the data processing device, all the computing nodes only need to call the gradient data from the corresponding storage positions when updating the weight parameters, the training efficiency can be improved, and all the computing nodes do not need to communicate with external equipment.

Description

Technical Field

[0001] The present application relates to the field of distributed training, and in particular to a data processing device, method and related equipment. Background Art

[0002] In the field of natural language processing, large language models based on deep learning are often used. In the process of processing natural language based on deep learning models, the text to be processed is often input into the Embedding layer of the deep learning model, and the word vectors of the text to be processed are obtained based on the word vector embedding matrix of the Embedding layer, and then the output result is obtained based on the word vector and the large language model.

[0003] The accuracy of each word vector of the text to be processed based on the word vector embedding matrix of the Embedding layer directly determines the accuracy of the output results of the large language model. Therefore, the parameters of the word vector embedding matrix of the Embedding layer must be trained. However, the number of parameters of the Embedding layer is large, and it is difficult for a single computing node to store the complete parameters of the Embedding layer. Therefore, a distributed training method is often used to distribute the complete parameters of the Embedding layer in multiple computing nodes, and multiple computing nodes are trained in parallel.

[0004] In the prior art, when multiple computing nodes train the Embedding layer in parallel, it is necessary to use an external device based on the output results of this round, use a gradient descent algorithm, calculate the gradient of the weight parameters of the Embedding layer deployed on each node, and send the gradient of the weight parameters of the Embedding layer deployed on each node to each computing node, so that each computing node can update its own weight parameters until the gap between the output result and the target result is minimized. In each round of updating the weight parameters, each computing node needs to communicate with the external device to request the gradient data of the weight parameters updated in this round, which requires huge communication overhead and is inefficient. Summary of the invention

[0005] The main purpose of this application is to propose a data processing device, method and related equipment, which aims to solve the problem that when training deep learning models in parallel, each computing node needs to communicate with an external device to obtain the gradient data for updating the weight parameters in this round, which causes high communication overhead and low efficiency.

[0006] To achieve the above-mentioned purpose, the present application proposes a data processing device, comprising: being applied to a distributed training system, the distributed training system comprising a plurality of computing nodes and a memory module, each of the computing nodes being used to deploy a portion of operators of a model to be trained, and the memory module being used to store gradient data when weight parameters of the operators deployed by each of the computing nodes are updated; The data processing device comprises: a first state machine, a second state machine, a third state machine and a multiplier-adder, wherein the first state machine, the second state machine, the third state machine and the multiplier-adder are used to be deployed on any of the computing nodes, and the first state machine, the second state machine, the third state machine and the multiplier-adder are deployed on the same computing node; The first state machine is configured to: in response to receiving a weight parameter update instruction, determine, based on the weight parameter update instruction, the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module; and, based on the storage address, send a read request to the memory module, and send the target computing node information to the second state machine; The second state machine is configured to: receive the target computing node information and the gradient data returned by the memory module; and send the gradient data to the multiplier and adder, and send the target computing node information to the third state machine; The multiplier-adder is configured to: perform dimension reduction on the gradient data to obtain a dimension reduction result, and send the dimension reduction result to the third state machine; The third state machine is configured to: package the dimension reduction result based on the target computing node information, and write it into a storage location corresponding to the target computing node for the target computing node to call.

[0007] In an embodiment of the present application, when the weight parameter update instruction contains multiple target computing node information and corresponding multiple gradient data storage addresses, the first state machine is further configured to: send a read request to the memory module based on each storage address, and send a first identifier to the second state machine before the gradient data of each read request flows back, wherein the first identifier is used to identify the starting position of the reflow data corresponding to each read request; and According to the request order of each read request, the target computing node information corresponding to each read request is sent to the second state machine in sequence.

[0008] In the embodiment of the present application, the second state machine is further configured to: send each first identifier and the corresponding return data to the multiplier-adder in sequence; and send each target computing node information to the third state machine; The multiplier is further configured to: in response to receiving a first identifier, accumulate data after the first identifier, and when receiving a next first identifier, output a current accumulation result as the dimensionality reduction result to the third state machine.

[0009] The third state machine is further configured to: respectively package the corresponding dimensionality reduction results of each target computing node based on its information, and write each packaged dimensionality reduction result into its corresponding storage location.

[0010] In an embodiment of the present application, the first state machine is further configured to: when all the gradient data corresponding to each storage address included in the weight parameter update instruction are reflowed, send a second identifier to the second state machine, the second identifier being used to identify that all the gradient data corresponding to the weight parameter update instruction are reflowed; The second state machine is further configured to: after all first identifiers and corresponding reflow data are sent to the multiplier-adder, send the second identifier to the multiplier-adder; The multiplier-accumulator is further configured to: in response to receiving the second identifier, end the dimension reduction operation and send the current accumulation result to the third state machine.

[0011] In the embodiment of the present application, the first state machine is further configured to: when the gradient data in the storage address corresponding to any read request is empty, send a third identifier to the second state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; The second state machine is further configured to: send the third identifier and the corresponding reflow data to the multiplier-adder in sequence; The multiplier-accumulator is further configured to: in response to receiving the third identifier, skip all data after the third identifier until other identifiers are received.

[0012] The present application also proposes a data processing method, which is applied to the data processing device described in any of the above embodiments. The data processing method includes: In response to receiving a weight parameter update instruction, determining, based on the first state machine, a storage address in the memory module of target computing node information for updating weight parameters in the weight parameter update instruction and gradient data corresponding to the target computing node; and, based on the storage address, the first state machine sends a read request to the memory module, and sends the target computing node information to the second state machine based on the first state machine; Based on the second state machine, receiving the target computing node information and the gradient data returned by the memory module; and, based on the second state machine, sending the gradient data to the multiplier and adder, and sending the target computing node information to the third state machine; Based on the multiplier-adder, reducing the dimension of the gradient data to obtain a dimension reduction result, and sending the dimension reduction result to the third state machine; The dimension reduction result is packaged based on the third state machine and the target computing node information, and written into a storage location corresponding to the target computing node for the target computing node to call.

[0013] In an embodiment of the present application, when the weight parameter update instruction includes multiple target computing node information and corresponding multiple gradient data storage addresses, the data processing method further includes: Based on the first state machine, respectively sending read requests to the memory module based on each storage address, and sending a first identifier to the second state machine before the gradient data of each read request flows back, wherein the first identifier is used to identify the starting position of the backflow data corresponding to each read request; Based on the first state machine, target computing node information corresponding to each read request is sent to the second state machine in sequence according to the request order of each read request.

[0014] In the embodiment of the present application, the data processing method further includes: Based on the second state machine, each first identifier and the corresponding return data are sent to the multiplier-adder in sequence; and based on the second state machine, each target computing node information is sent to the third state machine; Based on the multiplier, the data after the first identifier is accumulated, and when the next first identifier is received, the current accumulation result is output to the third state machine as the dimensionality reduction result.

[0015] In the embodiment of the present application, the data processing method further includes: Based on the third state machine, the dimensionality reduction results corresponding to each target computing node are packaged based on the information of each target computing node, and the packaged dimensionality reduction results are written into their corresponding storage locations respectively.

[0016] In the embodiment of the present application, the data processing method further includes: When all the gradient data corresponding to the storage addresses included in the weight parameter update instruction are all reflowed, sending a second identifier to the second state machine based on the first state machine, where the second identifier is used to identify that all the gradient data corresponding to the weight parameter update instruction are all reflowed; After all the first identifiers and the corresponding reflow data are sent to the multiplier-adder, sending the second identifier to the multiplier-adder based on the second state machine; When the multiplier-adder receives the second identifier, the dimension reduction operation of the multiplier-adder is terminated, and the current accumulation result is sent to the third state machine.

[0017] In the embodiment of the present application, the data processing method further includes: When the gradient data in the storage address corresponding to any read request is empty, sending a third identifier to the second state machine based on the first state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; Sending the third identifier and the corresponding return data to the multiplier-adder in sequence based on the second state machine; When the multiplier-adder receives the third identifier, the multiplier-adder skips all data after the third identifier until other identifiers are received.

[0018] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the data processing methods described above is implemented.

[0019] An embodiment of the present application further provides a computing device, which includes a processor, and the processor is used to implement the data processing method described in any of the above embodiments when executing a computer program stored in a memory.

[0020] In the embodiment of the present application, by setting the first to third state machines and the multiplier-adder, the first state machine is used to send a read request to the memory module according to the weight parameter update instruction, the second state machine is used to receive the refluxed gradient data, the multiplier-adder reduces the dimension of the gradient data to obtain the dimensionality reduction result, and the third state machine stores the dimensionality reduction result after wrapping it in the storage position corresponding to each computing node. Therefore, only the computing node that deploys the first state, the second state machine, the third state machine and the multiplier-adder needs to communicate with the external device, and other computing nodes do not need to communicate with the external device, and can directly call the dimensionality reduction result from their corresponding storage positions. During the training process, the first state machine can continuously send read requests based on the storage address, the second state machine can continuously receive the refluxed data, the multiplier-adder can continuously reduce the dimension of the refluxed gradient data, and the third state machine can continuously store the wrapped dimensionality reduction result in the storage position corresponding to each computing node, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0022] Figure 1 It is a flow chart of a distributed training system in the prior art; Figure 2 A module diagram of a data processing device in an embodiment of the present application; Figure 3 This is a flow chart of a data processing device in one embodiment of the present application when processing a weight parameter update instruction; Figure 4 A step diagram of a data processing method in an embodiment of the present application; Figure 5 A module diagram of a computer-readable storage medium in an embodiment of the present application; Figure 6 A module diagram of a computing device in one embodiment of the present application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0026] According to the implementation manner of the present application, a data processing apparatus, method and related equipment are proposed.

[0027] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0028] The principles and spirit of the present application are explained in detail below with reference to several representative implementations of the present application.

[0029] Exemplary Devices like Figure 1 As shown, Figure 1 It is a distributed training system in the prior art, wherein the distributed training system includes computing nodes 1-6. Assume that the model to be trained is the Embedding layer of the deep learning model, and the trainable parameters in the Embedding layer are mainly word vector embedding matrices (operators), such as Figure 1 As shown, For example, the matrix of is distributed on computing nodes 1-6, and each computing node is only responsible for updating the parameters of part of the word vector embedding matrix. During training, the training sample X is input into computing nodes 1-6, and computing nodes 1-6 respectively obtain six word embedding vectors X1-X6 based on the word vector embedding matrix deployed by each node. The six word embedding vectors are merged to obtain the total word embedding vector Y, and then output to the Transformer model. The Transformer model can output the prediction result Z based on the word embedding vector Y. After obtaining the prediction result Z, the external device (such as other computing nodes dedicated to calculating gradients) is used to calculate the descent gradient of the weight parameter of each computing node according to the error between the prediction result Z and the expected result, and then the descent gradient of the weight parameter of each computing node is sent to each computing node for each computing node to update its own weight parameter. After multiple rounds of iterations, the error between the prediction result output by the Transformer model and the expected result is minimized, and the training ends.

[0030] In this process, in each iteration, each computing node needs to request the gradient data of this round from the external device, and there are often many computing nodes in the distributed training system ( Figure 1 This is just an illustrative example), in each round, each computing node needs to communicate with external devices, so there will be huge communication overhead and it also limits the efficiency of training.

[0031] Based on this, an embodiment of the present application proposes a data processing device to solve the problem of high communication overhead and low training efficiency between each computing node of a distributed training system and an external device that calculates gradient data when acquiring gradient data.

[0032] like Figure 2As shown, in an embodiment of the present application, the data processing device is applied to a distributed training system, the distributed training system includes a plurality of computing nodes and a memory module, each of the computing nodes is used to deploy a part of the operators of the model to be trained, and the memory module is used to store the gradient data when the weight parameters of the operators deployed by each of the computing nodes are updated; The data processing device comprises: a first state machine, a second state machine, a third state machine and a multiplier-adder, wherein the first state machine, the second state machine, the third state machine and the multiplier-adder are used to be deployed on any of the computing nodes, and the first state machine, the second state machine, the third state machine and the multiplier-adder are deployed on the same computing node; The first state machine is configured to: in response to receiving a weight parameter update instruction, determine, based on the weight parameter update instruction, the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module; and, based on the storage address, send a read request to the memory module, and send the target computing node information to the second state machine; The second state machine is configured to: receive the target computing node information and the gradient data returned by the memory module; and send the gradient data to the multiplier and adder, and send the target computing node information to the third state machine; The multiplier-adder is configured to: perform dimension reduction on the gradient data to obtain a dimension reduction result, and send the dimension reduction result to the third state machine; The third state machine is configured to: package the dimension reduction result based on the target computing node information, and write it into a storage location corresponding to the target computing node for the target computing node to call.

[0033] Among them, the distributed training system can be Figure 1 The training system shown may also be any other distributed training system having multiple computing nodes, and the data processing device in the embodiment of the present application may be deployed on any computing node of the distributed training system.

[0034] The operators of the models to be trained are deployed on each computing node of the distributed training system. For example, each computing node is deployed with Figure 1 The partial word embedding vector matrix of the Embedding layer is shown in the figure. The gradient data of the weight parameters of each computing node in each iteration round are given by Figure 1 The external device shown is calculated, and the present application does not limit the specific calculation method of the gradient data of each computing node in each iteration round.

[0035] In the embodiment of the present application, the first to third state machines are all composed of state registers and combinational logic circuits, and can perform state transitions according to pre-set states based on control signals. They are the control center for coordinating related signal actions and completing specific operations.

[0036] like Figure 3 As shown, Figure 3 This is a flowchart of data processing in an embodiment of the present application. Among them, in each iteration round, the gradient data required for updating the weight parameters of the operators deployed on each computing node are calculated by an external device and stored in a memory module. The memory module can be the local memory of the computing node deployed by the data processing device in the embodiment of the present application. The weight parameter update instruction contains the target computing node information that needs to update the weight parameters, and the storage address of the gradient data required for the target computing node to update the weight parameters in the memory module. The target computing node information can be the number of the target computing node, etc. The weight parameter update instruction is generally issued by the host in the distributed training system. The host will pre-write the target computing node information that needs to update the weight parameters and the storage address of the corresponding gradient data into the weight parameter update instruction. When the first state machine receives the weight parameter update instruction, the weight parameter update instruction is parsed to determine the target computing node information that needs to update the weight parameters and the storage address of the corresponding gradient data.

[0037] After determining the target computing node information and the storage address of the corresponding gradient data, the first state machine can send a read request to the memory module based on the storage address to request to read the gradient data at the storage address. When the memory module receives the read request, it returns the gradient data at the storage address to the second state machine according to the requested storage address. In addition, the first state machine also sends the target computing node information to the second state machine so that the second state machine can determine the ownership of the gradient data returned from the memory module.

[0038] When the second state machine receives the reflowed gradient data from the memory module and receives the target computing node information from the first state machine, the reflowed gradient data is sent to the multiplier-adder, and the target computing node information is sent to the third state machine, wherein the gradient data in the memory module is stored in the form of data inclusion, a data packet includes a data header, that is, multiple sub-data, and the gradient data is also reflowed in the form of each sub-data when it is reflowed. Therefore, when the second state machine sends it to the multiplier-adder, it is also sent to it in the form of each sub-data. After receiving each sub-data in the gradient data, the multiplier-adder accumulates the sub-data to obtain a dimensionality reduction result, and sends the dimensionality reduction result to the third state machine.

[0039] After receiving the dimensionality reduction result, the third state machine packages the dimensionality reduction result according to the target computing node. The package can be formed into a data packet in the form of a data header and data content. Among them, the data header can indicate the computing node and length to which the dimensionality reduction result belongs. After the third state machine packages the dimensionality reduction result, it can store it in the storage location corresponding to the target computing node, so as to facilitate the target computing node to call it directly from the storage location.

[0040] In the embodiment of the present application, by setting the first to third state machines and the multiplier-adder, the first state machine is used to send a read request to the memory module according to the weight parameter update instruction, the second state machine is used to receive the refluxed gradient data, the multiplier-adder reduces the dimension of the gradient data to obtain the dimensionality reduction result, and the third state machine stores the dimensionality reduction result after wrapping it in the storage position corresponding to each computing node. Therefore, only the computing node that deploys the first state, the second state machine, the third state machine and the multiplier-adder needs to communicate with the external device, and other computing nodes do not need to communicate with the external device, and can directly call the dimensionality reduction result from their corresponding storage positions. During the training process, the first state machine can continuously send read requests based on the storage address, the second state machine can continuously receive the refluxed data, the multiplier-adder can continuously reduce the dimension of the refluxed gradient data, and the third state machine can continuously store the wrapped dimensionality reduction result in the storage position corresponding to each computing node, thereby improving the training efficiency.

[0041] In the embodiment of the present application, a weight parameter update instruction may include gradient data requests of multiple computing nodes. Figure 1 Taking the distributed training system shown as an example, computing nodes 1-6 may all need to update weight parameters, that is, computing nodes 1-6 all need to obtain gradient data for updating weight parameters. At this time, the first state machine can parse the weight parameter update instruction to obtain six target computing nodes and six storage addresses. The six target computing nodes are computing nodes 1-6, respectively. Assuming that the six storage addresses are addr1-addr6, respectively, the first state machine sends six read requests to the memory module based on addr1-addr6 to request to read the gradient data corresponding to computing nodes 1-6 respectively. After receiving the six read requests, the memory module returns the six gradient data corresponding to the six read requests to the second state machine respectively.

[0042] It should be noted that the read request corresponds to the gradient data one by one, and each time the first state machine sends a read request to the memory module, a first identifier is sent to the second state, and the first identifier is used to separate the gradient data corresponding to each read request. The first identifier can be, for example, a binary code 10.

[0043] In the embodiment of the present application, it is assumed that the six read requests corresponding to the read addresses addr1-addr6 are read1-read6, and the return data corresponding to read1-read6 are data1-data6, then the second state machine will receive in sequence: 10, data1, 10, data2, 10, data3, 10, data4, 10, data5, 10, data6. At the same time, the second state machine will send them to the multiplier in the order of 10, data1, 10, data2, 10, data3, 10, data4, 10, data5, 10, data6. After the multiplier receives a binary code 10, it will output the previous accumulation result. If there is no accumulation result at present, it will not output it.

[0044] For example, for "10, data1, 10, data2, 10, data3, 10, data4, 10, data5, 10, data6", when the multiplier-adder receives the first 10, there is no accumulated result at present, so it is not output; after receiving the first 10, it is the data in data1. At this time, the multiplier-adder will accumulate the data in data1. When the second 10 is received, it proves that all the data in data1 have been accumulated. The accumulated result at this time is the dimensionality reduction result corresponding to data1, which can be output to the third state machine. Then, the multiplier-adder continues to receive the data of data2 and accumulates it until the third binary code 10 is received. The accumulated result output at this time is the dimensionality reduction result corresponding to data2, which can be output to the third state machine. By analogy, the 6 dimensionality reduction results corresponding to data1-data6 can be obtained respectively and output to the third state machine one by one.

[0045] In addition, it should be noted that after the first state machine parses the weight parameter update instruction to obtain the target calculation information, it can send the target calculation node information to the third state machine in sequence according to the order of read1-read6, or it can send the target calculation node information to the second state machine in sequence according to the order of read1-read6, and then the second state machine forwards it to the third state machine in sequence. Among them, the multiplier outputs six dimensionality reduction results in the order of data1-data6, corresponding to the six target calculation node information received in sequence by the third state machine, such as the numbering information of the calculation nodes 1-6, then the third state machine can wrap and store each dimensionality reduction result based on the order of the received dimensionality reduction results and the order of the target calculation nodes. For example, when storing, the dimensionality reduction result of data1 is wrapped and stored in the storage position corresponding to the calculation node 1, the dimensionality reduction result of data2 is wrapped and stored in the storage position corresponding to the calculation node 2, and so on, the dimensionality reduction result of data6 is wrapped and stored in the storage position corresponding to the calculation node 6.

[0046] In the embodiment of the present application, when a weight parameter update instruction contains multiple target calculation nodes, the gradient data corresponding to each target calculation node can be separated by setting a separator, thereby avoiding multiplier-accumulator errors.

[0047] Continue to refer to Figure 3 In an embodiment of the present application, the first state machine is further configured to: when all the gradient data corresponding to each storage address included in the weight parameter update instruction are all reflowed, send a second identifier to the second state machine, the second identifier being used to identify that all the gradient data corresponding to the weight parameter update instruction are all reflowed; The second state machine is further configured to: after all first identifiers and corresponding reflow data are sent to the multiplier-adder, send the second identifier to the multiplier-adder; The multiplier-accumulator is further configured to: in response to receiving the second identifier, end the dimension reduction operation and send the current accumulation result to the third state machine.

[0048] For example, still Figure 1 Taking the computing nodes 1-6 shown as an example, the target computing nodes in the weight parameter update instruction include computing nodes 1-6 and storage addresses addr1-addr6. When the first state machine sends read1-read6 to the memory module based on addr1-addr6, and when data6 corresponding to read6 is sent back, the first state machine also sends a second identifier to the second state machine, such as binary code 11. At this time, the data received by the second state machine is: 10, data1, 10, data2, 10, data3, 10, data4, 10, data5, 10, data6, 11, and sends them to the multiplier-adder in sequence. When the multiplier-adder receives "10, data1, 10, data2, 10, data3, 10, data4, 10, data5, 10, data6, 11" in sequence, the processing process from the first binary code 10 to data6 remains the same as the process in the above embodiment, and will not be repeated here; when the multiplier-adder receives the binary code 11, it means that the gradient data of this batch has been completely reduced in dimension, that is, the gradient data corresponding to all the target calculation nodes in the weight parameter update instruction have been completely reduced in dimension. At this time, the multiplier-adder outputs the current accumulation result, that is, the dimension reduction result of data6, to the third state machine, and then clears all data to prepare for processing the next weight parameter update instruction.

[0049] In the embodiment of the present application, the first state machine is further configured to: when the gradient data in the storage address corresponding to any read request is empty, send a third identifier to the second state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; The second state machine is further configured to: send the third identifier and the corresponding reflow data to the multiplier-adder in sequence; The multiplier-accumulator is further configured to: in response to receiving the third identifier, skip all data after the third identifier until other identifiers are received.

[0050] In a distributed training system, the storage location corresponding to the storage address in the weight parameter update instruction may be empty. Figure 1Taking the computing nodes 1-6 shown as an example, the target computing nodes in the weight parameter update instruction include computing nodes 1-6 and storage addresses addr1-addr6. Assume that the data in the storage location corresponding to addr4 is empty, that is, there is no corresponding gradient data for the read request of read4. At this time, the reflux data corresponding to read4 is not the gradient data corresponding to computing node 4. It may be a preset identifier representing that the data is empty, such as binary code 00. At this time, when the storage location corresponding to the storage address is empty, the first state machine sends a third identifier to the second state machine, such as binary code 01, to indicate that the reflux data corresponding to the read request is not gradient data. At this time, the reflux data received by the second state machine is: 10, data1, 10, data2, 10, data3, 01, 00, 10, data5, 10, data6, 11, and sends it to the multiplier and adder in sequence. When the multiplier receives the first binary code 10, there is no accumulated data at present. Starting from the first binary code 10, each data of data1 is accumulated. When the second binary code 10 is received, the current accumulation result, that is, the dimensionality reduction result of data1, is output to the third state machine, and each data of data2 is accumulated. When the third binary code 10 is received, the current accumulation result, that is, the dimensionality reduction result of data2, is output to the third state machine, and each data of data3 is accumulated. When the binary code 11 is received, the current accumulation result, that is, the dimensionality reduction result corresponding to data3, is output to the third state machine, and the subsequent 00 data is skipped, that is, no dimensionality reduction is performed. dimensional operation is performed until the fifth binary code 10 is received. When the fifth binary code 10 is received, since no accumulation operation is currently being performed, a preset identifier can be output to the third state machine at this time. For example, the binary code 00 is output to the third state machine, which means that there is no dimensionality reduction result at present, and the subsequent data of data5 are accumulated. When the sixth binary code 10 is received, the current accumulation result, that is, the dimensionality reduction result of data5, is output to the third state machine, and the subsequent data of data6 are accumulated. When 11 is received, it means that all the gradient data of the current batch have been processed, and the current accumulation result is the dimensionality reduction result of data6. After outputting it to the third state machine, all data can be cleared.

[0051] For the third state machine, the dimensionality reduction results of data1-data3 and data5-data6 can be packaged and stored based on their corresponding target computing node information. For the identifier 00 corresponding to data4, it means that the dimensionality reduction result of the corresponding gradient data of computing node 4 has not been obtained, so there is no need to package and store it at this time.

[0052] In addition, in an embodiment of the present application, multiple storage buckets can be pre-set in the local memory of the computing node deployed by the data processing device, and each storage bucket corresponds to a different computing node. When the third state machine packages the dimensionality reduction results corresponding to each computing node and stores them, it can store them in the corresponding storage bucket, thereby facilitating each computing node to call from the corresponding storage bucket.

[0053] In the embodiment of the present application, by setting the first to third state machines and the multiplier-adder, the first state machine is used to send a read request to the memory module according to the weight parameter update instruction, the second state machine is used to receive the refluxed gradient data, the multiplier-adder reduces the dimension of the gradient data to obtain the dimensionality reduction result, and the third state machine stores the dimensionality reduction result after wrapping it in the storage position corresponding to each computing node. Therefore, only the computing node that deploys the first state, the second state machine, the third state machine and the multiplier-adder needs to communicate with the external device, and other computing nodes do not need to communicate with the external device, and can directly call the dimensionality reduction result from their corresponding storage positions. During the training process, the first state machine can continuously send read requests based on the storage address, the second state machine can continuously receive the refluxed data, the multiplier-adder can continuously reduce the dimension of the refluxed gradient data, and the third state machine can continuously store the wrapped dimensionality reduction result in the storage position corresponding to each computing node, thereby improving the training efficiency.

[0054] Exemplary Methods like Figure 4 As shown, this exemplary embodiment proposes a data processing method, which is applied to the data processing device described in any of the above exemplary devices. In the embodiment of the present application, the data processing method includes the following steps S100-S400: Step S100: In response to receiving a weight parameter update instruction, determining, based on the first state machine, the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module; and, based on the storage address, the first state machine sends a read request to the memory module, and sends the target computing node information to the second state machine based on the first state machine; Step S200: Based on the second state machine, receiving the target computing node information and the gradient data returned by the memory module; and, based on the second state machine, sending the gradient data to the multiplier and adder, and sending the target computing node information to the third state machine; Step S300: Based on the multiplier-adder, reduce the dimension of the gradient data to obtain a dimension reduction result, and send the dimension reduction result to the third state machine; Step S400: The dimension reduction result is packaged based on the third state machine and the target computing node information, and written into a storage location corresponding to the target computing node for the target computing node to call.

[0055] In an embodiment of the present application, when the weight parameter update instruction includes multiple target computing node information and corresponding multiple gradient data storage addresses, the data processing method further includes: Based on the first state machine, respectively sending read requests to the memory module based on each storage address, and sending a first identifier to the second state machine before the gradient data of each read request flows back, wherein the first identifier is used to identify the starting position of the backflow data corresponding to each read request; Based on the first state machine, target computing node information corresponding to each read request is sent to the second state machine in sequence according to the request order of each read request.

[0056] In the embodiment of the present application, the data processing method further includes: Based on the second state machine, each first identifier and the corresponding return data are sent to the multiplier-adder in sequence; and based on the second state machine, each target computing node information is sent to the third state machine; Based on the multiplier, the data after the first identifier is accumulated, and when the next first identifier is received, the current accumulation result is output to the third state machine as the dimensionality reduction result.

[0057] In the embodiment of the present application, the data processing method further includes: Based on the third state machine, the dimensionality reduction results corresponding to each target computing node are packaged based on the information of each target computing node, and the packaged dimensionality reduction results are written into their corresponding storage locations respectively.

[0058] In the embodiment of the present application, the data processing method further includes: When all the gradient data corresponding to the storage addresses included in the weight parameter update instruction are all reflowed, sending a second identifier to the second state machine based on the first state machine, where the second identifier is used to identify that all the gradient data corresponding to the weight parameter update instruction are all reflowed; After all the first identifiers and the corresponding reflow data are sent to the multiplier-adder, sending the second identifier to the multiplier-adder based on the second state machine; When the multiplier-adder receives the second identifier, the dimension reduction operation of the multiplier-adder is terminated, and the current accumulation result is sent to the third state machine.

[0059] In the embodiment of the present application, the data processing method further includes: When the gradient data in the storage address corresponding to any read request is empty, sending a third identifier to the second state machine based on the first state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; Sending the third identifier and the corresponding return data to the multiplier-adder in sequence based on the second state machine; When the multiplier-adder receives the third identifier, the multiplier-adder skips all data after the third identifier until other identifiers are received.

[0060] The specific implementation method of each step of the data processing method in the above embodiments refers to the various data processing device embodiments in the exemplary device, and will not be repeated here one by one.

[0061] In addition, the data processing method proposed in the embodiment of the present application is applied to the data processing device described in any of the above embodiments, and therefore, it has at least all the beneficial effects of the above data processing device, which are not described one by one here.

[0062] Exemplary Media After introducing the method, medium and system of the exemplary embodiments of the present application, next, reference is made to Figure 5 For a description of the computer-readable storage medium of the exemplary embodiment of the present application, please refer to Figure 5, the computer-readable storage medium shown is a CD 70, on which a computer program (i.e., a program product) is stored. When the computer program is executed by the processor, the steps described in the above method implementation are implemented, for example, in response to receiving a weight parameter update instruction, the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module are determined based on the first state machine; and, based on the storage address, the first state machine sends a read request to the memory module, and based on the first state machine, the target computing node information is sent to the second state machine; based on the second state machine, the target computing node information and the gradient data returned by the memory module are received; and, based on the second state machine, the gradient data is sent to the multiplier and adder, and the target computing node information is sent to the third state machine; based on the multiplier and adder, the gradient data is reduced in dimension to obtain a reduced dimension result, and the reduced dimension result is sent to the third state machine; based on the third state machine and the target computing node information, the reduced dimension result is packaged and written into the storage location corresponding to the target computing node for the target computing node to call. The specific implementation of each step will not be repeated here. It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which will not be repeated here.

[0063] Exemplary Computing Devices After introducing the methods, systems, and media of the exemplary embodiments of the present application, reference is now made to Figure 6 A computing device according to an exemplary embodiment of the present application.

[0064] Figure 6 A block diagram of an exemplary computing device 80 suitable for implementing embodiments of the present application is shown. The computing device 80 may be a computer system or a server. Figure 6 The computing device 80 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0065] like Figure 6 As shown, the components of the computing device 80 may include, but are not limited to: one or more processors or processing units 801 , a system memory 802 , and a bus 803 connecting various system components (including the system memory 802 and the processing unit 801 ).

[0066] The computing device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 80, including volatile and non-volatile media, removable and non-removable media.

[0067] The system storage 802 may include computer system readable media in the form of volatile storage, such as random access memory (RAM) 8021 and / or cache memory 8022. The computing device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 8023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 is not shown in the Figure 6 As shown in FIG. 8 , a disk drive for reading and writing a removable non-volatile disk (e.g., a “floppy disk”) and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to the bus 803 via one or more data medium interfaces. The system storage 802 may include at least one program product, which has a set (e.g., at least one) of program modules, which are configured to perform the functions of each embodiment of the present application.

[0068] A program / utility 8025 having a set (at least one) of program modules 8024 may be stored, for example, in system memory 802, and such program modules 8024 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 8024 generally perform the functions and / or methods of the embodiments described herein.

[0069] The computing device 80 may also communicate with one or more external devices 804 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface. In addition, the computing device 80 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. Figure 6 As shown, the network adapter 806 communicates with other modules (such as the processing unit 801, etc.) of the computing device 80 via the bus 803. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with computing device 80 .

[0070] The processing unit 801 executes various functional applications and data processing by running the program stored in the system memory 802. For example, in response to receiving a weight parameter update instruction, based on the first state machine, determine the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module; and, based on the storage address, the first state machine sends a read request to the memory module, and based on the first state machine, sends the target computing node information to the second state machine; based on the second state machine, receives the target computing node information and the gradient data returned by the memory module; and, based on the second state machine, sends the gradient data to the multiplier and the target computing node information to the third state machine; based on the multiplier and the gradient data, reduces the dimension of the gradient data to obtain a reduced dimension result, and sends the reduced dimension result to the third state machine; based on the third state machine and the target computing node information, the reduced dimension result is packaged and written into the storage location corresponding to the target computing node for the target computing node to call. The specific implementation of each step is not repeated here. It should be noted that, although several units / modules or sub-units / sub-modules of the computing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0071] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0072] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0073] In the several embodiments provided in the present application, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0076] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computing device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0077] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0078] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

Claims

1. A data processing device, applied to a distributed training system, wherein the distributed training system comprises a plurality of computing nodes and a memory module, wherein each computing node is used to deploy a portion of operators of a model to be trained, and the memory module is used to store gradient data when weight parameters of the operators deployed by each computing node are updated; The data processing device comprises: a first state machine, a second state machine, a third state machine and a multiplier-adder, wherein the first state machine, the second state machine, the third state machine and the multiplier-adder are used to be deployed on any of the computing nodes, and the first state machine, the second state machine, the third state machine and the multiplier-adder are deployed on the same computing node; The first state machine is configured to: in response to receiving a weight parameter update instruction, determine, based on the weight parameter update instruction, the target computing node information for updating the weight parameter in the weight parameter update instruction and the storage address of the gradient data corresponding to the target computing node in the memory module; and, based on the storage address, send a read request to the memory module, and send the target computing node information to the second state machine; The second state machine is configured to: receive the target computing node information and the gradient data returned by the memory module; and send the gradient data to the multiplier and adder, and send the target computing node information to the third state machine; The multiplier-adder is configured to: perform dimension reduction on the gradient data to obtain a dimension reduction result, and send the dimension reduction result to the third state machine; The third state machine is configured to: package the dimension reduction result based on the target computing node information, and write it into a storage location corresponding to the target computing node for the target computing node to call.

2. The data processing device according to claim 1, when the weight parameter update instruction contains multiple target computing node information and corresponding multiple gradient data storage addresses, the first state machine is further configured to: send a read request to the memory module based on each storage address, and send a first identifier to the second state machine before the gradient data of each read request flows back, wherein the first identifier is used to identify the starting position of the backflow data corresponding to each read request; and According to the request order of each read request, the target computing node information corresponding to each read request is sent to the second state machine in sequence.

3. The data processing device according to claim 2, wherein the second state machine is further configured to: sequentially send each first identifier and the corresponding return data to the multiplier; and send each target computing node information to the third state machine; The multiplier is further configured to: in response to receiving a first identifier, accumulate data after the first identifier, and when receiving a next first identifier, output a current accumulation result as the dimensionality reduction result to the third state machine.

4. As described in the data processing device of claim 3, the third state machine is also configured to: package the corresponding dimensionality reduction results of each target computing node based on its information, and write each packaged dimensionality reduction result into its corresponding storage location.

5. The data processing device according to claim 2, wherein the first state machine is further configured to: when all the gradient data corresponding to each storage address included in the weight parameter update instruction are all reflowed, send a second identifier to the second state machine, wherein the second identifier is used to identify that all the gradient data corresponding to the weight parameter update instruction are all reflowed; The second state machine is further configured to: after all first identifiers and corresponding reflow data are sent to the multiplier-adder, send the second identifier to the multiplier-adder; The multiplier-accumulator is further configured to: in response to receiving the second identifier, end the dimension reduction operation and send the current accumulation result to the third state machine.

6. The data processing device according to claim 2, wherein the first state machine is further configured to: when the gradient data in the storage address corresponding to any read request is empty, send a third identifier to the second state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; The second state machine is further configured to: send the third identifier and the corresponding reflow data to the multiplier-adder in sequence; The multiplier-accumulator is further configured to: in response to receiving the third identifier, skip all data after the third identifier until other identifiers are received.

7. A data processing method, applied to the data processing device according to any one of claims 1 to 6, the data processing method comprising: In response to receiving a weight parameter update instruction, determining, based on the first state machine, target computing node information for updating the weight parameter in the weight parameter update instruction and a storage address in the memory module of gradient data corresponding to the target computing node; and, based on the storage address, the first state machine sends a read request to the memory module, and based on the first state machine sends the target computing node information to the second state machine; Based on the second state machine, receiving the target computing node information and the gradient data returned by the memory module; and, based on the second state machine, sending the gradient data to the multiplier-adder, and sending the target computing node information to the third state machine; Based on the multiplier-adder, reducing the dimension of the gradient data to obtain a dimension reduction result, and sending the dimension reduction result to the third state machine; The dimension reduction result is packaged based on the third state machine and the target computing node information, and written into a storage location corresponding to the target computing node for the target computing node to call.

8. The data processing method according to claim 7, when the weight parameter update instruction contains multiple target computing node information and corresponding multiple gradient data storage addresses, the data processing method further comprises: Based on the first state machine, respectively sending read requests to the memory module based on each storage address, and sending a first identifier to the second state machine before the gradient data of each read request flows back, wherein the first identifier is used to identify the starting position of the backflow data corresponding to each read request; Based on the first state machine, target computing node information corresponding to each read request is sent to the second state machine in sequence according to the request order of each read request.

9. The data processing method according to claim 8, further comprising: Based on the second state machine, sending each first identifier and the corresponding return data to the multiplier-adder in sequence; and sending the target computing node information to the third state machine based on the second state machine; Based on the multiplier, the data after the first identifier is accumulated, and when the next first identifier is received, the current accumulation result is output to the third state machine as the dimensionality reduction result.

10. The data processing method according to claim 9, further comprising: Based on the third state machine, the dimensionality reduction results corresponding to each target computing node are packaged based on the information of each target computing node, and the packaged dimensionality reduction results are written into their corresponding storage locations respectively.

11. The data processing method according to claim 8, further comprising: When all the gradient data corresponding to the storage addresses included in the weight parameter update instruction are all reflowed, sending a second identifier to the second state machine based on the first state machine, where the second identifier is used to identify that all the gradient data corresponding to the weight parameter update instruction are all reflowed; After all the first identifiers and the corresponding reflow data are sent to the multiplier-adder, sending the second identifier to the multiplier-adder based on the second state machine; When the multiplier-adder receives the second identifier, the dimension reduction operation of the multiplier-adder is terminated, and the current accumulation result is sent to the third state machine.

12. The data processing method according to claim 8, further comprising: When the gradient data in the storage address corresponding to any read request is empty, sending a third identifier to the second state machine based on the first state machine, wherein the third identifier is used to identify that the reflow data of the current read request is non-gradient data; Sending the third identifier and the corresponding return data to the multiplier-adder in sequence based on the second state machine; When the multiplier-adder receives the third identifier, the multiplier-adder skips all data after the third identifier until other identifiers are received.

13. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 7 to 12.

14. A computing device, comprising a processor, wherein the processor is configured to implement the method according to any one of claims 7 to 12 when executing a computer program stored in a memory.

Images