Distributed data processing system and method thereof

By using the model parameter components and the broadcast and gradient aggregation mechanism of the update components in a distributed data processing system, the problem of waste of memory of computing devices in parallel data processing is solved, and more efficient memory and resource utilization is achieved.

CN113342525BActive Publication Date: 2024-11-26BEIJING SILICONFLOW TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110634765.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-24
Publication Date
2024-11-26
Estimated Expiration
2040-07-24

AI Technical Summary

Technical Problem

The existing parallel data processing has wasted memory resources in distributed deep learning training, especially in the distributed deployment and synchronization of model parameters and Optimizer parameters, resulting in excessive transmission overhead and memory usage of parameter synchronization between devices. .

Method used

By including model parameter components and model parameter update components on a distributed data processing system, each computing device is included in the model parameter component, and the model parameter is processed in parallel by using the broadcast component, and the global gradient value is obtained through the gradient aggregation component for updating, which reduces the number of other The need to deploy model parameters and update components on parallel computing devices, thereby reducing the use of memory space and computing resources.

Benefits of technology

It effectively reduces the memory space requirements of computing devices in model parameter deployment and update, reduces the waste of repeated computing and memory resources, and maintains the communication efficiency with the global gradient aggregation method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113342525B_ABST
    Figure CN113342525B_ABST
Patent Text Reader

Abstract

The present invention discloses a distributed data processing system for processing data in parallel on a plurality of computing devices, wherein each computing device comprises a forward data processing component and a backward data processing component, wherein at least one computing device comprises a model parameter component not possessed by other computing devices and a model parameter updating component corresponding to one of the model parameter components, wherein the model parameter component inputs a set of model parameters used by the data to be processed in parallel contained in it to the broadcast components of other computing devices through the broadcast components corresponding to it, and the model parameter updating component obtains the corresponding global gradient value from the gradient convergence component corresponding to it for update processing.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an invention patent application with a filing date of July 24, 2020, national application number 202010724559.4, and titled “Distributed data processing system, deployment system for distributed computing tasks and method thereof”. Technical Field

[0002] The present disclosure relates to a data processing technology, and more specifically, to a distributed data processing system, a distributed computing task deployment system and a method thereof. Background Art

[0003] With the development of machine learning and the gradual deepening of research on artificial neural networks, the concept of deep learning has received widespread attention and application. Deep learning is a special type of machine learning that uses a mesh hierarchical structure to express the object being learned, combines simple concepts into abstract concepts, and realizes abstract concept expression through simple concept calculations. At present, deep learning has made great progress in the fields of image recognition, speech recognition, and natural language processing. Deep learning involves many model parameters, resulting in a huge amount of calculation, and the scale of training data is large, so it requires more computing resources.

[0004] At present, both general-purpose processors GPU and special-purpose chips TPU are many times more powerful than CPUs, but the demand for computing power in real applications is endless. Practitioners need to process larger data at a faster speed and with larger models, which cannot be met by a single hardware device. The development of hardware is limited by the limitations of manufacturing processes (chip area, power consumption, and clock signal propagation range), and it is impossible to increase the processing power of a chip indefinitely. Therefore, people often connect multiple high-throughput devices together through high-speed interconnection technology to collaboratively complete large-scale tasks. A common GPU cluster architecture is that the GPUs in the same node (server) communicate through NVLink or PCIe, and multiple nodes are interconnected through high-speed Ethernet or Infiniband. In the hardware deployment of Google's internal TPU Cloud, each server manages several TPUs, and multiple servers are connected into a large-scale cluster through high-speed interconnection technology.

[0005] To this end, technicians in this field have proposed data parallelism. In this way, the data is divided into multiple parts, and each device processes one part. In this way, each device only needs to process a small part of the overall data. The system running time is reduced from the time required for one device to process all the data to the time required for one device to process a small part of the data, thereby accelerating. This is the most common parallel method in big data scenarios. Data parallelism is particularly suitable for scenarios such as convolutional neural networks. The model of convolutional neural networks is composed of many small convolution kernels, and the model size is small, so the communication volume is small when synchronizing model gradients between devices. Currently, all frameworks can well support this parallel mode.

[0006] However, when performing distributed deep learning training, existing data parallel processing divides a batch of data and places it on various computing devices for calculation. That is, when each computing device uses different input data to perform the same calculation process, it is also necessary to place the model parameters on each computing device in a mirrored manner, aggregate the gradients calculated by each device, and update the parameters. Currently, the most mainstream gradient aggregation method is AllReduce, which reduces the gradients generated on each computing device together and broadcasts them to each computing device. Each computing device obtains the same aggregated gradient and then updates the parameters. Note that in this case, the parameter update calculations performed on each computing device will be exactly the same, so the parameters on each computing device will continue to be kept consistent after the update. In addition to maintaining model parameters, it is also necessary to maintain the parameters of the update component (Optimizer) for model parameter update. For example, Adam Optimizer needs to maintain two parameters m and v with the same data volume and model parameters, plus the gradient. When Adam updates parameters, a total of 4 times the size of the model parameters will be involved (model parameters, gradients, first-order moments of gradients (m), first-order moments of gradients (v)). In this case, if N computing devices are used for data parallel training, the calculation of the parameter update part will be repeated N times, and N×4×model size device memory will be required to complete the parameter update. When training deep learning models, the forward and backward calculations on a device can be carried out independently, but after obtaining the updated gradients of the model on each model, they need to be synchronized between devices and aggregated into a complete batch of data gradients before the model parameters are updated.

[0007] The commonly used computing devices for deep learning usually include CPU, GPU, FPGA and ASIC (application-specific integrated circuit), etc. GPU, FPGA and ASIC can be collectively referred to as data acceleration processing devices. Data acceleration processing devices often have faster computing speeds and higher memory bandwidths, but their memory capacity often has problems such as limited memory, high unit price, and difficulty in expansion. If the memory capacity of data acceleration processing devices is blindly increased to meet the training of large-scale models, it will bring high costs to enterprises. Obviously, the repeated calculations and memory in the existing data parallelism come from the distributed training parameters and optimizers are also distributed (Distributed) placed on various computing devices. This will undoubtedly increase the memory resource usage of computing devices on the one hand, and on the other hand, it will inevitably lead to the transmission overhead of parameter synchronization between devices. For current computing devices, especially computing devices such as GPUs with expensive memory resources, this is a huge waste.

[0008] Therefore, people expect to obtain a deployment system for a distributed data processing system or a distributed computing task, which can reduce the waste of computing device memory and computing resources during data parallel processing. Summary of the invention

[0009] To this end, the purpose of the present invention is to solve at least one of the above problems. According to one aspect of the present disclosure, a distributed data processing system is provided for processing data in parallel on the plurality of computing devices, each computing device comprising a forward data processing component and a backward data processing component, wherein at least one computing device contains a model parameter component that other computing devices do not have and a model parameter update component corresponding to the model parameter component, the model parameter component inputs a set of model parameters used by the data to be processed in parallel to the broadcast component of other computing devices through the broadcast component corresponding to it, and the model parameter update component obtains the corresponding global gradient value from the gradient convergence component corresponding to it for update processing.

[0010] According to the distributed data processing system of the present disclosure, the broadcast component of each computing device inputs the model parameters to the corresponding forward data processing component to perform forward data processing and to the backward data processing component to perform backward data processing.

[0011] According to the distributed data processing system disclosed in the present invention, it also includes: one or more multiple computing task components connected in series between the model parameter component and its corresponding broadcast component, each of the multiple computing task components is a computing task component with a single input and a single output; and one or more multiple computing task components connected in series between the model parameter update component and its corresponding gradient convergence component, each of the multiple computing task components is a computing task component with a single input and a single output.

[0012] According to the distributed data processing system disclosed in the present invention, the broadcast component connected to the model parameter component inputs the model parameters in the model parameter component to the forward operation component and the backward operation component corresponding to the model parameter component on the computing device where the model parameter component is located, and also inputs them to the parallel broadcast components on other parallel computing devices, so that the parallel broadcast component inputs the received model parameters into the parallel forward operation component and the backward operation component.

[0013] According to another aspect of the present disclosure, a distributed data processing method is provided, including: sharding the data to be processed into multiple shards according to the number of computing devices in the distributed data processing architecture and sending them to each computing device, performing forward data processing and backward data processing, dividing the model parameter set into multiple subsets, and distributing a subset to a model parameter component on a computing device; a model parameter component sends the parameters maintained by itself to a corresponding broadcast component of each other computing device that processes the shard data in parallel through a broadcast component connected to it, so that the forward data processing component and the backward data processing component on each other computing device processes the shard data based on the obtained model parameters.

[0014] According to the distributed data processing method disclosed in the present invention, it also includes: a gradient aggregation component corresponding to the broadcast component connected to the model parameter component obtains the gradient value sent by the corresponding gradient aggregation component on other computing devices, thereby obtaining the global gradient value, and transmits it to a model parameter update component corresponding to the model parameter component, so that the model parameter update component performs model parameter update processing.

[0015] According to another aspect of the present disclosure, a deployment system for distributed computing tasks is provided, including: a job description component, which describes a job neural network model based on the job type and obtains computing resources to be processed by the job, wherein the computing resources include multiple computing devices that can perform parallel computing, and gives location labels of the computing devices to which the forward computing task nodes, broadcast nodes, backward computing task nodes, and gradient convergence nodes to which the shard data for parallel computing of the job data belongs; a model parameter node configuration component, which obtains model parameters for processing the job based on the description of the job neural network model, calculates the total amount of all model parameters, and load-balancedly divides all model parameters into several model parameter nodes, wherein one model parameter node is only configured on one computing device and is placed before its successor forward computing task node in the neural network via the broadcast node; and a model parameter update node configuration component, which configures a model parameter update node corresponding to each model parameter node, wherein one model parameter update node is only configured on one computing device and is connected to the gradient convergence node corresponding to the broadcast node corresponding to the model parameter node, and the gradient convergence node is arranged after the backward computing task node corresponding to the computing task node corresponding to the model parameter node.

[0016] The distributed computing task deployment system according to the present disclosure also includes: a single successor operation task node configuration component, which traverses each successor forward operation task node of the model parameter node, configures the same position label as the model parameter node for the single successor forward operation task node that only consumes the output of the model parameter node and has only a single output, and connects it in series between the model parameter node and its corresponding broadcast node, as well as between the model parameter update node and the gradient convergence node corresponding to the model parameter node and the corresponding broadcast node, and configures a backward operation task node corresponding to the single successor forward operation task node.

[0017] According to another aspect of the present disclosure, a method for deploying distributed computing tasks is also provided, including: describing the job neural network model based on the job type and obtaining the computing resources to be processed by the job, the computing resources including multiple computing devices that can perform parallel computing, and assigning location labels to the computing devices to which the forward computing task nodes, broadcast nodes, backward computing task nodes and gradient convergence nodes of the shard data for parallel computing of the job data belong; based on the neural network model of the job, obtaining the model parameters of the processing job, calculating the total amount of all model parameters, and load balancing all model parameters into multiple parts, configuring a corresponding number of model parameter nodes, and each model parameter node is only configured with a location label of one computing device; inserting a broadcast node between each model parameter node and its subsequent forward computing task node; assigning the same computing device location label to a model parameter update node corresponding to any model parameter node; and inserting a gradient convergence node corresponding to the inserted broadcast node between the model parameter update node and the corresponding backward computing task node, and configuring the same location label.

[0018] The method for deploying distributed computing tasks according to the present disclosure also includes: traversing each successor forward operation task node of the model parameter node, modifying the position label configured for a single successor forward operation task node that only consumes the output of the model parameter node and has only a single output to the same position label as the model parameter node, and connecting it in series between the model parameter node and its corresponding broadcast node; and modifying the position label of the backward operation task node corresponding to the single successor forward operation task node to the same position label as the model parameter node, and connecting it in series between the model parameter update node and the gradient convergence node.

[0019] According to another aspect of the present disclosure, a method for deploying distributed computing tasks is also provided, including: describing a job neural network model based on the job type and obtaining computing resources to be processed by the job, the computing resources including multiple computing devices that can perform parallel computing, and assigning a location label of the computing device to which the forward computing task node to which the shard data for parallel computing of the job data belongs; based on the forward part of the neural network model of the job, obtaining the model parameters of the processing job, calculating the total amount of all model parameters, and load balancing all model parameters into multiple parts, configuring a corresponding number of model parameter nodes, and configuring only one location label of the computing device for each model parameter node; inserting a broadcast between each model parameter node and its subsequent forward computing task node; based on the forward part of the neural network model that has been configured, configuring the backward part of the neural network model accordingly, including configuring a model parameter update node corresponding to any model parameter node so that it has the same location label, configuring a gradient convergence node corresponding to any broadcast node and having the same location label, and configuring a backward computing task node corresponding to any forward computing task node and having the same location label.

[0020] The method for deploying distributed computing tasks according to the present disclosure also includes: traversing each successor forward operation task node of the model parameter node, modifying the position label configured for a single successor forward operation task node that only consumes the output of the model parameter node and has only a single output to the same position label as the model parameter node, and connecting it in series between the model parameter node and its corresponding broadcast node; and modifying the position label of the backward operation task node corresponding to the single successor forward operation task node to the same position label as the model parameter node, and connecting it in series between the model parameter update node and the gradient convergence node.

[0021] By using the above-mentioned distributed computing task deployment system and method and distributed data processing system and method, the model parameters and update component parameters used for data parallel processing are deployed on one of the multiple computing devices that perform data parallel processing. The model parameters in the model parameter component are input to each other parallel computing device through the broadcast component for use by the subsequent forward data processing component and the backward data processing component, which reduces the need to deploy the same model parameters on other parallel computing devices, thereby reducing the memory space requirements of the computing device in terms of model parameter deployment. And correspondingly, the update component parameters are correspondingly deployed on the computing device where the model parameters are deployed, and the gradient generated by the subsequent backward data processing component on the corresponding other computing device is obtained through collective communication through the gradient convergence component to obtain the global gradient, so that the model parameters can be updated through the model parameter update component. This also reduces the need to deploy and update component parameters on other parallel computing devices at the same time, thereby reducing the memory space requirements of the computing device in terms of model parameter update. Therefore, compared with the data parallel method of global gradient aggregation (ALLREDUCE) adopted in the prior art, the computing resources and memory usage of the update component part will be effectively reduced. At the same time, because the global gradient aggregation cost is approximately equivalent to the gradient aggregation and broadcasting of the present disclosure, the communication efficiency of the present disclosure is consistent with the communication efficiency of the global gradient aggregation method of the prior art. As described above, in addition to the model parameters, the present disclosure also deploys the operation nodes (such as model type conversion or normalization processing) that only involve model parameters on the same computing device as the model parameter nodes and is located before the broadcast nodes corresponding to the model parameter nodes or components, thereby further reducing the parallel deployment of these operation nodes that only involve model parameters on other computing devices, and further saving the use of memory and computing resources on other computing devices.

[0022] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The figure shows a schematic diagram of the principle structure of a distributed data processing system according to the first embodiment of the present disclosure;

[0024] Figure 2 The figure shows a schematic diagram of the principle structure of a distributed data processing system according to the second embodiment of the present disclosure;

[0025] Figure 3 Shown is a flow chart of a distributed data processing method according to the present disclosure; and

[0026] Figure 4What is shown is a schematic diagram of the principle of a deployment system for distributed computing tasks according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with embodiments and drawings so that those skilled in the art can implement the invention with reference to the description.

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. In addition, when referring to "first", it does not mean that there is a "second", and sometimes the first or second is used only to simplify the expression. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining".

[0031] In order to enable those skilled in the art to better understand the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] Figure 1 The figure shows a schematic diagram of the principle structure of a distributed data processing system according to the first embodiment of the present disclosure. As shown in the figure, the distributed data processing system according to the present disclosure includes multiple computing devices, such as computing device 100, computing device 200, computing device 300 and computing device 400. For the sake of simplicity, Figure 1 Only four computing devices, such as GPUs, for processing data in parallel are shown in the figure. In actual use, the number of computing devices can be reduced or increased as needed, and at least two. Figure 1As shown, according to the distributed data processing system of the present disclosure, the data to be processed is sharded to various computing devices for processing, for example, shard data 1 is input into computing device 100, shard data 2 is input into computing device 200, shard data 3 is input into computing device 300, shard data 4 is input into computing device 400, etc. Taking computing device 100 as an example, computing device 100 includes forward data processing components 113, 213, 313, 413, etc. and backward data processing components 423, 323, 223 and 123, etc. This is only illustrative, and the neural network model according to actual data processing may include more. The model parameter component 110 for configuring a subset of model parameters is connected to the broadcast component 112, and the model parameter subset is sent to the subsequent forward data processing component 113 and the backward data processing component 123 on the computing device 100 through the broadcast component 112, and the model parameter subset is sent to the parallel broadcast components 112 in other computing devices 200, 300 and 400 at the same time, so that the parallel broadcast components 112 in other computing devices can provide the obtained model parameter subset to the parallel forward data processing components 113 and the backward data processing components 123 in other computing devices for processing other sliced ​​data.

[0033] Similarly, other subsets of the model parameter set for job processing are also configured to model parameter components 210, 310, and 410 of other computing devices, etc. Each computing device is configured with only one model parameter component, and the model parameter component only manages one model parameter subset. Optionally, when the number of divided model parameter subsets is greater than the number of computing devices, multiple model parameter components can be configured on one of the computing devices. For example, if there are 4 computing devices for parallel computing, and the model parameters are divided into 8 subsets, two model parameter components can be configured on each computing device to manage the two model parameter subsets. If the model parameters are divided into 6 subsets, two model parameter components are selectively configured on two of the computing devices to manage the two model parameter subsets, and one model parameter component is configured on each of the other two computing devices to manage the two model parameter subsets. The model parameter component 210 for configuring a subset of model parameters is connected to the broadcast component 212, and the model parameter subset is sent to the subsequent forward data processing component 213 and the backward data processing component 223 on the computing device 200 through the broadcast component 212, and the model parameter subset is sent to the parallel broadcast components 212 in other computing devices 100, 300 and 400 at the same time, so that the parallel broadcast components 212 in other computing devices provide the obtained model parameter subset to the parallel forward data processing components 213 and the backward data processing components 223 in other computing devices for processing other shard data. Similarly, the model parameter components 310 and 410 also perform similar processing.

[0034] Corresponding to the model parameter components 110, 210, 310 and 410 deployed on each computing device, model parameter update components 120, 220, 320 and 420 are deployed in the backward processing part. At the same time, corresponding to the broadcast components 112, 212, 312 and 412 deployed after each model parameter component 110, 210, 310 and 410, gradient aggregation components 122, 222, 322 and 422 are deployed in the backward processing part. Each model parameter update component aggregates the slice gradient data via the gradient aggregation component deployed on the same computing device. For example, the gradient aggregation component 122 will generate the gradient after the backward operation processing of different slice data sent by the parallel gradient aggregation components 122 on other computing devices 200, 300 and 400 through collective communication, so as to obtain the global gradient value of the gradient value of each parallel slice, and send it to the model parameter update component 120 for update processing. Similarly, the gradient convergence component 222 generates gradients after backward processing of different shard data sent by the parallel gradient convergence components 222 on other computing devices 100, 300 and 400 through collective communication, thereby obtaining the global gradient value of the gradient value of each parallel shard, and sending it to the model parameter update component 220 for model parameter update.

[0035] If the memory size required by the model or total model parameters is M, and the number of computing devices is N, according to the conventional data parallel distributed data processing system, the total model parameters need to be input as a whole to each parallel processing computing device, so each computing device needs to update all model parameters separately. Thus, taking AdamOptimizer as an example, for the AllReduce method, each computing device needs to allocate an M space for model parameters, an M memory space for gradients, an M space for the first-order moment of gradients (m), and an M memory space for the second-order moment of gradients (v) for the Optimizer, which is a memory size of 4×M (i.e., model parameters, gradients, m, v), so for N computing devices, the total is N×4×M. If the distributed data processing system described above according to the present disclosure is adopted, each computing device needs to allocate a memory of 2×M (model parameters after broadcasting, gradients before aggregation), that is, the memory space for model parameters of broadcast components 112, 212, 312 and 412 and the memory space for gradients required by gradient aggregation components 122, 222, 322 and 422. In addition, the total memory required by all parameter model update components 120, 220, 320 and 420 is 3×M (model parameters, m, v), where the memory of model parameters in all computing devices is M, the memory space of the first-order moment of gradient (m) is M, and the memory space of the second-order moment of gradient (v) is M, so that the total memory usage of N computing devices in model update is 2×N×M+3×M. Obviously, this saves a lot of memory space of computing devices compared with traditional distributed parallel data processing systems. Moreover, in an ideal case, when there are N computing devices, the computing overhead of the model parameter updating component of the distributed data processing system of the present invention is 1 / N of the computing overhead of the conventional updating component.

[0036] Figure 2 Shown is a schematic diagram of the principle structure of a distributed data processing system according to the second embodiment of the present disclosure. Figure 2 The embodiment shown is Figure 1The embodiment shown is different in that, on each computing device, there are computing task components 111, 211, 311 and 411 connected in series between the model parameter component and its corresponding broadcast component. Although only one computing task component 111 is shown in each place, it is actually any number. Correspondingly, in the backward part, the computing task component connected in series between the model parameter update component and its corresponding gradient convergence component, each of the multiple computing task components is a computing task component 121, 221, 321 and 421 with a single input and a single output. Each of these computing task components is a computing task component with a single input and a single output. This is a distributed data processing system for half-precision training. By adopting the technical distributed data processing system of the present disclosure in half-precision data training, the computing nodes (such as type conversion or normalization processing of the model) and the model parameter nodes that only participate in the model parameters are deployed on the same computing device and before the broadcast nodes corresponding to the model parameter nodes or components, so that the parallel deployment of these computing nodes that only participate in the model parameters on other computing devices can be further reduced, and the use of memory resources on other computing devices can be further saved. Figure 2 Zhongyu Figure 1 The same parts are replaced by dotted boxes and will not be described again.

[0037] Specifically, if the memory size required for the model or the total model parameters is M, and the number of computing devices is N, according to the conventional data parallel distributed data processing system for half-precision training, the total model parameters need to be input as a whole to each parallel processing computing device, so each computing device needs to update all model parameters separately. In this way, taking AdamOptimizer as an example, for the AllReduce method, each computing device needs to allocate 5×M (model parameters, first-order moment of gradient (m), first-order moment of gradient (v), half-precision model and half-precision gradient) size memory for the model parameter update component, a total of N×5×M. If the distributed data processing system described above according to the present disclosure is adopted, each computing device needs to allocate M (half-precision model after broadcasting, half-precision gradient before convergence (reduce)). In addition, a total of 4×M (model, m, v, single-precision gradient) is allocated to all computing devices. In this way, the total memory usage required to implement model parameter updates is N×M+4×M. Obviously, this saves a lot of memory space of computing devices compared to traditional distributed parallel data processing systems. Likewise, in an ideal case, when the number of computing devices is N, the computational overhead of the model parameter updating component of the distributed data processing system of the present disclosure is 1 / N of the computational overhead of the conventional updating component.

[0038] It should be pointed out that when deploying actual components, it is necessary to first traverse all data-parallel model parameters, and for each model parameter, traverse the model parameter's successor nodes. If the traversed model parameter has only one successor node or component and the successor node or component only consumes the model parameter and has only one output, then this successor node will be treated as a node that only consumes the model parameter, and the successors of the successor node will be traversed according to the same conditions until all qualified successor nodes are found, and the model parameters and all qualified successor nodes are taken as a group, and all model groups will form a group list.

[0039] Figure 3 FIG. 1 is a flow chart of a distributed data processing method according to the present disclosure. Figure 3 As shown, first at step S310, based on the number of computing devices for data parallel processing, the data to be processed is fragmented into multiple fragmented data and sent to each computing device to perform forward data processing and backward data processing. At the same time or subsequently, at step S320, the model parameter set required for data processing is divided into multiple subsets, and one subset is distributed only to the model parameter component on one computing device. Then, at step S330, each model parameter component sends the parameters maintained by itself to a corresponding broadcast component of each other computing device that processes the fragmented data in parallel through a broadcast component connected to it, so that the corresponding broadcast component sends the received model parameter subset to the subsequent forward data processing component and backward data processing component of the computing device where it is located that need the model parameter, so that the forward data processing component and the backward data processing component process the fragmented data based on the obtained model parameters. Finally, at step S340, the gradient convergence component obtains the gradient value sent by the corresponding gradient convergence component on other computing devices, thereby obtaining the global gradient value, and transmits it to a model parameter update component corresponding to the model parameter component, so that the model parameter update component performs model parameter update processing.

[0040] Figure 4 FIG. 1 is a schematic diagram showing a distributed computing task deployment system according to an embodiment of the present disclosure. Figure 4As shown, the deployment system 500 of distributed computing tasks includes: a job description component 510, a model parameter node configuration component 520, a model parameter node configuration component 520 and a model parameter update node configuration component 530. The job description component 510 describes the job neural network model based on the job type and obtains the computing resources to be processed by the job, and the computing resources include multiple computing devices that can perform parallel computing. For example, computing device 1, computing device 2, computing device 3... etc. The job description component 510 gives the location label of the computing device to which the forward computing task node, broadcast node, backward computing task node and gradient convergence node of the shard data for parallel computing of the job data belong. The model parameter node configuration component 520 obtains the model parameters of the processing job based on the description of the job neural network model, calculates the total amount of all model parameters, and divides all model parameters into several model parameter nodes in a load-balanced manner, wherein one model parameter node is only configured on one computing device and is placed before its successor forward computing task node in the neural network via the broadcast node. The model parameter update node configuration component 530 corresponds to each model parameter node and configures a model parameter update node, wherein one model parameter update node is configured only on one computing device and is connected to the gradient convergence node corresponding to the broadcast node corresponding to the model parameter node, and the gradient convergence node is arranged after the backward operation task node corresponding to the operation task node corresponding to the model parameter node.

[0041] Optionally, the distributed computing task deployment system 500 also includes a single successor operation task node configuration component 540, which traverses each successor forward operation task node of the model parameter node, configures a single successor forward operation task node that only consumes the output of the model parameter node and has only a single output with the same position label as the model parameter node, and connects it in series between the model parameter node and its corresponding broadcast node, as well as between the model parameter update node and the gradient convergence node corresponding to the model parameter node and the corresponding broadcast node, and configures any backward operation node corresponding to the single successor forward operation task node.

[0042] exist Figure 4 The distributed computing task deployment system 500 shown can implement a method for deploying distributed computing tasks on a distributed data processing system, and can be deployed before or after the forward and backward expansion of the neural network.

[0043] Specifically, before forward and backward expansion, all data-parallel model parameters are traversed. For each model parameter, the successor nodes of the model parameter are traversed. If the model parameter has only one successor node and the successor node only consumes the model parameter and has only one output, then this successor node will be treated as a node that only consumes the model parameter, and the successor of the successor node will be traversed according to the same conditions until all qualified successor nodes are found. The model parameters and all qualified successor nodes are taken as a group (i.e., a group formed by a model parameter node and a single input-output successor node, which may have multiple single-input-output successor nodes connected in series), and all model groups will form a list of groups.

[0044] For all the groups found, calculate the size of the model parameters in the group, and assign the group to each computing device according to the size of the model parameters, so that the size of the model parameters assigned to each computing device is as even as possible, that is, the model parameter subsets assigned to each computing device are load balanced. According to the allocation result of the model parameter subsets, modify the position label (placement) of the node in each group, and insert a broadcast node (e.g. Figure 1 The broadcast components 112, 212, 312, and 412 in FIG. 11 ). On the computing device where the model parameter node is deployed, the broadcast node takes the last node of the group where the model parameter node is located as input, and the parallel broadcast nodes on other devices take the broadcast node on the computing device where the model parameter node is deployed as input. Figure 2 As shown, for example, the broadcast node 112 on computing device 1 takes the operation node 111 as input, while the parallel broadcast nodes 112 on other computing devices take the broadcast node 112 on computing device 1 as input. Optionally, the topological order of the successor nodes of the last node of each group containing model parameter nodes in the entire computational graph is obtained, and all groups are sorted in ascending order according to the minimum value of the topological order of all successor nodes of the last node in the group, and the broadcast nodes corresponding to each group are sequentially controlled according to the sorting result to ensure that the broadcast process is executed in this order.

[0045] When the system performs backward expansion and builds a training graph, the backward nodes generated by the deployment system 500 based on the forward nodes already placed in a single device are also deployed on the same device. If it is a parallel forward node, the corresponding parallel backward nodes are deployed on multiple parallel computing devices. The corresponding update component related nodes also have the same position labels corresponding to the model parameter nodes, and are thus also deployed on the same computing device, and there will be gradient convergence nodes corresponding to the broadcast nodes in the backward direction, such as corresponding to Figure 1Or the gradient convergence components 122, 222, 322 and 42 in 2. Similarly, these gradient convergence nodes also have parallel gradient convergence nodes on other computing devices.

[0046] Optionally, the deployment of distributed computing tasks can also be carried out after all neural networks are expanded forward and backward. Specifically, after the forward and backward expansion, all forward operation nodes and backward operation nodes and loss function nodes have been obtained. Traverse all data parallel model parameters, and for each model parameter, traverse the successor nodes of the model parameter. If the model parameter has only one successor node and the successor node only consumes the model parameter and has only one output, then this successor node will be treated as a node that only consumes the model parameter, and the successor of the successor node will be traversed according to the same conditions until all qualified successor nodes are found. The model parameters and all qualified successor nodes are grouped as a group (i.e., a group formed by a model parameter node and a single input-output successor node, in which there may be multiple single input-output successor nodes connected in series), and all model groups will form a list of groups.

[0047] For all the groups found, calculate the size of the model parameters in the group, and assign the group to each computing device according to the size of the model parameters, so that the size of the model parameters assigned to each computing device is as even as possible, that is, the model parameter subsets assigned to each computing device are load balanced. According to the allocation result of the model parameter subsets, modify the position label (placement) of the node in each group, and insert a broadcast node (e.g. Figure 1 The broadcast components 112, 212, 312, and 412 in FIG. 11 ). On the computing device where the model parameter node is deployed, the broadcast node takes the last node of the group where the model parameter node is located as input, and the parallel broadcast nodes on other devices take the broadcast node on the computing device where the model parameter node is deployed as input. Figure 2 As shown, for example, the broadcast node 112 on computing device 1 takes the operation node 111 as input, while the parallel broadcast nodes 112 on other computing devices take the broadcast node 112 on computing device 1 as input. Optionally, the topological order of the successor nodes of the last node of each group containing model parameter nodes in the entire computational graph is obtained, and all groups are sorted in ascending order according to the minimum value of the topological order of all successor nodes of the last node in the group, and the broadcast nodes corresponding to each group are sequentially controlled according to the sorting result to ensure that the broadcast process is executed in this order.

[0048] Subsequently, based on the model parameter nodes and their corresponding broadcast nodes, the corresponding model parameter update nodes that have been expanded are searched. The model parameter update node configuration component 530 of the deployment system 500 will modify the position label of the expanded model parameter update node based on its corresponding model parameter node position label, so that the position labels of the two are consistent, so that they are only deployed on the same computing device. A gradient convergence node corresponding to the broadcast node is inserted between the first backward node (the backward node corresponding to the last forward node in the group) and the predecessor node of the node. If it is the computing device where the group is located, the output of the gradient convergence node is connected to the first backward node, and the parallel gradient convergence nodes on other devices have no output, or in other words, the outputs of the gradient convergence nodes on other computing devices are merged to the gradient convergence node corresponding to the model parameter update node through collective communication. Figure 2 As shown, for example, the gradient convergence node 122 on computing device 1 outputs to the operation node 121, while the parallel gradient convergence nodes 122 on other computing devices use the gradient convergence node 122 on computing device 1 as an output object.

[0049] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be pointed out that a person skilled in the art can understand that all or any steps or components of the method and apparatus of the present disclosure can be implemented in hardware, firmware, software or a combination thereof in any computing device (including a processor, storage medium, etc.) or a network of computing devices. This can be achieved by a person skilled in the art using their basic programming skills after reading the description of the present disclosure.

[0050] Therefore, the purpose of the present disclosure can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general-purpose device. Therefore, the purpose of the present disclosure can also be achieved by simply providing a program product containing a program code that implements the method or device. In other words, such a program product also constitutes the present disclosure, and a storage medium storing such a program product also constitutes the present disclosure. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future.

[0051] It should also be noted that, in the apparatus and method of the present disclosure, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. In addition, the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0052] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A distributed data processing system, for parallel processing of a plurality of shards of data into which data to be processed is sharded according to the number of computing devices of a distributed data processing architecture on a plurality of computing devices included in the distributed data processing system, wherein at least one computing device contains a model parameter component and a model parameter update component corresponding to one of the model parameter components that other computing devices do not have, one or more multiple computing task components are connected in series between the model parameter component and its corresponding broadcast component, each of the multiple computing task components is a computing task component with a single input and a single output, and one or more multiple computing task components are connected in series between the model parameter update component and its corresponding gradient convergence component, each of the multiple computing task components is a computing task component with a single input and a single output, wherein a model parameter set is divided into a plurality of subsets, a subset is distributed to only one computing device, the model parameter component is used to configure a subset in the model parameter set, and the broadcast component of each computing device inputs the model parameter into the corresponding forward data processing component to perform forward data processing and into the backward data processing component to perform backward data processing, comprising: The model parameter component on the at least one computing device inputs a set of model parameters used by the data to be processed in parallel to the broadcast component connected to the model parameter component on the computing device where the model parameter component is located, so as to send a model parameter subset to a subsequent forward data processing component and a backward data processing component on the at least one computing device through the broadcast component on the at least one computing device, and simultaneously send the model parameter subset to the parallel broadcast components in other computing devices, so that the parallel broadcast components in the other computing devices provide the obtained model parameter subset to the parallel forward data processing components and backward data processing components in other computing devices for processing other shard data, so that the forward data processing components and backward data processing components on each of the other computing devices process the shard data based on the obtained model parameters; The gradient aggregation component aggregates the gradients generated after backward operation processing of different shard data sent from parallel gradient aggregation components on other computing devices through collective communication to obtain the global gradient value of the gradient value of each parallel shard, and sends it to the model parameter update component corresponding to the model parameter component, so that the model parameter update component obtains the corresponding global gradient value from the corresponding gradient aggregation component for update processing.

2. A distributed data processing method, comprising: On multiple computing devices included in the distributed data processing system, data to be processed is sharded into multiple sharded data according to the number of computing devices of the distributed data processing architecture to perform parallel data processing, wherein at least one computing device contains a model parameter component and a model parameter update component corresponding to one of the model parameter components that other computing devices do not have, one or more multiple computing task components are connected in series between the model parameter component and its corresponding broadcast component, each of the multiple computing task components is a computing task component with a single input and a single output, and one or more multiple computing task components are connected in series between the model parameter update component and its corresponding gradient convergence component, each of the multiple computing task components is a computing task component with a single input and a single output, wherein the model parameter set is divided into multiple subsets, one subset is distributed to only one computing device, the model parameter component is used to configure a subset in the model parameter set, and the broadcast component of each computing device inputs the model parameter into the corresponding forward data processing component to perform forward data processing and into the backward data processing component to perform backward data processing, including: The model parameter component on the at least one computing device inputs a set of model parameters used by the data to be processed in parallel to the broadcast component connected to the model parameter component on the computing device where the model parameter component is located, so as to send a subset of model parameters to a subsequent forward data processing component and a backward data processing component on the at least one computing device through the broadcast component on the at least one computing device, and simultaneously send the subset of model parameters to the parallel broadcast components in other computing devices, so that the parallel broadcast components in the other computing devices provide the obtained subset of model parameters to the parallel forward data processing components and backward data processing components in other computing devices for processing other shard data, so that the forward data processing components and backward data processing components on each of the other computing devices process the shard data based on the obtained model parameters; and A gradient aggregation component corresponding to the broadcast component connected to the model parameter component aggregates the gradients generated by backward operation of different shard data sent from parallel gradient aggregation components on other computing devices through collective communication to obtain a global gradient value of the gradient values ​​of each parallel shard, and sends it to the model parameter update component corresponding to the model parameter component, so that the model parameter update component obtains the corresponding global gradient value from the gradient aggregation component corresponding to it for update processing.