Computing systems, model training methods, and storage media

Abstract

Embodiments of the present application provide a computing system, a model training method and a storage medium. In the embodiments of the present application, for a plurality of parallel processing units without P2P direct connection in a distributed training scene, the CPU copies gradient data generated by the plurality of parallel processing units in each iteration process to the memory; after the CPU performs a full reduction operation on the gradient data of the plurality of parallel processing units, the full reduction result is written back to the plurality of parallel processing units, thereby realizing synchronization of the gradient data between the plurality of parallel processing units and helping to improve the convergence speed of the local neural network model of each parallel processing unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a computing system, a model training method and a storage medium. BACKGROUND

[0002] In the distributed training, a plurality of GPU nodes are used to train the same neural network model to speed up the training. Each GPU node stores a copy of the parameters of the entire neural network model, and batch sample data is divided to each GPU node for parallel processing.

[0003] In the distributed model training process, the model gradients of all GPU nodes need to be synchronized to ensure the accuracy and convergence of the model training during the reverse gradient update of each iteration. However, for GPU nodes lacking inter-card high-speed direct connection interfaces (such as consumer-grade GPUs), how to realize the gradient data synchronization between GPU nodes lacking inter-card high-speed direct connection interfaces is a technical problem to be solved by those skilled in the art. Synchronizing the communication between multiple GPU nodes requires sharing the Peripheral Component Interconnect Express (PCIe) bandwidth of the host, which causes a serious communication bandwidth bottleneck. SUMMARY

[0004] The aspects of the present application provide a computing system, a model training method and a storage medium to realize the model gradient data synchronization between GPU nodes lacking inter-card high-speed direct connection interfaces.

[0005] The embodiment of the present application provides a computing system, comprising: a memory, a central processing unit (CPU) and a plurality of parallel processing units; the memory, the CPU and the plurality of parallel processing units are communicatively connected;

[0006] The CPU is provided with a plurality of processes corresponding to the plurality of parallel processing units; each parallel processing unit corresponds to an independent process;

[0007] The plurality of parallel processing units are used to perform distributed training on a neural network model, and generate gradient data of the neural network model in each iteration round; each iteration round corresponds to at least one reduction task;

[0008] The CPU is configured to, for any reduction task corresponding to any iteration round, utilize the plurality of processes to parallel copy gradient data blocks from gradient data generated by the plurality of parallel processing units in the any iteration round to the memory in sequence in batches; the plurality of processes copy gradient data blocks in the same batch with the same data volume; the same data block identifier is configured for the gradient data blocks copied in the same batch by the plurality of processes; a first semaphore corresponding to the data block identifier is utilized to count the gradient data blocks corresponding to the data block identifier in the memory; in the case that the increment value of the first semaphore reaches the number of the plurality of processes, the plurality of processes are utilized to parallel perform all-reduction operation on the gradient data blocks corresponding to the data block identifier to obtain all-reduction results of the any reduction task; the increment value of the first semaphore of the data block identifier is an increment compared to the value of the first semaphore when the all-reduction operation of the previous reduction task is performed.

[0009] The parallel processing unit is configured to update the model parameters of the neural network model according to the all-reduction results of the reduction task corresponding to the any iteration round.

[0010] The embodiment of the present application further provides a model training method, a plurality of parallel processing units perform distributed training on a neural network model and generate gradient data of the neural network model in each iteration round; each iteration round corresponds to at least one reduction task.

[0011] The method comprises:

[0012] For any reduction task corresponding to any iteration round, a plurality of processes are utilized to parallel copy gradient data blocks from gradient data generated by the plurality of parallel processing units in the any iteration round to the memory in sequence in batches; the plurality of processes copy gradient data blocks in the same batch with the same data volume.

[0013] The same data block identifier is configured for the gradient data blocks copied in the same batch by the plurality of processes;

[0014] A first semaphore corresponding to each data block identifier is utilized to count the gradient data blocks corresponding to the data block identifier in the memory;

[0015] In the case that the increment value of the first semaphore reaches the number of the plurality of processes, the plurality of processes are utilized to parallel perform all-reduction operation on the gradient data blocks corresponding to the data block identifier to obtain all-reduction results of the any reduction task; the increment value of the first semaphore of each data block identifier is an increment compared to the value of the first semaphore when the all-reduction operation of the previous reduction task is performed.

[0016] copy the full reduction result of each reduction task corresponding to any iteration round to the parallel processing unit, so that the parallel processing unit updates the model parameter of the neural network model according to the full reduction result of the reduction task corresponding to any iteration round.

[0017] The embodiment of the present application further provides a computer readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, causing the one or more processors to execute the steps in the model training method.

[0018] In the embodiment of the present application, for the multiple parallel processing units without P2P direct connection in the distributed training scene, the CPU copies the gradient data generated by the multiple parallel processing units in each iteration process to the memory; and after the CPU performs the full reduction operation on the gradient data of the multiple parallel processing units, the CPU writes the full reduction result back to the multiple parallel processing units, thereby realizing the gradient data synchronization between the multiple parallel processing units and helping to improve the convergence speed of the local neural network model of each parallel processing unit. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, but do not constitute improper limitations on the present application. In the drawings:

[0020] Figure 1 A process diagram of distributed training of a GPU with an inter-card high-speed direct connection interface provided by a traditional scheme;

[0021] Figure 2 A structure diagram of a computing system provided by the embodiment of the present application;

[0022] Figure 3 An execution process diagram of the full reduction operation provided by the embodiment of the present application;

[0023] Figure 4 A process diagram of distributed training provided by the embodiment of the present application;

[0024] Figure 5 A gradient data block diagram provided by the embodiment of the present application;

[0025] Figure 6 And Figure 7 A flow diagram of the model training method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0027] In the distributed training scene of the neural network model, as shown in Figure 1 , multiple GPU nodes train the same neural network model. The initial model of the neural network model stored by each GPU node is the same. In the distributed training of each GPU node, the neural network model is trained in parallel using the divided sample data. In the distributed model training process, each GPU node can use sample data to perform forward calculation of the neural network model; the GPU node calculates the error and the reverse gradient data layer by layer based on the respective batch sample data; and the reverse gradient data is used to update the parameters of the local neural network model.

[0028] In order to ensure the consistency of the neural network model of multiple GPU nodes, the model gradient data of all GPU nodes needs to be synchronized at each iteration of the reverse gradient update, to ensure the accuracy and convergence of the model training.

[0029] In some conventional schemes, as shown in Figure 1 , the GPU node has a high-speed direct connection interface, such as an NVLink interface, and multiple GPU nodes can communicate directly through the high-speed direct connection interface to realize point-to-point (P2P) connection between GPUs. In this way, the GPU nodes can copy the gradient data through the high-speed direct connection interface, realize the synchronization of the gradient data between the GPU nodes, and use the synchronized gradient data to update the parameters of the local neural network model.

[0030] However, the GPU with high-speed direct connection interface is expensive, and in order to save economic cost, more and more GPUs without inter-card high-speed direct connection interface are used in the distributed training of the neural network model. Since the GPU without inter-card high-speed direct connection interface cannot realize P2P connection communication between GPUs, the communication between multiple GPU nodes needs to share the PCIe bandwidth of the host, which causes a serious communication bandwidth bottleneck.

[0031] In the embodiments of the present application, in order to realize gradient data synchronization between multiple GPUs, a new distributed model training manner is provided, and the basic idea is as follows: for multiple parallel processing units without P2P direct connection in a distributed training scene, the gradient data generated by the multiple parallel processing units in each iteration process is copied to the memory by the CPU; after the CPU performs a full reduction operation on the gradient data of the multiple parallel processing units, the full reduction result is written back to the multiple parallel processing units, thereby realizing gradient data synchronization between the multiple parallel processing units and helping to improve the convergence speed of the local neural network model of each parallel processing unit. On the other hand, the embodiments of the present application introduce a lightweight semaphore mechanism, and when the increment of the semaphore reaches the number of the multiple parallel processing units, a full rule operation is performed on the gradient data blocks of the multiple parallel processing units, thereby realizing synchronization between the processes of the multiple parallel processing units, and compared with the inter-process socket (Socket) communication, the cross-process synchronization efficiency can be improved.

[0032] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] It should be noted that the same reference numerals represent the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in subsequent drawings and embodiments.

[0034] Figure 2 The structural schematic diagram of the computing system provided by the embodiments of the present application is shown in FIG. 1. Figure 2 As shown in the figure, the computing system S10 includes a central processing unit (CPU) 10, multiple parallel processing units 20, and a memory 30.

[0035] In the embodiments, the number of CPUs 10 is not limited. The CPU 10 can be at least one, i.e., one or more; each CPU 10 can be a single-core processor or a multi-core processor. The above-mentioned multiple refers to two or more than two. For example, the parallel processing units 20 can be 8, 10, or 16, etc. The multi-core refers to dual-core or more than dual-core. Figure 2 In the embodiments, the number of CPUs 10 is not limited. The CPU 10 can be at least one, i.e., one or more; each CPU 10 can be a single-core processor or a multi-core processor. The above-mentioned multiple refers to two or more than two. For example, the parallel processing units 20 can be 8, 10, or 16, etc. The multi-core refers to dual-core or more than dual-core.

[0036] The parallel processing unit 20 refers to a processing device capable of parallel computing. For example, the parallel processing unit 20 can be a GPU, a Field-Programmable Gate Array (FPGA), or an Application Specific Integrated Circuit (ASIC). In this embodiment, the parallel processing unit 20 has video memory for storing the model parameters of the neural network model to be trained and the gradient data generated in each iteration during the training of the neural network model.

[0037] The CPU 10, memory 30, and multiple parallel processing units 20 are interconnected. Regarding the multiple parallel processing units 20, in some embodiments, such as... Figure 2 As shown, multiple parallel processing units 20 may or may not communicate directly with each other. For parallel processing units 20 that can communicate directly, P2P communication can be achieved through a direct connection interface to synchronize gradient data. Therefore, this embodiment focuses on using the example of parallel processing units 20 without a direct connection communication interface to illustrate the distributed training method provided in this embodiment. Of course, parallel processing units 20 with a direct connection communication interface can also use the distributed training method provided in this embodiment to perform distributed training of the neural network model.

[0038] In this embodiment, the CPU 10, memory 30, and multiple parallel processing units 20 can be located on the same physical machine, and the CPU 10, parallel processing units 20, and memory 30 can communicate with each other through a communication interface. The communication interface can be a PCIe interface, a PCI interface, or a USB interface, etc. Preferably, the communication interface is a PCIe interface.

[0039] In other embodiments, the CPU 10 and memory 30 may be located on the same physical machine, but the multiple parallel processing units 20 may be located on different physical machines. When the CPU 10 and the multiple parallel processing units 20 are located on different physical machines, the CPU 10 and the multiple parallel processing units 20 can be connected via network communication. The multiple parallel processing units 20 can also be connected via network communication. For example, the CPU 10 and the multiple parallel processing units 20 may be located in a cloud server and connected via network communication, etc. Figure 2 The illustration only shows the CPU 10, memory 30 and multiple parallel processing units 20 set up on the same physical machine, but this does not constitute a limitation.

[0040] In this embodiment, as Figure 2As shown, multiple parallel processing units 20 can perform distributed training on the same neural network model. Specifically, the initial model parameters of the neural network model stored by the multiple parallel processing units 20 are the same. The multiple parallel processing units 20 can respectively read different sample data from the training sample data set, and respectively perform model training on the neural network model using the read sample data. The training process of the parallel processing units 20 on the neural network model is iterative training. At each iteration round (i.e., each iteration training), the parallel processing units 20 can perform forward calculation of the neural network model to obtain an output result, and then calculate an error using a loss function, and calculate the reverse gradient of the parameters of the neural network model in the error back propagation process. The reverse gradient calculated at each iteration round is the gradient data generated by the parallel processing unit at the iteration round. The number of gradient data is determined by the number of model parameters of the neural network model. The parallel processing units 20 can update the local neural network model according to the gradient data generated at each iteration round.

[0041] Since the sample data read by the parallel processing units 20 are different, the gradient data generated by the multiple parallel processing units 20 at the same iteration round are probably different. In order to ensure the consistency of the neural network model of the multiple parallel processing units 20, the gradient data of all parallel processing units 20 need to be synchronized at each iteration gradient data update to ensure the accuracy and convergence of model training.

[0042] In the embodiments of the present application, in order to realize the synchronization of the gradient data between the parallel processing units 20, the All Reduce (AR) operation is performed on the gradient data generated by the multiple parallel processing units 20 at the same iteration round by the collective communication algorithm of the CPU 10 to obtain the All Reduce result of the gradient data of the multiple parallel processing units 20. Then, the multiple parallel processing units 20 obtain the All Reduce result of the gradient data obtained by the CPU 10; and update the model parameters of the local neural network model according to the All Reduce result.

[0043] As shown in Figure 3 The above CPU-based collective communication method can be represented in a pipeline manner, mainly including three stages: (1) Stage 1: copying gradient data from the parallel processing unit 20 to the memory 30 (i.e., D2H in Figure 3 ); (2) Stage 2: performing All Reduce operation on the gradient data in the memory by the CPU (i.e., AR in Figure 3 ); (3) Stage 3: copying the All Reduce result from the memory to the parallel processing unit 20 (i.e., H2D in Figure 3 ). The three stages are executed in a pipeline manner, realizing the synchronization of the gradient data between the parallel processing units 20.

[0044] Specifically, as shown in Figure 2As shown, for the gradient data of the neural network model generated by the plurality of parallel processing units 20 at each iteration round X, the CPU 10 can copy the gradient data generated by the plurality of parallel processing units 20 at the iteration round X to the memory 30; then, the CPU 10 reads the gradient data corresponding to each of the plurality of parallel processing units 20 from the memory 30, and performs an all-reduce operation on the gradient data corresponding to each of the plurality of parallel processing units 20 to obtain an all-reduce result of the gradient data of the plurality of parallel processing units.

[0045] In the embodiments of the present application, the specific operation of the all-reduce operation on the gradient data of the plurality of parallel processing units 20 is not limited. In some embodiments, the sum of the gradient data of the plurality of parallel processing units 20 can be calculated. The sum of the gradient data of the plurality of parallel processing units 20 refers to the sum of the gradient data of the plurality of parallel processing units 20 at the same parameter. For example, the parallel processing units 20 are 3, and the gradient data are the gradient data corresponding to parameters x1-x100 respectively, then the sum of the gradient data of the 3 parallel processing units is the sum of the gradient of the parameters xi of the 3 parallel processing units 20, i = 1, 2, …, 100, and the sum of the gradient data obtained is still the gradient of 100 dimensions.

[0046] In other embodiments, the average of the gradient data of the plurality of parallel processing units 20 can be calculated. Accordingly, the average of the gradient data of the plurality of parallel processing units 20 refers to the average of the gradient data of the plurality of parallel processing units 20 at the same parameter.

[0047] The all-reduce operation shown in the above embodiments is only exemplary and does not constitute a limitation.

[0048] After obtaining the all-reduce result of the gradient data of the plurality of parallel processing units 20, the CPU 10 can copy the all-reduce result to the video memory of the plurality of parallel processing units 20. The parallel processing units 20 can update the model parameters of the local neural network model according to the all-reduce result. The above method is used for multiple iteration training until the neural network model converges, and a trained neural network model is obtained.

[0049] In practical applications, the model parameter scale of the neural network model is large, and the gradient data generated by one iteration is also large. The communication bandwidth of the computing system S10 is limited, and batch data replication is often required. In each iteration round, the CPU 10 needs to perform at least one reduction task. Each reduction task is used to perform a full reduction operation on at least part of the gradient data of the plurality of parallel processing units 20. The number of reduction tasks that the CPU 10 needs to perform in one iteration can be flexibly set according to actual conditions. For an embodiment in which the CPU 10 needs to perform 1 reduction task in one iteration, the CPU 10 performs a full reduction operation on the gradient data of all parameters of the plurality of parallel processing units 20 during the execution of the reduction task. For an embodiment in which the CPU 10 needs to perform multiple reduction tasks in one iteration, the CPU 10 performs a full reduction operation on the gradient data of part of the parameters of the plurality of parallel processing units 20 during the execution of a reduction task.

[0050] The gradient data of the plurality of parallel processing units 20 shares the communication bandwidth of the shared computing system S10 during the replication process. For an embodiment in which the CPU and the parallel processing unit are connected through a communication interface such as a PCIe interface, the plurality of parallel processing units 20 share the PCIe communication bandwidth. Therefore, for any reduction task A corresponding to any iteration round X, when the reduction task A starts to execute, in combination with Figure 3 and Figure 4 , the CPU 10 can start multiple processes to replicate gradient data blocks in sequence from the gradient data generated by the plurality of parallel processing units 20 in the iteration round X to the memory 30 in batches. The gradient data block refers to a data block composed of the gradient data replicated by each batch. The data amount of the gradient data block replicated by the plurality of parallel processing units 20 in the same batch is the same. Assuming that a certain parallel processing unit 20 replicates 2 KB of data in the Nth data replication, other parallel processing units also replicate 2 KB of data in the Nth data replication.

[0051] Each parallel processing unit corresponds to an independent process. The process is used to replicate the gradient data of the corresponding parallel processing unit. The plurality of processes can be numbered by Rank, such as Figure 3 Rank0, Rank1 in the above. Rank represents the process number, which is used for inter-process communication. Each process corresponds to a Rank number, and distributed training is completed by multiple processes.

[0052] The order refers to the storage order of the gradient data in the parallel processing unit 20. The storage order of the gradient data of different parallel processing units 20 in the respective video memory is the same. For example, assuming that the model parameters of the neural network model are x1-x100, the multiple parallel processing units 20 can all store the gradient data corresponding to x1-x100 in order. In this way, it can be ensured that the CPU 10 copies the gradient of the same dimension parameters in the same batch.

[0053] When the above batch copying of the gradient data blocks of the multiple parallel processing units 20 to the memory 30 is performed, the sizes of the gradient data blocks copied in different batches can be the same or different. In some embodiments, the size Z of the data block copied each time can be set in advance. Accordingly, as shown in FIG. 8, the CPU 10 can use the process corresponding to the parallel processing unit 20 to copy a gradient data block of size Z from the gradient data generated by the parallel processing unit 20 in any iteration round X to the memory 30 based on the set data block size Z each time. Figure 5

[0054] Since the computing resources of the CPU 10 and the communication bandwidth of the computing system S10 are limited. Therefore, the communication amount S of each reduction task for any iteration round X can be pre-configured. Wherein, the communication amount S of the reduction task refers to the data amount of each parallel processing unit 20 required for the full reduction operation of the reduction task.

[0055] Accordingly, the CPU 10 can use the process corresponding to the parallel processing unit 20 to copy a gradient data block of size Z from the gradient data generated by the parallel processing unit 20 in any iteration round X to the memory 30 based on the set data block size Z each time, until the data amount copied by the parallel processing unit 20 to the memory 30 reaches the communication amount S of the reduction task. Wherein, Figure 5 The data amount in one column in FIG. 8 is the communication amount S of the reduction task.

[0056] In some embodiments, the communication amount S is not an integer multiple of the data block size Z, and there is a gradient data block whose data amount is less than Z. In this embodiment, the CPU 10 can copy a gradient data block whose data amount is to be copied from the gradient data generated by the parallel processing unit 20 in any iteration round X to the memory 30 in the case that the data amount to be copied corresponding to the parallel processing unit 20 is less than the set data block size Z. Wherein, the data amount to be copied is equal to the difference between the data amount of the gradient data already copied to the memory by the parallel processing unit and the communication amount S of the reduction task A. For example, Figure 5 In FIG. 8, the data amount to be copied is the gradient data block identified as ID3, and the data amount of the gradient data block ID3 is less than Z. Therefore, the CPU 10 copies the gradient data block ID3 from the gradient data generated by the parallel processing unit 20 in any iteration round X to the memory 30.​

[0057] In the embodiment of the present application, the CPU 10 can allocate shared memory to each of the processes in the initialization stage. The shared memory refers to a space in the memory allocated by the operating system for synchronization between processes, and different processes can access the same block of shared memory. Based on the shared memory allocated to each process, when copying the gradient data from the parallel processing units 20 to the memory 30, the CPU 10 can use the process corresponding to the parallel processing unit 20 to copy the gradient data block of the set data block size Z from the gradient data generated by the parallel processing unit 20 in any iteration round X to the shared memory of the process corresponding to the parallel processing unit 20 in sequence.

[0058] Since the copying of the gradient data of the multiple parallel processing units 20 by the CPU 10 is independent of each other, and the all-reduce operation is performed on the gradient data of different parallel processing units 20, the gradient data of the same parameter needs to be reduced. Therefore, it is necessary to ensure the pipeline synchronization across processes. Based on this, in the embodiment of the present application, as shown in Figure 4 and Figure 5 , the CPU 10 can configure the same data block identifier (Chunk_Index) for the gradient data blocks copied in the same batch for multiple processes during the gradient data copying process.

[0059] Alternatively, the CPU 10 can configure the data block identifier for the currently copied gradient data block based on the copying order of the current batch in the above reduction task A. The gradient data blocks copied in the same batch by the multiple parallel processing units 20 have the same data block identifier.

[0060] For example, as shown in Figure 5 , it is assumed that there are four parallel processing units, and the identifiers of the gradient data blocks copied by the four parallel processing units in the Mth copying are all ID(i-1). i=1, 2, 3.

[0061] In the embodiment, in order to realize the synchronization between the data copying processes of the multiple independent parallel processing units 20, a semaphore (denoted as chunk_count) can be configured for each data block identifier. The semaphore mechanism is a method for realizing process synchronization. Processes can easily realize process synchronization by using a pair of primitives provided by the operating system to operate the semaphore. The semaphore is a variable. In the embodiment, the semaphore chunk_count is used to count the gradient data blocks corresponding to the corresponding data block identifier in the memory.

[0062] For example, the semaphore is incremented by 1 each time the gradient data block of the same data block identifier is copied to the memory. Based on this, as shown in Figure 4As shown, the CPU 10 can count the gradient data blocks corresponding to each data block identifier in the memory by using the signal quantity (chunk_count) corresponding to the data block identifier. For example, as shown in Figure 4 As shown, the signal quantity corresponding to the data block identifier IDi is incremented by 1 each time the gradient data block with the data block identifier IDi is copied into the memory 30. In Figure 4 , i = 1, 2, 3, 4,...

[0063] In the present embodiment, the signal quantity is incremented by the number of parallel processing units 20 each time the gradient data block with the same data block identifier is copied into the plurality of parallel processing units 20, indicating that the gradient data block with the data block identifier has been copied into all the parallel processing units 20. Then, the CPU 10 can perform the all-reduce operation on the gradient data blocks corresponding to the data block identifier. Figure 2 In the present embodiment, the signal quantity is incremented by the number of parallel processing units 20 each time the gradient data block with the same data block identifier is copied into the plurality of parallel processing units 20, indicating that the gradient data block with the data block identifier has been copied into all the parallel processing units 20. Then, the CPU 10 can perform the all-reduce operation on the gradient data blocks corresponding to the data block identifier.

[0064] Specifically, as shown in Figure 4 , when the increment value of the signal quantity chunk_count corresponding to the data block identifier reaches the number of parallel processing units, the CPU 10 can perform the all-reduce (AR) operation on the gradient data blocks corresponding to the data block identifier to obtain the all-reduce operation result of the data gradient block corresponding to the data block identifier. The gradient data blocks corresponding to each data block identifier in one reduce task can be all-reduced in the manner shown in the above embodiment to obtain the all-reduce result of the reduce task.

[0065] As described above, Figure 3 , the rectangles in the same column represent the gradient data blocks with the same data block identifier in different parallel processing units. When the increment value of the signal quantity chunk_count corresponding to the data block identifier reaches the number of parallel processing units, it indicates that the gradient data blocks with the same data block identifier have been copied into all the parallel processing units 20. Then, the all-reduce operation can be performed on the gradient data blocks corresponding to the data block identifier.

[0066] It should be noted that the increment value of the signal quantity chunk_count corresponding to each data block identifier is the increment of the value of the signal quantity chunk_count corresponding to the data block identifier when the all-reduce operation of the previous reduce task is performed. For example, when the all-reduce operation of the previous reduce task is performed, the value of the signal quantity chunk_count corresponding to a certain data block identifier is P1, and the increment of the signal quantity corresponding to the data block identifier is the increment compared to P1, i.e., the amount of increase based on P1.

[0067] Of course, in some embodiments, the semaphore chunk count of the data block identifier can be reset to a set value, such as 0, at each full-reduce operation of each reduction task, or at the full-reduce operation of the Nth reduction task of the N reduction tasks. Accordingly, the value of the semaphore chunk count of the data block identifier at the full-reduce operation of the previous reduction task is the reset set value. Accordingly, the increment value of the semaphore chunk count of the data block identifier is the increment compared to the reset set value.

[0068] The above embodiments exemplarily illustrate the full-reduce operation process of the reduction task A of any iteration round X. Each reduction task can perform full-reduce operation on the gradient data blocks of the plurality of parallel processing units 20 copied for the reduction task according to the full-reduce operation manner provided in the above embodiments, and further obtain the full-reduce result of each reduction task of any iteration round X.

[0069] The parallel processing units 20 can obtain the full-reduce result of each reduction task corresponding to the iteration round X, and update the model parameters of the local neural network model according to the full-reduce result of each reduction task corresponding to the iteration round X.

[0070] In the embodiments of the present application, for the plurality of parallel processing units without P2P direct connection in the distributed training scene, the CPU copies the gradient data generated by the plurality of parallel processing units in each iteration process to the memory; and after the CPU performs full-reduce operation on the gradient data of the plurality of parallel processing units, the CPU writes the full-reduce result back to the plurality of parallel processing units, thereby realizing synchronization of the gradient data between the plurality of parallel processing units, and helping to improve the convergence speed of the local neural network model of each parallel processing unit.

[0071] The multi-stage pipeline full-reduce process provided in the embodiments of the present application can perform full-reduce operation on the gradient data blocks corresponding to a data block identifier after completing copying of all gradient data blocks of the data block identifier, and write the full-reduce result corresponding to the data block identifier back to the plurality of parallel processing units after the full-reduce operation on the gradient data corresponding to the data block identifier, thereby guaranteeing pipeline execution of the full-reduce process, reducing the waiting time of each stage of the full-reduce process, and improving the full-reduce efficiency.

[0072] On the other hand, the embodiments of the present application introduce a lightweight semaphore mechanism. When the increment of the semaphore reaches the number of the plurality of parallel processing units, the full-reduce operation on the gradient data blocks of the plurality of parallel processing units is performed, thereby realizing synchronization between processes of the plurality of parallel processing units. Compared with socket communication between processes, the cross-process synchronization efficiency can be improved.

[0073] In the embodiments of the present application, the CPU 10 starts an independent process corresponding to each parallel processing unit. Accordingly, in order to improve the efficiency of the all-reduce operation, in some embodiments, for any data block identifier IDn, the CPU 10 can use multiple processes corresponding to multiple parallel processing units 20 to perform the all-reduce operation on the gradient data blocks corresponding to the data block identifier IDn in parallel.

[0074] Specifically, if the data amount Q of each gradient data block in the gradient data blocks corresponding to the data block identifier IDn is an integer multiple of the number M of processes (assuming K times, K is a positive integer), i.e., K = Q / M, then the all-reduce operation can be performed on the gradient data blocks corresponding to the data block identifier IDn in parallel using multiple processes. It should be noted that Q refers to the data amount of a single gradient data block, and the unit is byte (Byte). The data amount Q being an integer multiple of the number M of processes means that Q can be divided by M without a remainder. Accordingly, each process can process K bytes of gradient data.

[0075] If there is a first gradient data block in the gradient data blocks corresponding to the data block identifier IDn whose data amount Q is not an integer multiple of the number M of processes, then the CPU 10 can use a first process in the M processes to perform the all-reduce operation on the first gradient data block in the gradient data blocks corresponding to the data block identifier IDn. The first process can be any process in the multiple processes, or a process corresponding to a specified parallel processing unit, or a process numbered Rank0. Figure 3

[0076] For example, in the Figure 5 , the number of parallel processing units 20 is 4, and the number of processes is 4. For the gradient data blocks with data block identifiers ID0 and ID1, the data amount is an integer multiple of 4, so the all-reduce operation can be performed on the four gradient data blocks with the data block identifier ID0 in parallel using the four processes. For the gradient data blocks with the data block identifier ID2, the data amount is not an integer multiple of 4, so the all-reduce operation can be performed on the four gradient data blocks with the data block identifier ID2 using the process numbered Rank0.

[0077] In some embodiments, in addition to the first gradient data block, there can also be a second gradient data block in the gradient data blocks corresponding to the data block identifier IDn. The second gradient data block is other than the first gradient data block in the gradient data blocks corresponding to the data block identifier IDn, and the number of second gradient data blocks is equal to the quotient obtained by dividing the total data amount of the gradient data blocks corresponding to the data block identifier IDn by the number M of processes. For the second gradient data in the gradient data blocks corresponding to the data block identifier IDn, the all-reduce operation can be performed on the second gradient data in parallel using multiple processes.​

[0078] To further improve the efficiency of all-reduce operation, in each independent process, the all-reduce operation can be performed in parallel on the gradient data responsible for by each process using Single Instruction Multiple Data (SIMD) instructions. The SIMD instruction can process multiple data with one instruction. Each process is responsible for different gradient data in the gradient data block, and the gradient data copied by multiple processes constitutes the gradient data block.

[0079] Since the processes of the multiple parallel processing units 20 are independent of each other, and the all-reduce operation requires inter-process synchronization, in order to realize the inter-process synchronization of the all-reduce operation, another semaphore (denoted as ar_count) can be configured for each data block identifier. The semaphore ar_count is used to count the processes that have completed the all-reduce operation.

[0080] For example, each time a process that has completed the all-reduce operation is generated, the semaphore ar_count is incremented by 1. Based on this, the CPU 10 can use the semaphore ar_count corresponding to each data block identifier to count the processes that have completed the all-reduce operation; in the case where the increment value of the semaphore ar_count reaches the number of the multiple parallel processing units 20, the all-reduce result corresponding to the data block identifier is copied to the multiple parallel processing units 20.

[0081] Specifically, the CPU 10 can write the all-reduce result corresponding to the data block identifier to the shared memory corresponding to the second process in the case where the increment value of the semaphore ar_count of the data block identifier reaches the number of the multiple parallel processing units 20. The second process can be any process in the multiple processes, or can be the process numbered Rank0. Further, the CPU 10 can copy the all-reduce result corresponding to the data block identifier to the multiple parallel processing units 20 through the shared memory of the second process. The all-reduce result generated by each process in the next iteration round can overwrite the all-reduce result of the current iteration round stored in the shared memory of the process, realizing the reuse of the shared memory in the entire model training process.

[0082] In the above CPU-side collective communication process, the shared memory of the multiple processes is reusable in the entire model training process, and there is no need to create a shared memory separately for each gradient synchronization in each iteration.

[0083] It should be noted that the increment value of the semaphore ar_count identified by each data block is the increment of the value of the semaphore ar_count identified by the data block compared to the value of the semaphore ar_count identified by the data block when the all-reduce operation of the previous reduction task is completed. For example, the value of the semaphore ar_count identified by a data block is P2 when the all-reduce operation of the previous reduction task is completed, and the increment of the semaphore ar_count identified by the data block is the increment compared to P2, that is, the amount of increase based on P2.

[0084] Of course, in some embodiments, the semaphore ar_count identified by the data block can be reset to a set value, such as 0, when the all-reduce operation of each reduction task is completed. Accordingly, the value of the semaphore ar_count identified by the data block when the all-reduce operation of the previous reduction task is completed is the reset set value. Accordingly, the increment value of the semaphore ar_count identified by the data block is the increment compared to the reset set value.

[0085] Since the semaphore reset also requires a certain time overhead, in order to further reduce the time overhead and improve the gradient data synchronization in the distributed training process, the semaphores chunk_count and ar_count can be reset after accumulating multiple invocations of the all-reduce operation of the collective communication. Accordingly, the number threshold R of the all-reduce operation of the reduction task can be configured for the reduction task unit object, and the semaphores chunk_count and ar_count can be reset when the number of completed all-reduce operations of the reduction task unit object reaches the set number threshold R. Accordingly, the CPU 10 can use a counter to count the number of completed all-reduce operations of the reduction task unit object. The counter is incremented by 1 each time a reduction task level all-reduce operation is completed. Further, when the value of the counter reaches the set number threshold, the semaphores chunk_count and ar_count can be initialized to a set value. The set value can be any value that is not equal to a positive integer multiple of the number of parallel processing units 20. Alternatively, the set value can be 0.

[0086] The reset process of the semaphores chunk_count and ar_count will be described below with reference to Table 1.

[0087] Table 1: Correspondence between the number of all-reduce operations of the reduction task and the value of the semaphore

[0088]

[0089] In Table 1, the number of parallel processing units 20 is 8. The preset number threshold is (N-1). N , and is an integer. Preferably, N .

[0090] In Table 1, the i-th all-reduce refers to the order of the all-reduce of the objects in the reduce task, and the number in the row where the i-th all-reduce is located represents the values of the semaphores chunk_count and ar_count corresponding to the i-th all-reduce. i = 1, 2, …, N. As shown in Table 1, the communication amount of the 1st all-reduce is 4 gradient data blocks of data block identifiers ID0-ID3, and accordingly, the values of the semaphores chunk_count and ar_count corresponding to ID0-ID4 are both 8 at the 1st all-reduce. The communication amount of the 2nd all-reduce is 3 gradient data blocks of data block identifiers ID0-ID2, and accordingly, the values of the semaphores chunk_count and ar_count corresponding to ID0-ID3 are both increased by 8 to 16 at the 2nd all-reduce. The communication amount of the Nth all-reduce is 7 gradient data blocks of data block identifiers ID0-ID6, and the values of the semaphores chunk_count and ar_count corresponding to ID0-ID7 are 8 at the Nth all-reduce, which indicates that after the completion of the (N-1)th all-reduce, the number of completed all-reduce operations of the objects in the reduce task reaches the preset number threshold (N-1), and then the semaphores chunk_count and ar_count corresponding to each data block identifier are all initialized to 0.

[0091] In the above embodiment, the counting values of the semaphores are multiplexed in the continuous all-reduce operations of the objects in the reduce task, and the semaphores are reset when the number of completed all-reduce operations of the objects in the reduce task reaches the preset number threshold, which can further reduce the time overhead of the all-reduce operation and improve the gradient synchronization efficiency of the multiple parallel processing units in the distributed training process.

[0092] After the CPU 10 calculates the all-reduce operation of the gradient data block corresponding to one data block identifier, the CPU 10 can copy the all-reduce result of the gradient data block corresponding to the data block identifier to the multiple parallel processing units 20. In the same way, the parallel processing units 20 can obtain the all-reduce result of each reduce task corresponding to any iteration round X; and then, the parallel processing units 20 can update the model parameters of the local neural network model according to the all-reduce result of all reduce tasks corresponding to any iteration round X.

[0093] In addition to the computing system provided in the above embodiment, the present embodiment also provides a model training method. The model training method provided in the present embodiment is exemplarily described below.

[0094] In the model training method provided in the embodiments of the present application, the plurality of parallel processing units can perform distributed training on the same neural network model. For the process of the parallel processing units performing distributed training on the neural network model, reference can be made to the related content of the system embodiments described above, which will not be repeated here. Each iteration round (i.e., each iteration training), the parallel processing units generate gradient data in the iteration round. In the embodiments, in order to synchronize the gradient data between the parallel processing units, the CPU can execute the model training method provided in the embodiments of the present application. Figure 6 and Figure 7 The model training method provided in the embodiments of the present application.

[0095] Figure 6 The flowchart of the model training method provided in the embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the model training method mainly includes the following steps. Figure 6

[0096] 601. For any iteration round X, the gradient data generated by the plurality of parallel processing units in the iteration round is copied to the memory.

[0097] 602. The gradient data generated by the plurality of parallel processing units in the iteration round X is subjected to a global reduction operation to obtain a global reduction result.

[0098] 603. The global reduction result is copied to the plurality of parallel processing units, so that the plurality of parallel processing units update the local neural network model according to the global reduction result.

[0099] The above method is used for multiple iteration training until the neural network model converges, and a trained neural network model is obtained.

[0100] In actual applications, the model parameter scale of the neural network model is large, and the scale of the gradient data generated in one iteration is also large. However, the communication bandwidth of the computing system is limited, and batch data copying is often required. In each iteration round, at least one reduction task needs to be executed. Each reduction task is used to perform a global reduction operation on at least part of the gradient data of the plurality of parallel processing units. In one iteration, the number of reduction tasks to be executed can be flexibly set according to actual conditions.

[0101] Since the gradient data of the plurality of parallel processing units shares the communication bandwidth of the computing system in the copying process, the present application provides a new model training method. Figure 7 The flowchart of another model training method provided in the embodiments of the present application is shown in FIG. 7. As shown in FIG. 7, the method mainly includes the following steps. Figure 7

[0102] ​​701. For any reduction task A corresponding to any iteration X, multiple processes are used to copy gradient data blocks to memory in batches sequentially from the gradient data generated by multiple parallel processing units in any iteration. The amount of gradient data blocks copied by multiple processes in the same batch is the same.

[0103] 702. Configure the same data block identifier for gradient data blocks copied by multiple processes in the same batch.

[0104] 703. Using the first semaphore corresponding to each data block identifier, count the gradient data blocks corresponding to that data block identifier in memory.

[0105] 704. When the increment value of the first semaphore reaches the number of multiple processes, use multiple processes to perform full reduction operations on the gradient data block corresponding to the data block identifier in parallel to obtain the full reduction result of any reduction task A.

[0106] The increment value of the first semaphore for each data block is the increment of the value of the first semaphore compared to the value of the first semaphore when the full reduction operation of the previous reduction task was executed.

[0107] 705. Copy the full reduction result of each reduction task corresponding to any iteration round X to the parallel processing unit so that the parallel processing unit can update the model parameters of the neural network model according to the full reduction result of the reduction task corresponding to iteration round X.

[0108] Because the gradient data of multiple parallel processing units share the communication bandwidth of the computing system during the copying process, in embodiments where the CPU and parallel processing units are connected via a communication interface such as a PCIe interface, the multiple parallel processing units share the PCIe communication bandwidth. Therefore, for any reduction task A corresponding to any iteration round X, when the reduction task A begins execution, in step 701, multiple processes can be used to sequentially copy gradient data blocks into memory from the gradient data generated by the multiple parallel processing units in iteration round X in batches. Each parallel processing unit corresponds to an independent process. A gradient data block refers to a data block composed of the gradient data copied in each batch. The amount of gradient data blocks copied by multiple parallel processing units in the same batch is the same.

[0109] When copying gradient data blocks from multiple parallel processing units to memory in batches as described above, the size of the gradient data blocks copied in different batches can be the same or different. In some embodiments, the size Z of the data block copied each time can be preset. Accordingly, the process corresponding to the parallel processing unit can use the preset data block size Z to sequentially copy gradient data blocks of size Z from the gradient data generated by the parallel processing unit in any iteration X to memory each time.

[0110] Since the computing resource of the CPU and the communication bandwidth of the computing system are limited, the corresponding communication volume S of each reduction task for any iteration round X can be pre-configured. The communication volume S of the reduction task refers to the data volume of each parallel processing unit required for performing the full-reduction operation.

[0111] Accordingly, based on the set data block size Z, the process corresponding to the parallel processing unit can sequentially copy the gradient data block of the size Z from the gradient data generated by the parallel processing unit in any iteration round X to the memory 30 until the data volume copied to the memory by the parallel processing unit reaches the communication volume of the reduction task.

[0112] In some embodiments, the communication volume S is not an integer multiple of the data block size Z, and there is a gradient data block whose data volume is less than Z. In this embodiment, in the case that the data volume to be copied by the parallel processing unit corresponding to the process is less than the set data block size Z, the process corresponding to the parallel processing unit can copy the gradient data block of the data volume to be copied from the gradient data generated by the parallel processing unit in any iteration round X to the memory. The data volume to be copied is equal to the difference between the data volume of the gradient data already copied to the memory by the parallel processing unit and the communication volume S of the reduction task A.

[0113] In the embodiments of the present application, the CPU can allocate shared memory to each of the plurality of processes in the initialization phase. The shared memory refers to a space delimited in the memory by the operating system for synchronization between processes, and different processes can access the same block of shared memory. Based on the shared memory allocated to each process, when copying the gradient data from the parallel processing unit to the memory, the process corresponding to the parallel processing unit can sequentially copy the gradient data block of the set data block size Z from the gradient data generated by the parallel processing unit in any iteration round X to the shared memory of the process corresponding to the parallel processing unit.

[0114] Since the copying of the gradient data of the plurality of parallel processing units is independent of each other, and the full-reduction operation is performed on the gradient data of different parallel processing units, the gradient data of the same parameter needs to be reduced. Therefore, it is necessary to ensure the pipeline synchronization across processes. Based on this, in step 702, the same chunk index can be configured for the gradient data blocks copied in the same batch by the plurality of processes.

[0115] Alternatively, the chunk index can be configured for the currently copied gradient data block based on the copying order of the current batch in the above reduction task A. The gradient data blocks copied in the same batch by the plurality of parallel processing units have the same chunk index.

[0116] In this embodiment, in order to realize synchronization between data replication processes of multiple independent parallel processing units, a first semaphore (denoted as chunk_count) can be configured for each data block identifier. The first semaphore chunk_count is used to count the corresponding gradient data blocks of the data block identifier in the memory. Wherein, the semaphore is incremented by 1 each time the gradient data blocks of the same data block identifier are replicated into the memory. Based on this, in step 703, the first semaphore (chunk_count) corresponding to each data block identifier can be used to count the gradient data blocks corresponding to the data block identifier in the memory.

[0117] In this embodiment, the semaphore is incremented by the number of parallel processing units each time, indicating that the gradient data blocks of the same data block identifier in multiple parallel processing units are replicated, and then the all-reduce operation can be performed on the gradient data blocks corresponding to the data block identifier.

[0118] Specifically, in step 704, when the increment value of the first semaphore chunk_count corresponding to the data block identifier reaches the number of parallel processing units, the all-reduce (AR) operation can be performed on the gradient data blocks corresponding to the data block identifier to obtain the all-reduce operation result of the data gradient blocks corresponding to the data block identifier. The gradient data blocks corresponding to each data block identifier of one reduce task can be all-reduced in the manner shown in the above embodiments to obtain the all-reduce result of the reduce task. For the increment value of the first semaphore chunk_count, please refer to the related content of the above system embodiment, which will not be described here.

[0119] The above embodiments take any reduce task A of any iteration round X as an example to illustrate the all-reduce operation process of the reduce task A. Wherein, each reduce task can be all-reduced in the manner provided in the above embodiments to obtain the all-reduce result of each reduce task of any iteration round X. Further, in step 705, the all-reduce result of each reduce task of an iteration round X can be replicated to multiple parallel processing units. The parallel processing units update the model parameters of the local neural network model according to the all-reduce result of each reduce task corresponding to the iteration round X.

[0120] In the embodiments of the present application, for multiple parallel processing units without P2P direct connection in a distributed training scene, the CPU copies the gradient data generated by the multiple parallel processing units in each iteration process to the memory; after the CPU performs the all-reduce operation on the gradient data of the multiple parallel processing units, the all-reduce result is written back to the multiple parallel processing units, thereby realizing the synchronization of the gradient data between the multiple parallel processing units and helping to improve the convergence speed of the local neural network model of each parallel processing unit.

[0121] The multi-stage pipeline all-reduce process provided by the embodiments of the present application can perform the all-reduce operation on the gradient data block corresponding to the data block identifier after completing the copying of all the gradient data blocks of the data block identifier, and can write the all-reduce result corresponding to the data block identifier back to the multiple parallel processing units after the all-reduce operation on the gradient data corresponding to the data block identifier, thereby ensuring the pipeline execution of the all-reduce process, reducing the waiting time of each stage of the all-reduce process, and improving the all-reduce efficiency.

[0122] On the other hand, the embodiments of the present application introduce a lightweight semaphore mechanism, and perform the all-reduce operation on the gradient data blocks of the multiple parallel processing units when the increment of the semaphore reaches the number of the multiple parallel processing units, thereby realizing the synchronization between the processes of the multiple parallel processing units and improving the cross-process synchronization efficiency compared with the socket communication between processes.

[0123] In the embodiments of the present application, the CPU starts an independent process corresponding to each parallel processing unit. Correspondingly, in order to improve the all-reduce efficiency, in some embodiments, for any data block identifier IDn, the multiple processes corresponding to the multiple parallel processing units can be used to perform the all-reduce operation on the gradient data block corresponding to the data block identifier IDn in parallel.

[0124] Specifically, if the data amount Q of each gradient data block in the gradient data block corresponding to the data block identifier IDn is an integer multiple (assuming K times, K is a positive integer) of the number M of the multiple processes, i.e., K=Q / M, the multiple processes can be used to perform the all-reduce operation on the gradient data block corresponding to the data block identifier IDn in parallel.

[0125] If there is a first gradient data block in the gradient data block corresponding to the data block identifier IDn, the data amount Q of which is not an integer multiple of the number M of the multiple processes, the first process in the M processes can be used to perform the all-reduce operation on the first gradient data block in the gradient data block corresponding to the data block identifier IDn. The first process can be any process in the multiple processes, or a process corresponding to a specified parallel processing unit, or a process numbered Rank0.

[0126] In some embodiments, the data block identification IDn corresponds to a plurality of gradient data blocks, and the plurality of gradient data blocks includes a first gradient data block and a second gradient data block. The second gradient data block is other than the first gradient data block in the plurality of gradient data blocks corresponding to the data block identification IDn, and the number of the second gradient data blocks is equal to the total number of the plurality of gradient data blocks corresponding to the data block identification IDn divided by the number M of the plurality of processes. The second gradient data blocks in the plurality of gradient data blocks corresponding to the data block identification IDn can be parallelly subjected to the all-reduce operation by the plurality of processes.

[0127] To further improve the efficiency of the all-reduce operation, in each independent process, the gradient data corresponding to each of the plurality of processes can be parallelly subjected to the all-reduce operation by the SIMD instruction. Each process is responsible for different gradient data in the gradient data block, and the gradient data corresponding to the plurality of processes constitutes the gradient data block. The SIMD instruction can process multiple data by one instruction.

[0128] Since the processes of the plurality of parallel processing units are independent of each other, and the all-reduce operation requires inter-process synchronization, to realize the inter-process synchronization of the all-reduce operation, another second semaphore (denoted as ar_count) can be configured for each data block identification. The second semaphore ar_count is used to count the processes that have completed the all-reduce operation. When a process that has completed the all-reduce operation is generated, the semaphore ar_count is incremented by 1. Based on this, the processes that have completed the all-reduce operation can be counted by using the semaphore ar_count corresponding to each data block identification, and when the increment value of the semaphore ar_count reaches the number of the plurality of parallel processing units, the all-reduce result corresponding to the data block identification is copied to the plurality of parallel processing units.

[0129] Specifically, when the increment value of the semaphore ar_count corresponding to the data block identification reaches the number of the plurality of parallel processing units, the all-reduce result corresponding to the data block identification is written into the shared memory corresponding to the second process. The second process can be any process in the plurality of processes, or can be the process numbered Rank0. Further, the all-reduce result corresponding to the data block identification can be copied to the plurality of parallel processing units through the shared memory of the second process. The all-reduce result generated by each process in the next iteration round can overwrite the all-reduce result of the current iteration round stored in the shared memory of the process, so as to realize the reuse of the shared memory in the entire model training process.

[0130] In the above CPU-side collective communication process, the shared memory of the plurality of processes is reusable in the entire model training process, and it is not necessary to create a shared memory for each gradient synchronization in each iteration.

[0131] Since the resetting of the semaphores also needs a certain time overhead, in order to further reduce the time overhead and improve the gradient data synchronization in the distributed training process, the first semaphore chunk_count and the second semaphore ar_count can be reset after accumulating a plurality of times of calling the all-reduce operation of the collective communication. Correspondingly, the number threshold R of the all-reduce operation of the reduce task can be configured for the reduce task as a unit object, and when the number of completed all-reduce operations of the reduce task as a unit object reaches the set number threshold R, the first semaphore chunk_count and the second semaphore ar_count are reset. Correspondingly, a counter can be used to count the number of completed all-reduce operations of the reduce task as a unit object. In each time the all-reduce operation of the reduce task level is completed, the counter is incremented by 1. Further, when the value of the counter reaches the set number threshold, the first semaphore chunk_count and the second semaphore ar_count can be initialized to a set value. The set value can be any value that is not equal to a positive integer multiple of the number of parallel processing units. Alternatively, the set value can be 0.

[0132] In the above embodiments, the count value of the semaphore is multiplexed in the continuous all-reduce operation of the reduce task as a unit object, and when the number of completed all-reduce operations of the reduce task as a unit object reaches the set number threshold, the semaphore is reset, which can further reduce the time overhead of the all-reduce operation and improve the gradient synchronization efficiency of the plurality of parallel processing units in the distributed training process.

[0133] After the all-reduce operation of the gradient data block corresponding to the data block identifier is calculated, the all-reduce result of the gradient data block corresponding to the data block identifier can be copied to the plurality of parallel processing units. In the same way, the parallel processing units can obtain the all-reduce result of each reduce task corresponding to any iteration round X; then, the parallel processing units can update the model parameters of the local neural network model according to the all-reduce result of all reduce tasks corresponding to any iteration round X.

[0134] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 701 and 702 can be device A; for example, the execution subject of step 701 can be device A, and the execution subject of step 702 can be device B; and the like.

[0135] In addition, in some of the methods described in this specification and in the accompanying drawings, numerous specific details are provided such as examples of specific types of processing engines, memory, and the like. One skilled in the relevant art will recognize that the application can be practiced without these specific details. In other instances, well-known structures have been omitted so as not to obscure the application. In particular, although the specification and drawings describe and / or illustrate particular flowcharts, it is to be understood that various steps of the flowcharts can be performed in an order different than that shown in the figures. In addition, various steps of the flowcharts can be performed concurrently, in sub-processes, in a different order than depicted in the figures, with other components, with or without specific input, with or without specific output, etc. In addition, some of the flowcharts can include more or fewer steps than shown in the figures. Furthermore, although the specification and drawings describe and / or illustrate particular software, it is to be understood that the application can be practiced using other types of software.

[0136] Accordingly, an embodiment of the present application also provides a computer readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, causing the one or more processors to perform the steps in the model training method.

[0137] It should be noted that the terms "first", "second" and the like in the description do not denote any order. Rather, the terms should be construed as names.

[0138] Those skilled in the art will understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) containing computer usable program code.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing apparatus, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks in the flowcharts and / or block diagrams.

[0140] These computer program instructions can also be stored in a computer readable storage medium that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including an instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks or one or more blocks.

[0141] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed by the computer or other programmable devices provide a process for implementing the flowchart Figure 1 one or more processes and / or blocks Figure 1 Figure 1 the function specified in the one or more blocks or one or more blocks.

[0142] In a typical configuration, a computing device includes one or more processors (CPUs, etc.), input / output interfaces, network interfaces, and memory.

[0143] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as ROM, disk drives, tape drives, etc. Persistent and / or non-volatile memory can be used to store programs, such as an operating system, and processing device data, such as databases and files. The memory is an example of computer readable media.

[0144] The storage media of the computer is a readable storage medium, also known as a readable medium. The readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of storage media of the computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, the computer readable medium does not include transitory media, such as modulated data signals and carriers.

[0145] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0146] The above embodiments of the present application are only used to illustrate the technical solutions of the present application, and not intended to limit the present application. Although the present application is illustrated in detail, those skilled in the art should understand that the present application can make various modifications and changes without departing from the spirit and scope of the present application. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A computing system, characterized in that, include: Memory, central processing unit (CPU), and multiple parallel processing units; The memory, the CPU, and the plurality of parallel processing units are communicatively connected; The CPU has multiple processes corresponding to the multiple parallel processing units; each parallel processing unit corresponds to an independent process. The multiple parallel processing units are used to perform distributed training on the neural network model and generate gradient data of the neural network model in each iteration round; each iteration round corresponds to at least one reduction task; The CPU is used to copy gradient data blocks sequentially and in batches from the gradient data generated by the multiple parallel processing units in each iteration cycle to the memory for any reduction task corresponding to any iteration cycle. The multiple processes replicate gradient data blocks of the same amount in the same batch; Configure the same data block identifier for gradient data blocks that are copied in the same batch by the multiple processes; Using the first semaphore corresponding to the data block identifier, the gradient data blocks corresponding to the data block identifier in the memory are counted; when the increment value of the first semaphore reaches the number of the plurality of processes, the plurality of processes are used to perform a full reduction operation on the gradient data blocks corresponding to the data block identifier in parallel to obtain the full reduction result of any reduction task; the increment value of the first semaphore of the data block identifier is the increment of the value of the first semaphore compared to the value of the first semaphore when the full reduction operation of the previous reduction task was executed; The parallel processing unit is used to update the model parameters of the neural network model based on the full reduction result of the reduction task corresponding to any iteration round.

2. The system according to claim 1, characterized in that, The CPU is also used for: The second semaphore corresponding to the data block identifier is used to count the processes that have completed the full reduction operation; When the increment value of the second semaphore reaches the number of the plurality of parallel processing units, the full reduction result corresponding to the data block identifier is copied to the plurality of parallel processing units; The increment value of the second semaphore is the increment of the value of the second semaphore compared to the value when the full reduction operation of the previous reduction task is completed.

3. The system according to claim 2, characterized in that, Also includes: Use a counter to count the number of times the full reduction operation has been completed for each reduction task. When the value of the counter reaches a set threshold number of times, the first semaphore and the second semaphore are initialized to the set values ​​respectively.

4. A model training method, characterized in that, Multiple parallel processing units perform distributed training on the neural network model and generate gradient data for the neural network model in each iteration round; each iteration round corresponds to at least one reduction task; The method includes: For any reduction task corresponding to any iteration round, multiple processes are used to copy gradient data blocks to memory in batches sequentially from the gradient data generated by each of the multiple parallel processing units in any iteration round; the multiple processes copy gradient data blocks of the same amount in the same batch. Configure the same data block identifier for gradient data blocks that are copied in the same batch by the multiple processes; Using the first semaphore corresponding to each data block identifier, count the gradient data blocks corresponding to that data block identifier in the memory; When the increment value of the first semaphore reaches the number of the plurality of processes, the plurality of processes are used to perform a full reduction operation on the gradient data block corresponding to the data block identifier in parallel to obtain the full reduction result of any reduction task; the increment value of the first semaphore of each data block identifier is the increment of the value of the first semaphore compared to the value of the first semaphore when the full reduction operation of the previous reduction task was executed; The full reduction result of each reduction task corresponding to any iteration round is copied to the parallel processing unit so that the parallel processing unit can update the model parameters of the neural network model based on the full reduction result of the reduction task corresponding to any iteration round.

5. The method according to claim 4, characterized in that, The step of using multiple processes in parallel to perform a full reduction operation on the gradient data block corresponding to the data block identifier includes: In each process, a single instruction multiple data stream instruction is used to perform a full reduction operation in parallel on the gradient data that each of the multiple processes is responsible for, wherein each process is responsible for different gradient data in the gradient data block.

6. The method according to claim 4, characterized in that, Also includes: The second semaphore corresponding to the data block identifier is used to count the processes that have completed the full reduction operation; When the increment value of the second semaphore reaches the number of the plurality of parallel processing units, the full reduction result corresponding to the data block identifier is copied to the plurality of parallel processing units; The increment value of the second semaphore is the increment of the value of the second semaphore compared to the value when the full reduction operation of the previous reduction task is completed.

7. The method according to claim 6, characterized in that, Also includes: Use a counter to count the number of times the full reduction operation has been completed for each reduction task. When the value of the counter reaches a set threshold number of times, the first semaphore and the second semaphore are initialized to the set values ​​respectively.

8. The method according to claim 4, characterized in that, The gradient data block corresponding to the data block identifier contains a first gradient data block whose data volume is not an integer multiple of the number of the multiple processes; the step of using the multiple processes to perform a full reduction operation on the gradient data block corresponding to the data block identifier in parallel includes: For the first gradient data block in the gradient data block corresponding to the data block identifier, the first process among the plurality of processes performs a full reduction operation on the first gradient data block; For the second gradient data block in the gradient data block corresponding to the data block identifier, the multiple processes are used to perform a full reduction operation on the second gradient data in parallel; the second gradient data block is the other gradient data block in the gradient data block corresponding to the data block identifier, excluding the first gradient data block; the number of the second gradient data blocks is equal to the quotient of the total data volume of the gradient data block corresponding to the data block identifier divided by the number of the multiple processes.

9. The method according to claim 4, characterized in that, Also includes: Determine the communication volume of any of the protocol tasks; The step of using the multiple processes to sequentially copy gradient data blocks from the gradient data generated by each of the multiple parallel processing units in any iteration round to the memory includes: For any parallel processing unit, using the process corresponding to the parallel processing unit, each time gradient data blocks of a set data block size are sequentially copied from the gradient data generated by the parallel processing unit in any iteration round to the memory, until the amount of gradient data copied by the parallel processing unit to the memory reaches the communication volume of the reduction task.

10. The method according to claim 9, characterized in that, The communication volume of any protocol task is not an integer multiple of the set data block size; the method further includes: If the amount of data to be copied corresponding to any parallel processing unit is less than the set data block size, a gradient data block of the size of the data to be copied is copied from the gradient data generated by any parallel processing unit in any iteration round to the memory. The amount of data to be copied is equal to the difference between the amount of gradient data that any parallel processing unit has copied to the memory and the amount of communication for any reduction task.

11. The method according to claim 9, characterized in that, Also includes: During the initialization phase, shared memory is allocated to each of the multiple processes. The step of utilizing the process corresponding to any of the parallel processing units to sequentially copy gradient data blocks of the set data block size from the gradient data generated by any of the parallel processing units in any iteration round to the memory includes: Using the process corresponding to any of the parallel processing units, each time gradient data blocks of the set data block size are sequentially copied from the gradient data generated by any of the parallel processing units in any iteration round to the shared memory of the process corresponding to any of the parallel processing units; The method further includes: After the multiple processes perform a full reduction operation on the gradient data block corresponding to the data block identifier in parallel, the full reduction result corresponding to the data block identifier is written into the shared memory of the second process. The full reduction result corresponding to the data block identifier is copied to the multiple parallel processing units through the shared memory corresponding to the second process.

12. The method according to claim 4, characterized in that, The step of configuring the same data block identifier for gradient data blocks copied by the multiple processes in the same batch includes: Based on the replication order of each batch in any of the reduction tasks, a data block identifier is configured for the gradient data blocks replicated in that batch; gradient data blocks replicated in the same replication order in different reduction tasks have the same data block identifier.

13. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 4-12.

Images