Distributed training method and device of neural network with weight pipelining

Abstract

The application provides a neural network distributed training method and device with weight pipeline parallelism, which comprises the following steps: splitting a neural network into multiple calculation stages, representing the calculation stages as forward / backward nodes according to forward / backward operations and arranging the calculation stages in a ring according to sequence to obtain a target wheel disc; marking the positions of the forward / backward nodes in the target wheel disc with device frame labels to evenly distribute the positions of the forward / backward nodes to calculation devices; obtaining the current forward / backward nodes of each calculation device according to the rotation state of the target wheel disc and the device frame; and for each calculation device, performing calculation according to the corresponding forward / backward operation of the current forward / backward node and transmitting the corresponding weight by using training data, thereby obtaining a training result. The application enables each calculation device to perform forward reasoning or backward propagation according to the rotation state of the target wheel disc, reduces the data transmission amount between devices in the neural network training process, and improves the overall training efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a weighted pipelined parallel distributed training method and apparatus for neural networks. Background Technology

[0002] The development of artificial intelligence technology centered on neural networks is inseparable from the support of computing power. Neural networks, especially the highly promising LLM (Limited Linear Model), often require a long training process to achieve good performance. To address this challenge, multi-chip / server clusters are often used to accelerate the entire training process through distributed parallel training. Pipeline parallelism can effectively reduce training time, but it consumes a large number of computing chips such as GPUs. Common distributed parallel techniques for neural networks include data parallelism, tensor parallelism, pipelined parallelism, and hybrid parallelism. These parallel techniques also involve trade-offs in various resources such as computing, communication, and storage.

[0003] A significant drawback of pipelined parallelism is the tendency for devices to enter a bubble state (because the next stage needs to wait for the previous stage to complete), leading to a waste of computational resources. To address this issue, some studies have proposed micro-batch-based pipelined schemes, in which multiple samples in a batch are further split into multiple micro-batches. Each micro-batch is executed separately, further reducing the proportion of bubbles.

[0004] However, during the training of a batch, the amount of communication between devices is directly proportional to the number of samples contained in the batch. When a batch contains a large number of samples, the amount of communication becomes very large.

[0005] In summary, existing pipeline parallel solutions suffer from high communication overhead. Summary of the Invention

[0006] This invention provides a weighted pipelined parallel distributed training method and apparatus for neural networks, which solves the problem of high communication volume in the prior art and realizes pipelined parallel distributed training of neural networks with low communication volume.

[0007] This invention provides a weighted pipelined parallel distributed training method for neural networks, comprising the following steps: The neural network is broken down into multiple computational stages, and these computational stages are represented as forward / backward nodes according to forward / backward operations and arranged in a circular order to obtain the target roulette wheel. The positions of the forward and reverse nodes of the target roulette wheel are marked with device frames to evenly distribute the positions of the forward and reverse nodes to computing devices. Based on the rotation state of the target wheel and the device frame, the current forward / reverse node of each computing device is obtained. For each computing device, the training data is used to calculate and transmit the corresponding weights according to the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result.

[0008] According to the weighted pipelined parallel neural network distributed training method provided by the present invention, the forward and reverse nodes of the target roulette wheel also include weight labels required for the corresponding forward / reverse operations.

[0009] According to the present invention, a weighted pipelined parallel distributed training method for neural networks, wherein for each computing device, the corresponding weights are calculated and transmitted based on the forward / reverse operations corresponding to the current forward / reverse node using training data, specifically including: The training data is divided into several batches of sub-data based on the number of computing devices, and the batches of sub-data are sent to the computing devices. For each computing device, the batch sub-data is used to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node, and the weights are transmitted to the next device.

[0010] According to the distributed training method for neural networks with parallel weight pipeline provided by the present invention, the order of weight transmission between computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

[0011] According to the present invention, a weighted pipelined parallel distributed training method for neural networks is provided, wherein the current forward / backward node of each computing device is obtained based on the rotation state of the target roulette wheel and the device frame; for each computing device, the corresponding weights are calculated and transmitted using training data according to the forward / backward operation corresponding to the current forward / backward node, thereby obtaining the training result; specifically, the method includes: S1: Initialize the target roulette wheel at the initial moment, and obtain the current forward and reverse nodes of the computing device based on the forward and reverse nodes included in the device frame of each computing device; S2: If the computing device is capable of computation, the training data is used to calculate the forward / reverse operation corresponding to the current forward / reverse node, and the corresponding weights are transmitted; if the computing device is unable to perform computation, the computing device is placed in an initial idle state, and the corresponding weights are transmitted. S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until training ends, and output the training results.

[0012] According to the distributed training method for neural networks with weighted pipeline parallelism provided by the present invention, step S2 further includes: When the computing device completes all the reverse operations of a computing phase, the updated weights of the computing phase are transmitted to the next device so that the subsequent device can update based on the updated weights.

[0013] The present invention also provides a weighted pipelined parallel distributed training device for neural networks, comprising the following modules: The roulette unit is used to divide the neural network into multiple computational stages, and the computational stages are represented as forward / backward nodes according to forward / backward operations and arranged in a circular order to obtain the target roulette. A device frame unit is used to mark the positions of the forward and reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the forward and reverse nodes to the computing devices. The computing unit is used to obtain the current forward / reverse node of each computing device based on the rotation state of the target wheel and the device frame. For each computing device, the unit uses training data to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node and transmits the corresponding weights to obtain the training result.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a distributed training method for a neural network with weighted pipeline parallelism as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a distributed training method for a neural network with weighted pipeline parallelism as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a distributed training method for a neural network with weighted pipeline parallelism as described above.

[0017] The present invention provides a weighted pipelined parallel neural network distributed training method and apparatus. This method divides the neural network into multiple computational stages, representing each stage as forward / backward nodes according to their forward / backward operations, and arranging these nodes in a circular sequence to obtain a target wheel. Device frames are used to mark the positions of the forward / backward nodes on the target wheel, distributing these positions evenly among the computing devices. Based on the rotation state of the target wheel and the device frames, the current forward / backward node of each computing device is obtained. For each computing device, training data is used to calculate and transmit corresponding weights based on the forward / backward operations corresponding to the current forward / backward node, thereby obtaining the training result. This invention divides the neural network according to computational stages and the forward / backward operations of each stage, and uses the resulting forward / backward nodes to form the target wheel. Each computing device performs forward inference or backward propagation according to the rotation state of the target wheel. Since the main function between devices in this invention is weight transfer, the amount of data transmitted between devices during neural network training is reduced, improving overall training efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the weighted pipelined parallel distributed training method for neural networks provided by the present invention.

[0020] Figure 2 This is a schematic diagram of parallel data transfer in existing technologies.

[0021] Figure 3 This is a schematic diagram of device execution orchestration for an embodiment of the weighted pipelined parallel neural network distributed training method provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the target roulette wheel rotation state in an embodiment of the weighted pipeline parallel neural network distributed training method provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the target wheel and device frame of an embodiment of the weighted pipelined parallel neural network distributed training method provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the target wheel and device frame of another embodiment of the weighted pipeline parallel neural network distributed training method provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the target wheel and device frame of another embodiment of the weighted pipeline parallel neural network distributed training method provided by the present invention.

[0026] Figure 8 This is one of the schematic diagrams of the single-device execution orchestration of the weighted pipelined parallel neural network distributed training method provided by the present invention.

[0027] Figure 9 This is the second schematic diagram of the single-device execution orchestration of the weighted pipelined parallel neural network distributed training method provided by the present invention.

[0028] Figure 10 This is a schematic diagram of the structure of the weighted pipelined parallel neural network distributed training device provided by the present invention.

[0029] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] The following is combined with Figures 1-9 This invention describes a weighted pipelined parallel distributed training method for neural networks. Figure 1 This is a flowchart illustrating the weighted pipelined parallel distributed training method for neural networks provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 110: Decompose the neural network into multiple computational stages, represent the computational stages as forward / backward nodes according to forward / backward operations, and arrange them in a circular order to obtain the target roulette.

[0032] It should be noted that a neural network consists of multiple computational stages (multiple network layers), each stage including a forward inference process, a backward training process, and a weight update process. For ease of explanation, in this embodiment of the invention, the forward stage is denoted by F*, and the backward stage by B*. In one specific embodiment, the neural network is divided into four computational stages, and the forward and backward nodes include: F1, F2, F3, F4, B4, B3, B2, and B1. It is important to note that the number of computational stages is not related to the number of computing devices; they can be the same or different.

[0033] In existing technologies, the basic pipelined parallel approach breaks down neural networks into multiple computational stages with a sequential relationship according to certain rules. Each stage is assigned to a computing device, and the devices communicate point-to-point. The training is completed by the computing devices in a "relay" manner, such as... Figure 2 As shown. Figure 2 The vertical axis represents different devices, and the horizontal axis represents time. In this basic pipelined parallel approach, the training process of the neural network mainly involves three types of data transfer: the transfer of activation values ​​in the forward pass, the transfer of activation value gradients in the backward pass, and the transfer of activation values ​​from the forward pass required for calculating the gradients of the weights in the backward pass. Throughout the entire training process, the weights at each stage remain fixed on the corresponding device.

[0034] Unlike the basic pipelined parallel approach, this invention transfers weights between multiple devices instead of activation values, and different layers of the network are executed simultaneously between the devices.

[0035] Based on this, the data transmission method of the present invention relies on the target roulette wheel for implementation. Specifically, the present invention first needs to divide the neural network to be trained into multiple computation stages. Since each computation stage includes a forward inference process, a backward training process, and a weight update process, the N computation stages are divided into 2N forward and backward nodes according to the forward inference process and the backward training process.

[0036] Further, the forward and reverse nodes are arranged in a circular pattern to obtain the target roulette wheel. The target roulette wheel is circular and can rotate around its center. The number of its nodes is fixed, K (i.e., the number of forward and reverse nodes), and the wheel rotates a fixed angle each time. In one embodiment, the fixed angle for each rotation is... .

[0037] Step 120: Mark the positions of the forward and reverse nodes of the target wheel with device frames to evenly distribute the positions of the forward and reverse nodes to computing devices.

[0038] After obtaining the target roulette wheel, the device frames are marked. Specifically, each computing device has its own corresponding device frame, which is selected from the position of the forward and reverse nodes in the roulette wheel. In other words, the position of the device frame is fixed. Figure 4 , 5 6 and 7 show different ways to mark device frames.

[0039] It is important to emphasize that during the device frame marking process, the positions of the forward and reverse nodes need to be evenly distributed among the existing computing devices. This invention does not restrict the distribution method; any method can be used for even distribution. It is also important to emphasize that the number of nodes contained in each device frame is fixed, and the shape of the device frame is also fixed after distribution. A valid strategy is one where all frames of all devices can cover all nodes of the wheel.

[0040] Furthermore, in some embodiments, node positions are allocated in such a way that the device frame shapes of the multiple computing devices are all the same.

[0041] Furthermore, the present invention does not limit the computing device; it can be any chip or server.

[0042] Step 130: Based on the rotation state of the target wheel and the device frame, obtain the current forward / reverse node of each computing device. For each computing device, use the training data to calculate and transmit the corresponding weights according to the forward / reverse operation corresponding to the current forward / reverse node, and then obtain the training result.

[0043] After constructing the target wheel and marking the device frames, the forward and backward nodes enclosed by the device frames in the current rotation state of the target wheel are recorded as the current forward and backward nodes. Since each computing device has its corresponding device frame, each computing device also has its corresponding current forward and backward nodes. Therefore, for each device, calculations are performed using training data based on the forward / backward operations corresponding to the current forward and backward nodes, and the corresponding weights are transmitted. This method is used to train the neural network, ultimately obtaining the training result.

[0044] This invention utilizes a target roulette wheel to determine the transmission strategy. As the target roulette wheel rotates, the device frame can contain different forward and reverse nodes. Each computing device performs corresponding operations based on the nodes contained in the device frame, thereby cyclically transmitting weights and weight gradients (i.e., weight pipeline) between the computing devices. Because this invention transmits weights rather than activation values ​​between multiple devices, different layers of the network are executed simultaneously between devices, thereby reducing the amount of communication data between computing devices (chips or servers) and lowering the requirements for communication latency.

[0045] Furthermore, the following describes the corresponding operations performed by each computing device based on the nodes contained in the device frame. In some embodiments, for each computing device, calculating and transmitting corresponding weights using training data based on the forward / reverse operation corresponding to the current forward / reverse node specifically includes: The training data is divided into several batches of sub-data based on the number of computing devices, and the batches of sub-data are sent to the computing devices. For each computing device, the batch sub-data is used to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node, and the weights are transmitted to the next device.

[0046] Specifically, to further accelerate training speed, a data parallelism approach is used, where each computing device processes only a subset of the training data at a time, thus reducing the cost of time bubbles. Specifically, during training, the training data is divided into multiple batches, also known as batch sub-data. Each batch consists of multiple data samples. Each batch is sequentially input into the computing device to perform corresponding operations. In the implementation of this invention, when the neural network is divided into four computational stages, the computation process of one batch includes F1, F2, F3, F4, B4, B3, B2, and B1. In one specific embodiment, the orchestration is as follows: Figure 3 As shown.

[0047] Furthermore, the target roulette wheel is described in more detail. In some embodiments, the forward / reverse nodes of the target roulette wheel also include weight tags required for the corresponding forward / reverse operations.

[0048] Specifically, such as Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the target roulette wheel includes not only the markers for forward / backward operations, but also the markers for the weights corresponding to each operation (the markers for the weight data currently stored in the device), where the weight data can be denoted as W*.

[0049] Furthermore, once the target wheel and device frame are determined, the weight transfer method can be determined. In some embodiments, the order of weight transfer between the computing devices is determined based on the rotation direction of the target wheel and the position of the device frame.

[0050] Specifically, it can be understood that in a certain rotation state of the target wheel, the operation corresponding to the device frame position is the forward or reverse operation that the current device needs to perform, and the weight data (W*) contained in the device frame is the weight data stored by the current device. In a specific embodiment, assuming the target wheel rotates counterclockwise, the neural network is divided into four computational stages, the number of computational devices is also 4, and the shape of the device frame is set as a rectangle with a length parallel to the radius of the target wheel, such as... Figure 4 As shown, the wheel rotates 45° at each interval. Therefore, the order of weight transfer between computing devices is: 1→2→3→4→1.

[0051] Further, step 130 is explained in more detail. In some embodiments, obtaining the current forward / reverse node of each computing device based on the rotation state of the target wheel and the device frame, and for each computing device, calculating and transmitting corresponding weights based on the forward / reverse operation corresponding to the current forward / reverse node using training data to obtain the training result, specifically includes: S1: Initialize the target roulette wheel at the initial moment, and obtain the current forward and reverse nodes of the computing device based on the forward and reverse nodes included in the device frame of each computing device; S2: If the computing device is capable of computation, the training data is used to calculate the forward / reverse operation corresponding to the current forward / reverse node, and the corresponding weights are transmitted; if the computing device is unable to perform computation, the computing device is placed in an initial idle state, and the corresponding weights are transmitted. S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until training ends, and output the training results.

[0052] Specifically, to provide a clearer explanation of step 130, a specific embodiment is given. Assuming the target wheel rotates counter-clockwise, the neural network is divided into four computational stages, and the number of computational devices is also four. The shape of the device frame is set as a rectangle with a length parallel to the radius of the target wheel, as shown below. Figure 4 As shown, the wheel rotates 45° at each interval. Therefore, the operations performed by each device and the weights of the transmissions are as follows: 1) Time 1: Device 1 holds W1 and W4, and transmits these two weights to Device 2, using W1 to calculate its own F1. Device 2 holds W4 and W1, and transmits these two weights to Device 3, and is in an initial idle state. Device 3 holds W2 and W3, and transmits these two weights to Device 4, and is in an initial idle state. Device 4 holds W3 and W2, and transmits these two weights to Device 1, and is in an initial idle state.

[0053] 2) Time 2: Device 1 holds W2 and W3, and transmits these two weights to Device 2, using W2 to calculate its own F2. Device 2 holds W1 and W4, and transmits these two weights to Device 3, using W1 to calculate its own F1. Device 3 holds W1 and W4, and transmits these two weights to Device 4, and is in an initial idle state. Device 4 holds W2 and W3, and transmits these two weights to Device 1, and is in an initial idle state.

[0054] 3) Time 3: Device 1 holds W3 and W2, and transmits these two weights to Device 2, using W3 to calculate its own F3. Device 2 holds W2 and W3, and transmits these two weights to Device 3, using W2 to calculate its own F2. Device 3 holds W1 and W4, and transmits these two weights to Device 4, using W1 to calculate its own F1. Device 4 holds W1 and W4, and transmits these two weights to Device 1, and is in the initial idle state.

[0055] 4) Time 4: Device 1 holds W4 and W1, and transmits these two weights to Device 2, using W4 to calculate its own F4. Device 2 holds W3 and W2, and transmits these two weights to Device 3, using W3 to calculate its own F3. Device 3 holds W2 and W3, and transmits these two weights to Device 4, using W2 to calculate its own F2. Device 4 holds W1 and W4, and transmits these two weights to Device 1, using W1 to calculate its own F1.

[0056] 5) Time 5: Device 1 holds W4 and W1, and transmits these two weights to Device 2, using W4 to calculate its own B4. Device 2 holds W4 and W1, and transmits these two weights to Device 3, using W4 to calculate its own F4. Device 3 holds W3 and W2, and transmits these two weights to Device 4, using W3 to calculate its own F3. Device 4 holds W2 and W3, and transmits these two weights to Device 1, using W2 to calculate its own F2.

[0057] 6) At time 6: Device 1 holds W3 and W2, and transmits these two weights to Device 2, using W3 to calculate its own B3. Device 2 holds W4 and W1, and transmits these two weights to Device 3, using W4 to calculate its own B4. Device 3 holds W4 and W1, and transmits these two weights to Device 4, using W4 to calculate its own F4. Device 4 holds W3 and W2, and transmits these two weights to Device 1, using W3 to calculate its own F3.

[0058] This process can continue indefinitely until training ends and the training results are output. This process only involves circular point-to-point communication from 1 to 2 to 3 to 4 to 1.

[0059] Furthermore, during the training process, the training data is divided into multiple batches of sub-data and distributed to various devices for computation. The more samples each batch of sub-data contains, the more communication is saved compared to traditional pipeline parallelism.

[0060] Furthermore, based on the above embodiments, in some embodiments, step S2 further includes: When the computing device completes all the reverse operations of a computing phase, the updated weights of the computing phase are transmitted to the next device so that the subsequent device can update based on the updated weights.

[0061] Specifically, after a computing device completes the reverse of a stage (layer), it can pass the updated weights of that layer to the next device, and the subsequent device will update based on the updated weights.

[0062] It should be added that, since the reverse calculation time of a stage may be longer than the forward calculation time, not all devices necessarily start or end the calculation of a stage synchronously during the above process; rather, calculation can begin as soon as the data arrives.

[0063] Furthermore, when a computing device contains more than one set of weights, the data transmission time can be adjusted accordingly. For example, at time 2, device 1 transmits weights W2 and W3 to device 2. However, device 2 needs to calculate F2 at time 3, which requires W2. Therefore, device 1 can prioritize transmitting W2 to device 2 and then transmit W3 at time 3.

[0064] Furthermore, based on the above embodiments, in order to reduce the bubble rate of parallel processing and balance the storage at different times, this embodiment of the invention further arranges the execution order within the computing device frame, further achieving forward and reverse interleaved execution, such as... Figure 8 As shown in the diagram. For clarity, green represents F, blue represents B, and yellow represents an idle state. Further, as... Figure 9 As shown, Figure 9 Another example is shown. Figure 8 A similar but more compact approach, which requires more precise control over the timing of communication, requires a forward-reverse interleaving execution scheme.

[0065] The weighted pipelined parallel neural network distributed training method provided by this invention avoids the transmission of activation value gradients between devices, but still requires the transmission of weight gradients between devices. It should be noted that this invention does not limit the method of weight gradient transmission; it can be performed in any manner. In some embodiments, after one device completes the reverse process of a stage x, it transmits the weight gradient Dx of that stage to the next device. After the next device also completes the reverse process of stage x, it averages its own Dx with the received Dx and transmits it again to the subsequent devices. After multiple iterations, all devices update the corresponding weights based on the weight gradients of one or more stages they hold.

[0066] The weighted pipelined parallel distributed training method for neural networks provided by this invention decomposes the neural network into multiple computational stages. These stages are represented as forward / backward nodes according to their forward / backward operations and arranged in a circular order to obtain a target wheel. Device frames are used to mark the positions of the forward / backward nodes on the target wheel, thus evenly distributing these positions among the computing devices. Based on the rotation state of the target wheel and the device frames, the current forward / backward node of each computing device is obtained. For each computing device, training data is used to calculate and transmit corresponding weights based on the forward / backward operations corresponding to the current forward / backward node, thereby obtaining the training result. This invention decomposes the neural network according to computational stages and the forward / backward operations of each stage. The forward / backward nodes obtained from the decomposition form the target wheel, allowing each computing device to perform forward inference or backward propagation according to the rotation state of the target wheel. Since the main function between devices in this invention is weight transfer, the amount of data transmitted between devices during neural network training is reduced, improving overall training efficiency.

[0067] The following describes the weighted pipelined parallel neural network distributed training device provided by the present invention. The weighted pipelined parallel neural network distributed training device described below can be referred to in correspondence with the weighted pipelined parallel neural network distributed training method described above. Figure 10 This is a schematic diagram of the structure of the weighted pipelined parallel neural network distributed training device provided by the present invention, as shown below. Figure 10 As shown, the device includes: The roulette unit 1010 is used to split the neural network into multiple computational stages, represent the computational stages as forward / backward nodes according to forward / backward operations, and arrange them in a circular order to obtain the target roulette. Device frame unit 1020 is used to mark the positions of the forward and reverse nodes of the target wheel with device frames so as to evenly distribute the positions of the forward and reverse nodes to the computing devices. The computing unit 1030 is used to obtain the current forward / reverse node of each computing device based on the rotation state of the target wheel and the device frame. For each computing device, the unit uses training data to calculate and transmit the corresponding weights based on the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result.

[0068] According to the distributed training device for a weighted pipelined parallel neural network provided by the present invention, the forward and reverse nodes of the target roulette wheel further include weight labels required for the corresponding forward / reverse operations.

[0069] According to the present invention, a weighted pipelined parallel neural network distributed training device, wherein for each computing device, the corresponding weights are calculated and transmitted based on the forward / reverse operation corresponding to the current forward / reverse node using training data, specifically including: The training data is divided into several batches of sub-data based on the number of computing devices, and the batches of sub-data are sent to the computing devices. For each computing device, the batch sub-data is used to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node, and the weights are transmitted to the next device.

[0070] According to the present invention, a weighted pipelined parallel neural network distributed training device is provided, wherein the order of weight transmission between the computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

[0071] According to the present invention, a weighted pipelined parallel neural network distributed training device is provided, wherein the current forward / backward node of each computing device is obtained based on the rotation state of the target roulette wheel and the device frame; for each computing device, the corresponding weights are calculated and transmitted using training data according to the forward / backward operation corresponding to the current forward / backward node, thereby obtaining the training result, specifically including: S1: Initialize the target roulette wheel at the initial moment, and obtain the current forward and reverse nodes of the computing device based on the forward and reverse nodes included in the device frame of each computing device; S2: If the computing device is capable of computation, the training data is used to calculate the forward / reverse operation corresponding to the current forward / reverse node, and the corresponding weights are transmitted; if the computing device is unable to perform computation, the computing device is placed in an initial idle state, and the corresponding weights are transmitted. S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until training ends, and output the training results.

[0072] According to the distributed training apparatus for a weighted pipelined parallel neural network provided by the present invention, step S2 further includes: When the computing device completes all the reverse operations of a computing phase, the updated weights of the computing phase are transmitted to the next device so that the subsequent device can update based on the updated weights.

[0073] The weighted pipelined parallel neural network distributed training device provided by this invention decomposes the neural network into multiple computational stages. These stages are represented as forward / backward nodes according to their forward / backward operations and arranged in a circular order to obtain a target wheel. Device frames are used to mark the positions of the forward / backward nodes on the target wheel, thus evenly distributing these positions among the computing devices. Based on the rotation state of the target wheel and the device frames, the current forward / backward node of each computing device is obtained. For each computing device, training data is used to calculate and transmit corresponding weights based on the forward / backward operations corresponding to the current forward / backward node, thereby obtaining the training result. This invention decomposes the neural network according to computational stages and the forward / backward operations of each stage. The forward / backward nodes obtained from the decomposition form the target wheel, allowing each computing device to perform forward inference or backward propagation according to the rotation state of the target wheel. Since the main function between devices in this invention is weight transfer, the amount of data transmitted between devices during neural network training is reduced, improving overall training efficiency.

[0074] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a weighted pipelined parallel neural network distributed training method. This method includes: splitting the neural network into multiple computational stages; representing the computational stages as forward / backward nodes according to forward / backward operations and arranging them in a circular order to obtain a target wheel; marking the positions of the forward / backward nodes of the target wheel with device frames to evenly distribute the positions of the forward / backward nodes to computing devices; obtaining the current forward / backward node of each computing device according to the rotation state of the target wheel and the device frames; for each computing device, using training data, calculating and transmitting corresponding weights according to the forward / backward operations corresponding to the current forward / backward node, thereby obtaining the training result.

[0075] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the weight pipelined parallel neural network distributed training method provided by the above methods. The method includes: splitting the neural network into multiple computational stages; representing the computational stages as forward / backward nodes according to forward / backward operations and arranging them in a circular order to obtain a target roulette wheel; marking the positions of the forward / backward nodes of the target roulette wheel with device frames to evenly distribute the positions of the forward / backward nodes to computing devices; obtaining the current forward / backward node of each computing device according to the rotation state of the target roulette wheel and the device frames; for each computing device, using training data to calculate and transmit corresponding weights according to the forward / backward operations corresponding to the current forward / backward node, thereby obtaining a training result.

[0077] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a weight-pipelined parallel distributed training method for neural networks provided by the above methods. The method includes: splitting the neural network into multiple computational stages; representing the computational stages as forward / backward nodes according to forward / backward operations and arranging them in a circular order to obtain a target roulette wheel; marking the positions of the forward / backward nodes of the target roulette wheel with device frames to evenly distribute the positions of the forward / backward nodes to computing devices; obtaining the current forward / backward node of each computing device according to the rotation state of the target roulette wheel and the device frames; for each computing device, using training data to perform calculations according to the forward / backward operations corresponding to the current forward / backward node and transmitting the corresponding weights to obtain a training result.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weighted pipelined parallel distributed training method for neural networks, characterized in that, include: The neural network is divided into multiple computational stages, and the computational stages are represented as forward / backward nodes according to forward / backward operations and arranged in a circular order to obtain the target roulette. The forward / backward nodes of the target roulette also include the weight labels required for the corresponding forward / backward operations. The positions of the forward and reverse nodes of the target roulette are marked with device frames to evenly distribute the positions of the forward and reverse nodes to computing devices; Based on the rotation state of the target wheel and the device frame, the current forward / reverse node of each computing device is obtained. For each computing device, the training data is used to calculate and transmit the corresponding weights according to the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result.

2. The weighted pipelined parallel distributed training method for neural networks according to claim 1, characterized in that, For each computing device, the step of calculating and transmitting corresponding weights based on the forward / reverse operation corresponding to the current forward / reverse node using training data specifically includes: The training data is divided into several batches of sub-data based on the number of computing devices, and the batches of sub-data are sent to the computing devices. For each computing device, the batch sub-data is used to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node, and the weights are transmitted to the next device.

3. The weighted pipelined parallel distributed training method for neural networks according to claim 1, characterized in that, The order of weight transfer between the computing devices is determined based on the rotation direction of the target wheel and the position of the device frame.

4. The weighted pipelined parallel distributed training method for neural networks according to claim 1 or 2, characterized in that, The process involves obtaining the current forward / reverse node of each computing device based on the rotation state of the target wheel and the device frame. For each computing device, training data is used to calculate and transmit corresponding weights based on the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result. Specifically, this includes: S1: Initialize the target roulette wheel at the initial moment, and obtain the current forward and reverse nodes of the computing device based on the forward and reverse nodes included in the device frame of each computing device; S2: If the computing device is capable of computation, the training data is used to calculate the forward / reverse operation corresponding to the current forward / reverse node, and the corresponding weights are transmitted; if the computing device is unable to perform computation, the computing device is placed in an initial idle state, and the corresponding weights are transmitted. S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until training ends, and output the training results.

5. The weighted pipelined parallel distributed training method for neural networks according to claim 4, characterized in that, Step S2 further includes: When the computing device completes all the reverse operations of a computing phase, the updated weights of the computing phase are transmitted to the next device so that the subsequent device can update based on the updated weights.

6. A weighted pipelined parallel distributed training device for neural networks, characterized in that, include: The roulette unit is used to divide the neural network into multiple computational stages, and to represent the computational stages as forward / backward nodes according to forward / backward operations and arrange them in a circular order to obtain the target roulette; the forward / backward nodes of the target roulette also include weight labels required for the corresponding forward / backward operations. A device frame unit is used to mark the positions of the forward and reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the forward and reverse nodes to the computing devices. The computing unit is used to obtain the current forward / reverse node of each computing device based on the rotation state of the target wheel and the device frame. For each computing device, the unit uses training data to perform calculations based on the forward / reverse operation corresponding to the current forward / reverse node and transmits the corresponding weights to obtain the training result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the weighted pipelined parallel neural network distributed training method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the weighted pipelined parallel neural network distributed training method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the weighted pipelined parallel neural network distributed training method as described in any one of claims 1 to 5.

Images