Weight pipeline parallel neural network distributed training method and device

By splitting the neural network into multiple computing stages and forming a target roulette arranged in a ring, combining the marking and rotation state of the device box, the circular transmission of weights is achieved, solving the problem of large communication volume in pipeline parallelism, and improving training efficiency.

CN119067196AActive Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410975763.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-12-03
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

The existing pipeline parallel neural network distributed training technology has the problem of large communication volume, resulting in waste of computing resources and low training efficiency.

Method used

Using the weight flow parallel method, the neural network is split into multiple calculation stages, and the target roulette is arranged in an annular arrangement according to the front/reverse operation, and the calculation device is distributed evenly through the device box, using the rotation state and equipment of the target roulette. The box determines the order of weight transmission and realizes the ring transmission of weights.

Benefits of technology

It reduces the amount of data transmission between devices, improves the overall efficiency of neural network training, and reduces communication delay and waste of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067196A_ABST
    Figure CN119067196A_ABST
Patent Text Reader

Abstract

The invention provides a weight pipeline parallel neural network distributed training method and device, and the method comprises the steps: dividing a neural network into a plurality of calculation stages, representing the calculation stages as forward and reverse nodes according to forward / reverse operation, and carrying out the annular arrangement of the forward and reverse nodes according to a sequence, and obtaining a target wheel disc; carrying out equipment frame marking on the positions of the front reverse nodes of the target wheel disc so as to averagely distribute the positions of the front reverse nodes to computing equipment; according to the rotation state of the target wheel disc and the device frame, obtaining a current forward and reverse node of each computing device, and for each computing device, using training data to calculate and transmit a corresponding weight according to forward / reverse operation corresponding to the current forward and reverse node so as to obtain a training result. According to the method, each computing device carries out forward reasoning or back propagation according to the rotation state of the target wheel disc, the data transmission amount between devices in the neural network training process is reduced, and the overall training efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a weighted pipeline parallel distributed training method and device for a neural network. Background Art

[0002] The development of artificial intelligence technology centered on neural networks cannot be separated from the support of computing power. Neural networks, especially LLMs with great potential, often require a long training process to achieve a good performance level. In order to meet such challenges, people often use multi-chip / server clusters to accelerate the entire training process through distributed parallel training. Pipeline parallel solutions can effectively reduce training time, but they will consume a large number of computing chips such as GPUs. Common neural network distributed parallel technologies include data parallelism, tensor parallelism, pipeline parallelism, and hybrid parallelism. These parallel technologies will also make trade-offs in various resources such as computing, communication, and storage.

[0003] An obvious disadvantage of pipeline parallelism is that the device is prone to bubble idle state (because the next stage needs to wait for the previous stage to complete execution), resulting in a waste of computing resources. To solve this problem, some studies have proposed a pipeline solution based on micro-batch, in which multiple samples in a batch are further split into multiple micro-batches. Each micro-batch is executed separately, and the proportion of bubbles will be further reduced under this strategy.

[0004] However, during the training of a batch, the amount of communication between devices is proportional to the number of samples in the batch. When a batch contains many samples, the amount of communication will be very large.

[0005] In summary, the existing pipeline parallel scheme has the problem of large communication volume. Summary of the invention

[0006] The present invention provides a weighted pipeline parallel distributed training method and device for a neural network, which are used to solve the defect of large communication volume in the prior art and realize pipeline parallel distributed training of a neural network with small communication volume.

[0007] The present invention provides a weight pipeline parallel distributed neural network training method, comprising the following steps: Splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette wheel; Marking the positions of the front reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the front reverse nodes to computing devices; The current forward / reverse node of each computing device is obtained according to the rotation state of the target wheel and the device frame. For each computing device, the training data is used to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result.

[0008] According to a weighted pipeline parallel distributed training method for a neural network provided by the present invention, the forward and reverse nodes of the target roulette wheel also include weight tags required for corresponding forward / reverse operations.

[0009] According to a weight pipeline parallel distributed training method for a neural network provided by the present invention, for each of the computing devices, using training data to calculate and transmit corresponding weights according to the forward / reverse operations corresponding to the current forward and reverse nodes, specifically includes: Splitting the training data into a plurality of batches of sub-data according to the number of the computing devices, and sending the batches of sub-data to the computing devices; For each of the computing devices, the batch sub-data is used to perform calculations according to the forward / reverse operations corresponding to the current forward / reverse node and weight transmission is performed to the next device.

[0010] According to a weight pipeline parallel distributed training method for a neural network provided by the present invention, the order of weight transmission between the computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

[0011] According to a weighted pipeline parallel distributed training method for a neural network provided by the present invention, the current forward and reverse nodes of each computing device are obtained according to the rotation state of the target wheel and the device frame, and for each computing device, the training data is used to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward and reverse node, thereby obtaining the training result, which specifically includes: S1: Initializing the target roulette wheel at an initial moment, and obtaining the current forward reverse node of the computing device according to the forward reverse node included in the device frame of each computing device; S2: when the computing device is capable of computing, using the training data to compute the forward / reverse operation corresponding to the current forward / reverse node, and transmitting the corresponding weight; when the computing device is unable to compute, placing the computing device in an initial idle state, and transmitting the corresponding weight; S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until the training is completed, and output the training results.

[0012] According to a weight pipeline parallel distributed neural network training method provided by the present invention, step S2 further includes: When the computing device completes all reverse operations of a computing phase, the weights updated in the computing phase are transmitted to the next device, so that the subsequent devices are updated based on the updated weights.

[0013] The present invention also provides a weighted pipeline parallel neural network distributed training device, comprising the following modules: A roulette unit, used for splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette; A device frame unit, used for marking the positions of the front reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the front reverse nodes to computing devices; The computing unit is used to obtain the current forward / reverse node of each computing device according to the rotation state of the target wheel and the device frame, and for each computing device, calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using the training data, so as to obtain the training result.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the weight pipeline parallel distributed training methods for a neural network as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the weight pipeline parallel distributed training methods for a neural network as described above.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the weight pipeline parallel distributed training methods for a neural network as described above.

[0017] The weighted pipeline parallel neural network distributed training method and device provided by the present invention splits the neural network into multiple calculation stages, represents the calculation stages as forward / reverse nodes according to forward / reverse operations and arranges them in a circular order to obtain a target wheel; marks the positions of the forward / reverse nodes of the target wheel with device frames to evenly distribute the positions of the forward / reverse nodes to the calculation devices; obtains the current forward / reverse node of each calculation device according to the rotation state of the target wheel and the device frame, and for each calculation device, calculates and transmits the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using training data, thereby obtaining the training result. The present invention splits the neural network according to the calculation stage and the forward / reverse operation of each stage, and forms a target wheel according to the forward / reverse nodes obtained by the splitting, so that each calculation device performs forward reasoning or reverse propagation according to the rotation state of the target wheel. Since the weights are mainly transferred between the various devices of the present invention, the data transmission volume between the devices during the neural network training process is reduced, and the overall training efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the weight pipeline parallel distributed training method of a neural network provided by the present invention.

[0020] Figure 2 It is a schematic diagram of pipeline parallel data transmission in the prior art.

[0021] Figure 3 It is a schematic diagram of the device execution arrangement of an embodiment of the weight pipeline parallel distributed training method of a neural network provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the target wheel rotation state of an embodiment of the weighted pipeline parallel neural network distributed training method provided by the present invention.

[0023] Figure 5 It is a schematic diagram of a target wheel and a device block diagram of an embodiment of a weighted pipeline parallel distributed training method for a neural network provided by the present invention.

[0024] Figure 6 It is a schematic diagram of a target wheel and a device block diagram of another embodiment of the weighted pipeline parallel distributed training method for a neural network provided by the present invention.

[0025] Figure 7 It is a schematic diagram of a target wheel and a device block diagram of another embodiment of the weighted pipeline parallel distributed training method for a neural network provided by the present invention.

[0026] Figure 8 This is one of the schematic diagrams of single device execution arrangement of the weight pipeline parallel neural network distributed training method provided by the present invention.

[0027] Fig. 9 This is the second schematic diagram of the execution arrangement of a single device of the weight pipeline parallel neural network distributed training method provided by the present invention.

[0028] Fig.10 It is a structural schematic diagram of a weighted pipeline parallel neural network distributed training device provided by the present invention.

[0029] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Combine the following Figure 1-Figure 9 Describe the weight pipeline parallel distributed neural network training method of the present invention, Figure 1 : is a flow chart of the weighted pipeline parallel neural network distributed training method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: Split the neural network into multiple calculation stages, represent the calculation stages as forward / reverse nodes according to forward / reverse operations and arrange them in a ring in sequence to obtain a target roulette wheel.

[0032] It should be noted that the neural network consists of multiple computing stages (multiple network layers), each of which includes a forward reasoning process, a reverse training process, and a weight update process. For the convenience of explanation, in an embodiment of the present invention, the forward stage is represented by F*, and the reverse stage is represented by B*. In a specific embodiment, the neural network is divided into 4 computing stages, and the forward and reverse nodes include: F1, F2, F3, F4, B4, B3, B2, and B1. It should be noted that the number of computing stages is not related to the number of computing devices, and can be the same or different.

[0033] In the prior art, the basic pipeline parallel method divides the neural network into multiple computing stages with a sequential relationship according to certain rules, assigns each stage to a computing device, and the devices communicate point-to-point. The computing devices complete the training in a "relay" manner, such as Figure 2 shown. Figure 2 The vertical axis represents different devices, and the horizontal axis represents time. In this basic pipeline parallel method, the training process of the neural network mainly involves three types of data transfer: the transfer of activation values ​​in the forward process, the transfer of activation value gradients in the reverse process, and the transfer of activation values ​​corresponding to the forward process required when the reverse process calculates the gradient of the weight. During the entire training process, the weights of each stage will be fixed on the corresponding device.

[0034] The present invention is different from the basic pipeline parallel method. The present invention transmits weights instead of activation values ​​between multiple devices, and different layers of the network are executed simultaneously between each device.

[0035] On this basis, the data transmission method of the present invention is implemented by relying on the target roulette wheel. Specifically, the present invention first needs to divide the neural network to be trained into multiple calculation stages. Since each calculation stage includes a forward reasoning process, a reverse training process and a weight updating process, the N calculation stages are divided into 2N forward and backward nodes according to the forward reasoning process and the reverse training process.

[0036] Furthermore, the forward and reverse nodes are arranged in a circular order to obtain a target wheel. The target wheel is circular and can rotate around the center of the circle. The number of its nodes is fixed, which is K (i.e., the number of forward and backward nodes). The wheel rotates a fixed angle each time. In one embodiment, the fixed angle of each rotation is .

[0037] Step 120: Mark the positions of the front reverse nodes of the target wheel with device frames to evenly distribute the positions of the front reverse nodes to computing devices.

[0038] After obtaining the target wheel, the device frame is marked. Specifically, each computing device has its own corresponding device frame, and the device frame is selected at the position of the front and back nodes in the wheel, that is, the position of the device frame is fixed. Figure 4 , 5 , 6, and 7 show different ways of marking device frames.

[0039] It should be emphasized that in the process of marking the device frame, the positions of the front and back nodes need to be evenly distributed to the existing computing devices. The present invention does not limit the distribution method, and the even distribution can be performed in any way. It should be emphasized that the number of nodes contained in each device frame is fixed, and after the distribution is completed, the shape of the device frame is also fixed. As long as all frames of all devices can cover all nodes of the roulette wheel, it can be a legal strategy.

[0040] Furthermore, in some embodiments, node positions are allocated in such a way that device frame shapes of multiple computing devices are all the same.

[0041] In addition, the present invention is not limited to the computing device, and it can be any chip or server.

[0042] Step 130: Obtain the current forward / reverse node of each computing device according to the rotation state of the target wheel and the device frame. For each computing device, calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using the training data, thereby obtaining the training result.

[0043] After completing the construction of the target wheel and marking the device frame, the forward and backward nodes framed by the device frame in the current rotation state of the target wheel are recorded as the current forward and backward nodes. Since each computing device has its corresponding device frame, each computing device has its corresponding current forward and backward nodes. Therefore, for each device, the training data is used to calculate the forward / reverse operation corresponding to the current forward and backward nodes, and the corresponding weights are transmitted at the same time. In this way, the neural network is trained to finally obtain the training results.

[0044] The present invention determines the transmission strategy with the help of the target wheel. As the target wheel rotates, the device frame can contain different forward and reverse nodes. Each computing device performs corresponding operations according to the nodes contained in the device frame, so that the weights and the gradients of the weights (i.e., weight flow) are cyclically transmitted between the computing devices. Since the present invention transmits weights between multiple devices instead of activation values, different layers of the network are executed simultaneously between the devices, thereby reducing the amount of communication data between computing devices (chips or servers) and reducing the requirements for communication delay.

[0045] Further, the following further describes the operations performed by each computing device according to the nodes included in the device box. In some embodiments, for each computing device, using training data to calculate and transmit corresponding weights according to the forward / reverse operations corresponding to the current forward / reverse nodes, specifically includes: Splitting the training data into a plurality of batches of sub-data according to the number of the computing devices, and sending the batches of sub-data to the computing devices; For each of the computing devices, the batch sub-data is used to perform calculations according to the forward / reverse operations corresponding to the current forward / reverse node and weight transmission is performed to the next device.

[0046] Specifically, in order to further speed up the training, the idea of ​​data parallelism is used to reduce the cost of generating time bubbles by allowing each computing device to process only a subset of the training data at a time. Specifically, during the training process, the training data will be divided into multiple batches, which can also be called batch sub-data. Among them, a batch consists of multiple data samples. Each batch is input into the computing device in turn to perform the corresponding operation. In the implementation process of the present invention, when the neural network is divided into four computing stages, the calculation process of a batch includes F1, F2, F3, F4, B4, B3, B2, and B1. In a specific embodiment, the execution is arranged as follows Figure 3 shown.

[0047] Further, the target wheel is further described. In some embodiments, the forward / reverse node of the target wheel further includes a weight mark required for the corresponding forward / reverse operation.

[0048] Specifically, Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the target roulette wheel includes not only the mark of the forward / backward operation, but also the mark of the weight corresponding to each operation (the mark of the weight data stored in the current device), where the weight data can be recorded as W*.

[0049] Further, after the target wheel and the device frame are determined, the weight transmission mode can be determined. In some embodiments, the order of weight transmission between the computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

[0050] Specifically, it can be understood that, in a certain rotation state of the target wheel, the operation corresponding to the position of the device frame is the forward or reverse operation that the current device needs to perform, and the weight data (W*) contained in the device frame is the weight data stored in the current device. In a specific embodiment, assuming that the target wheel rotates counterclockwise, the neural network is divided into four calculation stages, the number of calculation devices is also 4, and the shape of the device frame is set to a rectangle with a length parallel to the radius of the target wheel, such as Figure 4 As shown, at each moment, the wheel rotates 45°. Then, the order of weight transmission between computing devices is: 1→2→3→4→1.

[0051] Further, step 130 is further described. In some embodiments, the current forward and reverse nodes of each computing device are obtained according to the rotation state of the target wheel and the device frame, and for each computing device, the training data is used to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward and reverse node, thereby obtaining the training result, which specifically includes: S1: Initializing the target roulette wheel at an initial moment, and obtaining the current forward reverse node of the computing device according to the forward reverse node included in the device frame of each computing device; S2: when the computing device is capable of computing, using the training data to compute the forward / reverse operation corresponding to the current forward / reverse node, and transmitting the corresponding weight; when the computing device is unable to compute, placing the computing device in an initial idle state, and transmitting the corresponding weight; S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until the training is completed, and output the training results.

[0052] Specifically, in order to explain step 130 more clearly, a specific embodiment is given. Assuming that the target wheel rotates counterclockwise, the neural network is divided into four calculation stages, the number of calculation devices is also 4, and the shape of the device frame is set to a rectangle with a length parallel to the radius of the target wheel, such as Figure 4 As shown, at each moment, the wheel rotates 45°. Then, the operations performed by each device and the weight of the transmission are as follows: 1) Time 1: Device 1 holds W1 and W4, and transmits these two weights to device 2, which uses W1 to calculate its own F1. Device 2 holds W4 and W1, and transmits these two weights to device 3, and is in the initial idle state. Device 3 holds W2 and W3, and transmits these two weights to device 4, and is in the initial idle state. Device 4 holds W3 and W2, and transmits these two weights to device 1, and is in the initial idle state.

[0053] 2) Time 2: Device 1 holds W2 and W3, and transmits these two weights to device 2, using W2 to calculate its own F2. Device 2 holds W1 and W4, and transmits these two weights to device 3, using F1 to calculate its own F1. Device 3 holds W1 and W4, and transmits these two weights to device 4, and is in the initial idle state. Device 4 holds W2 and W3, and transmits these two weights to device 1, and is in the initial idle state.

[0054] 3) Time 3: Device 1 holds W3 and W2, and transmits these two weights to device 2, using W3 to calculate its own F3. Device 2 holds W2 and W3, and transmits these two weights to device 3, using W2 to calculate its own F2. Device 3 holds W1 and W4, and transmits these two weights to device 4, using W1 to calculate its own F1. Device 4 holds W1 and W4, and transmits these two weights to device 1, and is in the initial idle state.

[0055] 4) Time 4: Device 1 holds W4 and W1, and transmits these two weights to device 2, which uses W4 to calculate its own F4. Device 2 holds W3 and W2, and transmits these two weights to device 3, which uses W3 to calculate its own F3. Device 3 holds W2 and W3, and transmits these two weights to device 4, which uses W2 to calculate its own F2. Device 4 holds W1 and W4, and transmits these two weights to device 1, which uses W1 to calculate its own F1.

[0056] 5) Time 5: Device 1 holds W4 and W1, and transmits these two weights to device 2, which uses W4 to calculate its own B4. Device 2 holds W4 and W1, and transmits these two weights to device 3, which uses W4 to calculate its own F4. Device 3 holds W3 and W2, and transmits these two weights to device 4, which uses W3 to calculate its own F3. Device 4 holds W2 and W3, and transmits these two weights to device 1, which uses W2 to calculate its own F2.

[0057] 6) Time 6: Device 1 holds W3 and W2, and transmits these two weights to device 2, which uses W3 to calculate its own B3. Device 2 holds W4 and W1, and transmits these two weights to device 3, which uses W4 to calculate its own B4. Device 3 holds W4 and W1, and transmits these two weights to device 4, which uses W4 to calculate its own F4. Device 4 holds W3 and W2, and transmits these two weights to device 1, which uses W3 to calculate its own F3.

[0058] This process can be continued in this way until the training is completed and the training results are output. This process only includes the circular point-to-point communication of 1→2→3→4→1.

[0059] Furthermore, during the training process, the training data is divided into multiple batches of sub-data and allocated to each device for calculation. The more samples each device's batch sub-data contains, the more communication traffic is saved compared to traditional pipeline parallelism.

[0060] Further, based on the above embodiment, in some embodiments, step S2 further includes: When the computing device completes all reverse operations of a computing phase, the weights updated in the computing phase are transmitted to the next device, so that the subsequent devices are updated based on the updated weights.

[0061] Specifically, after a computing device completes the reverse of a stage (a layer), the updated weights of this layer can be passed to the next device, and the subsequent devices are updated based on the updated weights.

[0062] It should be added that, since the time for reverse calculation of a stage may be longer than that for forward calculation, in the above process, not all devices necessarily start or end the calculation of a stage synchronously, but calculation can start as soon as data arrives.

[0063] Furthermore, in the case where a computing device includes more than one weight, the time of data transmission can also be changed according to the situation. For example, at time 2, device 1 transmits two weights, W2 and W3, to device 2. However, device 2 needs to calculate F2 first at time 3, and needs W2. Therefore, device 1 can preferentially transmit W2 to device 2, and transmit W3 at time 3.

[0064] Furthermore, based on the above embodiment, in order to reduce the bubble rate of pipeline parallelism and balance the storage at different times, the embodiment of the present invention further arranges the execution order within the computing device frame, and further achieves forward and reverse interleaved execution, such as Figure 8 In this figure, for the sake of clarity, green represents F, blue represents B, and yellow represents idle state. Fig. 9 As shown, Fig. 9 Shows another Figure 8 A similar but more compact and more precise forward-reverse interleaving implementation is required for control of communication timing.

[0065] The weighted pipeline parallel distributed training method for a neural network provided by the present invention avoids the transmission of the gradient of the activation value between devices, but still needs to transmit the gradient of the weight between devices. It should be noted that the present invention does not limit the way the weight gradient is transmitted, and the transmission can be performed in any way. In some embodiments, when a device calculates the reverse process of a stage x, the weight gradient Dx of this stage is transmitted to the next device. After the next device also calculates the reverse of stage x, it adds its own Dx and the received Dx to obtain the average, and transmits it to the subsequent device again. After multiple cycles, all devices update the corresponding weights with the weight gradients of a certain stage or several stages that they hold.

[0066] The weighted pipeline parallel neural network distributed training method provided by the present invention is to split the neural network into multiple calculation stages, represent the calculation stages as forward and reverse nodes according to forward / reverse operations and arrange them in a ring in order to obtain a target wheel; mark the position of the forward and reverse nodes of the target wheel with a device frame to evenly distribute the position of the forward and reverse nodes to the calculation devices; obtain the current forward and reverse nodes of each calculation device according to the rotation state of the target wheel and the device frame, and for each calculation device, calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward and reverse node using training data, thereby obtaining the training result. The present invention splits the neural network according to the calculation stage and the forward / reverse operation of each stage, and forms a target wheel according to the forward and reverse nodes obtained by the splitting, so that each calculation device performs forward reasoning or reverse propagation according to the rotation state of the target wheel. Since the weights are mainly transferred between the various devices of the present invention, the data transmission amount between the devices during the neural network training process is reduced, and the overall training efficiency is improved.

[0067] The weight-pipeline parallel distributed neural network training device provided by the present invention is described below. The weight-pipeline parallel distributed neural network training device described below and the weight-pipeline parallel distributed neural network training method described above can be referenced to each other. Fig.10 : is a schematic diagram of the structure of the weighted pipeline parallel neural network distributed training device provided by the present invention, such as Fig.10 As shown, the device comprises: The roulette unit 1010 is used to split the neural network into multiple calculation stages, represent the calculation stages as forward / reverse nodes according to the forward / reverse operation and arrange them in a ring in sequence to obtain a target roulette; The device frame unit 1020 is used to mark the positions of the front reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the front reverse nodes to computing devices; The computing unit 1030 is used to obtain the current forward / reverse node of each computing device according to the rotation state of the target wheel and the device frame, and for each computing device, calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using the training data, so as to obtain the training result.

[0068] According to a weighted pipeline parallel distributed training device for a neural network provided by the present invention, the forward and reverse nodes of the target roulette wheel also include weight tags required for corresponding forward / reverse operations.

[0069] According to a weight pipeline parallel distributed training apparatus for a neural network provided by the present invention, for each of the computing devices, using training data to calculate and transmit corresponding weights according to the forward / reverse operations corresponding to the current forward and reverse nodes, specifically includes: Splitting the training data into a plurality of batches of sub-data according to the number of the computing devices, and sending the batches of sub-data to the computing devices; For each of the computing devices, the batch sub-data is used to perform calculations according to the forward / reverse operations corresponding to the current forward / reverse node and weight transmission is performed to the next device.

[0070] According to a weight pipeline parallel distributed training device for a neural network provided by the present invention, the order of weight transmission between the computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

[0071] According to a weighted pipeline parallel neural network distributed training device provided by the present invention, the current forward and reverse nodes of each computing device are obtained according to the rotation state of the target wheel and the device frame, and for each computing device, the training data is used to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward and reverse node, thereby obtaining the training result, which specifically includes: S1: Initializing the target roulette wheel at an initial moment, and obtaining the current forward reverse node of the computing device according to the forward reverse node included in the device frame of each computing device; S2: when the computing device is capable of computing, using the training data to compute the forward / reverse operation corresponding to the current forward / reverse node, and transmitting the corresponding weight; when the computing device is unable to compute, placing the computing device in an initial idle state, and transmitting the corresponding weight; S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until the training is completed, and output the training results.

[0072] According to a weight pipeline parallel distributed neural network training device provided by the present invention, step S2 further includes: When the computing device completes all reverse operations of a computing phase, the weights updated in the computing phase are transmitted to the next device, so that the subsequent devices are updated based on the updated weights.

[0073] The weighted pipeline parallel neural network distributed training device provided by the present invention splits the neural network into multiple calculation stages, represents the calculation stages as forward and reverse nodes according to forward / reverse operations and arranges them in a circular order to obtain a target wheel; marks the position of the forward and reverse nodes of the target wheel with a device frame to evenly distribute the position of the forward and reverse nodes to the calculation devices; obtains the current forward and reverse nodes of each calculation device according to the rotation state of the target wheel and the device frame, and for each calculation device, calculates and transmits the corresponding weight according to the forward / reverse operations corresponding to the current forward and reverse nodes using training data, thereby obtaining the training result. The present invention splits the neural network according to the calculation stage and the forward / reverse operations of each stage, and forms a target wheel according to the forward and reverse nodes obtained by the splitting, so that each calculation device performs forward reasoning or reverse propagation according to the rotation state of the target wheel. Since the weights are mainly transferred between the various devices of the present invention, the data transmission volume between the devices during the neural network training process is reduced, and the overall training efficiency is improved.

[0074] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute a weight pipeline parallel neural network distributed training method, the method comprising: splitting the neural network into multiple calculation stages, representing the calculation stages as forward and reverse nodes according to forward / reverse operations and arranging them in a circular order to obtain a target wheel; marking the positions of the forward and reverse nodes of the target wheel with device frames to evenly distribute the positions of the forward and reverse nodes to the calculation devices; obtaining the current forward and reverse nodes of each of the calculation devices according to the rotation state of the target wheel and the device frame, and for each of the calculation devices, calculating and transmitting the corresponding weights according to the forward / reverse operations corresponding to the current forward and reverse nodes using training data, thereby obtaining a training result.

[0075] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the weight pipeline parallel distributed training method of the neural network provided by the above methods, and the method includes: splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette wheel; marking the positions of the forward reverse nodes of the target roulette wheel with device frames to evenly distribute the positions of the forward reverse nodes to the computing devices; obtaining the current forward reverse node of each computing device according to the rotation state of the target roulette wheel and the device frame, and for each computing device, using training data to calculate and transmit corresponding weights according to the forward / reverse operations corresponding to the current forward reverse node, thereby obtaining a training result.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the weight pipeline parallel distributed training method of a neural network provided by the above-mentioned methods, the method comprising: splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette wheel; marking the positions of the forward reverse nodes of the target roulette wheel with device frames to evenly distribute the positions of the forward reverse nodes to the computing devices; obtaining the current forward reverse node of each of the computing devices according to the rotation state of the target roulette wheel and the device frame, and for each of the computing devices, using training data to calculate and transmit corresponding weights according to the forward / reverse operations corresponding to the current forward reverse node, thereby obtaining a training result.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weight pipeline parallel distributed training method for neural networks, It is characterized in that include: Splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette wheel; Marking the positions of the front reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the front reverse nodes to computing devices; The current forward / reverse node of each computing device is obtained according to the rotation state of the target wheel and the device frame. For each computing device, the training data is used to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node, thereby obtaining the training result.

2. The weighted pipeline parallel neural network distributed training method according to claim 1, It is characterized in that The forward / reverse node of the target roulette wheel also includes a weight mark required for the corresponding forward / reverse operation.

3. The weighted pipeline parallel neural network distributed training method according to claim 1, It is characterized in that For each of the computing devices, using the training data to calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node, specifically includes: Splitting the training data into a plurality of batches of sub-data according to the number of the computing devices, and sending the batches of sub-data to the computing devices; For each of the computing devices, the batch sub-data is used to perform calculations according to the forward / reverse operations corresponding to the current forward / reverse node and weight transmission is performed to the next device.

4. The weighted pipeline parallel distributed neural network training method according to claim 1, It is characterized in that The order of weight transmission between the computing devices is determined according to the rotation direction of the target wheel and the position of the device frame.

5. The weighted pipeline parallel distributed neural network training method according to claim 1 or 3, It is characterized in that The step of obtaining the current forward / reverse node of each computing device according to the rotation state of the target wheel and the device frame, and for each computing device, calculating and transmitting the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using the training data, thereby obtaining the training result, specifically includes: S1: Initializing the target roulette wheel at an initial moment, and obtaining the current forward reverse node of the computing device according to the forward reverse node included in the device frame of each computing device; S2: when the computing device is capable of computing, using the training data to compute the forward / reverse operation corresponding to the current forward / reverse node, and transmitting the corresponding weight; when the computing device is unable to compute, placing the computing device in an initial idle state, and transmitting the corresponding weight; S3: Rotate the target wheel, update the current forward and reverse nodes of the computing device, jump to step S2, until the training is completed, and output the training results.

6. The weighted pipeline parallel distributed neural network training method according to claim 5, It is characterized in that The step S2 further comprises: When the computing device completes all reverse operations of a computing phase, the weights updated in the computing phase are transmitted to the next device, so that the subsequent devices are updated based on the updated weights.

7. A weighted pipeline parallel neural network distributed training device, It is characterized in that include: A roulette unit, used for splitting the neural network into multiple calculation stages, representing the calculation stages as forward / reverse nodes according to forward / reverse operations and arranging them in a ring in sequence to obtain a target roulette; A device frame unit, used for marking the positions of the front reverse nodes of the target wheel with device frames, so as to evenly distribute the positions of the front reverse nodes to computing devices; The computing unit is used to obtain the current forward / reverse node of each computing device according to the rotation state of the target wheel and the device frame, and for each computing device, calculate and transmit the corresponding weight according to the forward / reverse operation corresponding to the current forward / reverse node using the training data, so as to obtain the training result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the weight pipeline parallel distributed training method of a neural network is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the weight pipeline parallel distributed training method of a neural network as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the weight pipeline parallel distributed training method of a neural network as claimed in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Data parallel training method, device and equipment for deep neural network model

    CN110378472A

  • Deep learning weight updating method and device

    CN111722937A

  • Distributed weight update for back propagation of neural networks

    CN114631102A

  • Neural network training method, system and device and storage medium

    CN115293289A

  • Model training method and device based on distributed parallel computing

    CN117436514A