Neural network training method, device and storage medium based on distributed communication
By adopting the synchronization mechanism of gradient sequence and importance index in neural network training, combined with parallel segmentation calculation and sparse gradient update, the bottleneck problem of inter-node communication in distributed training is solved, and the training speed and efficiency are improved.
Patent Information
- Application Number
- CN202110221266.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-02-27
AI Technical Summary
Traditional stand-alone neural network training takes a long time and lacks computing power. Although distributed training methods can improve computing power, due to high bandwidth consumption and long communication lag, communication between nodes has become a bottleneck in training speed.
By synchronizing based on gradient sequence and importance index between training nodes, the communication frequency and data volume are reduced, and parallel segmentation calculations and sparse gradient updates are adopted to ensure that each node obtains the same neural network parameters and reduces communication with non-important gradients.
It significantly improves the training speed of neural networks, reduces communication pressure and resource consumption, and improves the accuracy of training efficiency and parameter adjustment.
Smart Images

Figure CN112766502B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of machine learning, and in particular to a neural network training method, device and storage medium based on distributed communication. Background Art
[0002] With the development of information technology and the rise of artificial intelligence, neural networks are increasingly used in daily life. There are more and more types of neural networks and their complexity is getting higher and higher. Traditional single-machine training may require tens of thousands of iterations and several months to converge. The computing power of a single machine can no longer match the computing power requirements of neural network training. Although distributed training methods can improve computing power by distributing training tasks to multiple nodes in parallel, they require mutual communication between nodes to complete training. The large amount of communication data and high frequency of each node result in high bandwidth consumption and long communication delays, making the problem of communication between nodes a bottleneck for accelerating neural network training. Summary of the invention
[0003] In order to solve at least one of the above technical problems, the present disclosure proposes a neural network training solution based on distributed communication.
[0004] According to the first aspect of the present disclosure, a neural network training based on distributed communication is provided, which is applied to a training node, including: training the neural network corresponding to the training node, and saving the generated gradient in a first gradient sequence; obtaining a cumulative gradient sequence according to the first gradient sequence and the second gradient sequence; the second gradient sequence is used to record the gradient that has not yet participated in synchronization; calculating an importance index sequence according to the cumulative gradient sequence; obtaining an important gradient indication sequence, and determining the important gradient in the cumulative gradient sequence according to the important gradient indication sequence; obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence; synchronizing between training nodes based on the information to be synchronized, and obtaining a synchronized gradient sequence and a synchronized important gradient indication sequence; adjusting the parameters of the neural network according to the synchronized gradient sequence, and using the synchronized important gradient indication sequence as the new important gradient indication sequence. Based on the above configuration, the important gradient and the related information for determining the important gradient can be synchronized in one communication process, and the synchronization result can ensure that the important gradients obtained by each training node correspond to the same neural network parameters, without the need to perform additional node-to-node communication on the position of the important gradient, thereby reducing the frequency of node communication and improving the training speed.
[0005] In some possible implementations, the cumulative gradient sequence is obtained according to the first gradient sequence and the second gradient sequence, including: segmenting the first gradient sequence and the second gradient sequence respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein, if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameter; setting a plurality of parallel computing threads, each of which obtains at least one first gradient segment and a second gradient segment corresponding to the first gradient segment; each of which accumulates the first gradient segment and the corresponding second gradient segment for each first gradient segment obtained to obtain the corresponding cumulative gradient segment; and obtaining the cumulative gradient sequence according to the cumulative gradient segments obtained by each of the computing threads. Based on the above configuration, the cumulative gradient can be calculated by parallel segmentation to improve the calculation speed of the cumulative gradient sequence.
[0006] In some possible implementations, the importance index sequence is calculated based on the accumulated gradient sequence, including: each of the computing threads calculates the corresponding importance index based on the accumulated gradient segments obtained; and the importance index sequence is obtained based on the importance index calculation results of each of the computing threads. Based on the above configuration, the importance index obtained can characterize the importance of a gradient segment including multiple gradients, and then in the subsequent synchronization process, it can be determined whether the gradient segment is an important gradient segment, which is convenient for determining important gradients with gradient segments as the granularity, thereby completing the gradient update of the gradient segment granularity, realizing sparse gradient update, further reducing the amount of communication data, and improving the training speed.
[0007] In some possible implementations, determining the important gradients in the cumulative gradient sequence according to the important gradient indication sequence includes: for each cumulative gradient segment calculated by each computing thread, extracting the corresponding important gradient indication value in the important gradient indication sequence; if the important gradient indication value is a first indication value, then determining the cumulative gradients in the cumulative gradient segment as important gradients, and submitting the cumulative gradient segment to the communication buffer of the training node; the first indication value indicates that the cumulative gradients in the cumulative gradient segment are all important gradients. Based on the above configuration, important gradients and non-important gradients can be quickly determined with gradient segments as the granularity, thereby improving the speed of determining important gradients.
[0008] In some possible implementations, the information to be synchronized of the training node is obtained according to the important gradient and the importance index sequence, including: obtaining the gradient sequence to be synchronized according to the accumulated gradient segments in the communication buffer; wherein the position of each accumulated gradient in each accumulated gradient segment in the communication buffer in the gradient sequence to be synchronized is the same as the position of the accumulated gradient in the accumulated gradient sequence, and the other positions in the sequence to be synchronized are set to preset gradient values; splicing the gradient sequence to be synchronized and the importance index sequence to be synchronized to obtain the information to be synchronized. Based on the above configuration, the important gradient can be quickly determined with the gradient segment as the granularity to obtain the gradient sequence to be synchronized, and the gradient information to be synchronized corresponding to each parameter can be recorded in the gradient sequence to be synchronized. The splicing result of the gradient sequence to be synchronized and the importance index is used as the information to be synchronized, so that the information to be synchronized can include the gradient, the gradient position, and the importance index sequence for determining the important gradient position in the next iteration with a smaller amount of data, thereby reducing the amount of communication data and the communication frequency, which can significantly alleviate the communication pressure generated by neural network training in a distributed communication environment.
[0009] In some possible implementations, the synchronization between training nodes based on the information to be synchronized is performed to obtain a synchronized gradient sequence and a synchronized important gradient indication sequence, including: adding the synchronized gradient sequences in the information to be synchronized of each training node element by element to obtain a synchronized cumulative gradient sequence; dividing each synchronized cumulative gradient in the synchronized cumulative gradient sequence by the total number of training nodes to obtain the synchronized gradient sequence; adding the importance index sequences in the information to be synchronized of each training node element by element to obtain a cumulative importance index sequence; dividing each cumulative importance index in the cumulative importance index sequence by the total number of training nodes to obtain an average importance index sequence; calculating the important gradient indication value corresponding to each average importance index in the average importance index sequence to obtain a synchronized important gradient indication sequence. Based on the above configuration, a synchronized gradient sequence that can be accurately used to adjust the parameters of a neural network can be calculated, and a synchronized important gradient indication sequence that can be used to quickly determine important gradients can be obtained, thereby improving the training speed of the neural network.
[0010] In some possible implementations, the calculation of the important gradient indicator value corresponding to each average importance indicator in the average importance indicator sequence includes: obtaining the sorting results of the average importance indicators in the average importance indicator sequence in descending order of the average importance indicators; obtaining a threshold indicator, and determining the importance indicator threshold in the sorting result according to the threshold indicator; for each average importance indicator in the average importance indicator sequence, if the average importance indicator is greater than the importance indicator threshold, setting the corresponding important gradient indicator value to a first indicator value; otherwise, setting the corresponding important gradient indicator value to a second indicator value; wherein the first indicator value is used to indicate an important gradient, and the second indicator value is used to indicate a non-important gradient. Based on the above configuration, the important gradient indicator value corresponding to each average importance indicator can be accurately calculated to improve the accuracy of important gradient judgment.
[0011] In some possible implementations, the obtaining of the threshold indicator includes: determining the number of segments according to the preset segmentation rule; obtaining a preset compression rate; and taking the product of the compression rate and the number of segments as the threshold indicator. Based on the above configuration, a more reasonable threshold indicator can be obtained, thereby determining a reasonable importance indicator threshold, and ultimately improving the accuracy of important gradient judgment.
[0012] In some possible implementations, adjusting the parameters of the neural network according to the synchronized gradient sequence includes: extracting the gradients in the synchronized gradient sequence in sequence; if the gradient is not equal to a preset gradient value, adjusting the corresponding neural network parameters according to the gradient; if the gradient is equal to the preset gradient value, extracting the next gradient. Based on the above configuration, non-important gradients are indicated by setting a preset gradient value, and the parameter values corresponding to the non-important gradients do not need to be adjusted, thereby avoiding excessive adjustment of the parameters of the neural network and further improving the convergence speed of the neural network.
[0013] In some possible implementations, the method further includes: determining the non-significant gradients in the cumulative gradient sequence according to the important gradient indication sequence; updating the second gradient sequence according to the non-significant gradients; and iteratively performing neural network training based on distributed communication until a preset training stop condition is reached. Based on the above configuration, each training node continuously adjusts its own parameters through iteration, and performs inter-node communication and collaboration during each iteration to complete the training of the neural network, thereby improving training efficiency.
[0014] According to the second aspect of the present disclosure, a neural network training method based on distributed communication is provided, which is applied to a training system including multiple training nodes, and the method includes: each training node trains the corresponding neural network, and saves the generated gradient in a first gradient sequence; according to the first gradient sequence and the second gradient sequence, a cumulative gradient sequence is obtained; the second gradient sequence is used to record the gradients in the training node that have not yet participated in synchronization; according to the cumulative gradient sequence, an importance index sequence is calculated; the important gradient indication sequence in the training node is obtained, and the important gradient in the cumulative gradient sequence is determined according to the important gradient indication sequence; according to the important gradient and the importance index sequence, the information to be synchronized of the training node is obtained; each training node synchronizes between training nodes based on the corresponding information to be synchronized, and obtains a synchronized gradient sequence and a synchronized important gradient indication sequence; each training node adjusts the parameters of the neural network according to the synchronized gradient sequence, and uses the synchronized important gradient indication sequence as a new synchronized important gradient indication sequence. Based on the above configuration, each training node can obtain the same synchronized gradient sequence and synchronized important gradient indication sequence in one synchronization process, and the synchronized important gradient indication sequence is used to obtain a new synchronized gradient sequence in the next iterative synchronization. The synchronized gradient sequence includes important gradients and their location information, so there is no need to communicate between training nodes about the locations of important gradients. This reduces the communication frequency, ensures the consistency of the locations of important gradients in each training node, and improves the training speed.
[0015] According to a third aspect of the present disclosure, a neural network training device based on distributed communication is provided, the device is arranged at a training node, and includes: a training module, used to train the neural network corresponding to the training node, and save the generated gradient in a first gradient sequence; a cumulative gradient acquisition module, used to obtain a cumulative gradient sequence according to the first gradient sequence and the second gradient sequence; the second gradient sequence is used to record the gradients that have not yet participated in synchronization; an importance index sequence calculation module, used to calculate the importance index sequence according to the cumulative gradient sequence; a gradient classification module, used to obtain an important gradient indication sequence, and determine the important gradients in the cumulative gradient sequence according to the important gradient indication sequence; a synchronization module, used to obtain the information to be synchronized of the training node according to the important gradient and the importance index sequence, synchronize the training nodes based on the information to be synchronized, and obtain a synchronized gradient sequence and a synchronized important gradient indication sequence; an update module, used to use the synchronized important gradient indication sequence as the new important gradient indication sequence; a parameter adjustment module, used to adjust the parameters of the neural network according to the synchronized gradient sequence.
[0016] In some possible implementations, the cumulative gradient acquisition module includes: a segmentation unit, configured to segment the first gradient sequence and the second gradient sequence respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein, if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameter; a multi-thread processing unit, configured to set a plurality of parallel computing threads, each of the computing threads acquiring at least one first gradient segment and a second gradient segment corresponding to the first gradient segment; an accumulation unit, configured to accumulate the first gradient segment and the corresponding second gradient segment for each first gradient segment acquired by each computing thread to obtain a corresponding cumulative gradient segment; and a cumulative gradient sequence acquisition unit, configured to obtain the cumulative gradient sequence according to the cumulative gradient segments acquired by each of the computing threads.
[0017] In some possible implementations, the importance indicator sequence calculation module includes: an importance indicator calculation unit, which is used for each of the computing threads to calculate the corresponding importance indicator based on the obtained cumulative gradient segment; and an importance indicator sequence acquisition unit, which is used to obtain the importance indicator sequence based on the importance indicator calculation results of each of the computing threads.
[0018] In some possible implementations, the gradient classification module is used to extract the corresponding important gradient indication value in the important gradient indication sequence for each cumulative gradient segment calculated by each of the computing threads; if the important gradient indication value is a first indication value, the cumulative gradients in the cumulative gradient segment are all determined to be important gradients, and the cumulative gradient segment is submitted to the communication buffer of the training node; the first indication value indicates that the cumulative gradients in the cumulative gradient segment are all important gradients.
[0019] In some possible implementations, the synchronization module includes: a gradient sequence to be synchronized acquisition unit, used to obtain a gradient sequence to be synchronized according to the accumulated gradient segments in the communication buffer; wherein the position of each accumulated gradient in each accumulated gradient segment in the communication buffer in the gradient sequence to be synchronized is the same as the position of the accumulated gradient in the accumulated gradient sequence, and other positions in the sequence to be synchronized are set to preset gradient values; and a splicing unit, used to splice the gradient sequence to be synchronized and the importance indicator sequence to obtain the information to be synchronized.
[0020] In some possible implementations, the synchronization module further includes: a synchronized cumulative gradient sequence acquisition unit, which is used to add the to-be-synchronized gradient sequences in the to-be-synchronized information of each training node element by element to obtain a synchronized cumulative gradient sequence; a post-synchronization gradient sequence acquisition unit, which is used to divide each synchronized cumulative gradient in the synchronized cumulative gradient sequence by the total number of the training nodes to obtain the post-synchronization gradient sequence; a cumulative importance indicator sequence acquisition unit, which is used to add the importance indicator sequences in the to-be-synchronized information of each training node element by element to obtain a cumulative importance indicator sequence; an average importance indicator sequence acquisition unit, which is used to divide each cumulative importance indicator in the cumulative importance indicator sequence by the total number of the training nodes to obtain an average importance indicator sequence; and a post-synchronization important gradient indication sequence calculation unit, which is used to calculate the important gradient indication value corresponding to each average importance indicator in the average importance indicator sequence to obtain a post-synchronization important gradient indication sequence.
[0021] In some possible embodiments, the post-synchronization important gradient indication sequence calculation unit includes: a sorting unit, used to obtain the sorting results of each average importance indicator in the average importance indicator sequence according to the descending order of the average importance indicator; an indicator threshold acquisition unit, used to obtain a threshold indicator, and determine the importance indicator threshold in the sorting result according to the threshold indicator; an important gradient indication value calculation unit, used to, for each average importance indicator in the average importance indicator sequence, if the average importance indicator is greater than the importance indicator threshold, set the corresponding important gradient indication value to a first indication value; otherwise, set the corresponding important gradient indication value to a second indication value; wherein, the first indication value is used to indicate an important gradient, and the second indication value is used to indicate a non-important gradient.
[0022] In some possible implementations, the indicator threshold acquisition unit is used to determine the number of segments according to the preset segmentation rule; obtain a preset compression rate; and use the product of the compression rate and the number of segments as the threshold indicator.
[0023] In some possible implementations, the parameter adjustment module is used to sequentially extract the gradients in the synchronized gradient sequence; if the gradient is not equal to a preset gradient value, adjust the corresponding neural network parameter according to the gradient; if the gradient is equal to the preset gradient value, extract the next gradient.
[0024] In some possible implementations, the update module is further used to determine the non-important gradients in the accumulated gradient sequence according to the important gradient indication sequence; and update the second gradient sequence according to the non-important gradients; the device also includes an iteration control module, which is used to iteratively perform distributed communication-based neural network training until a preset training stop condition is reached.
[0025] According to a fourth aspect of the present disclosure, there is provided a training device, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the distributed communication-based neural network training method as described in any one of the first aspects by executing the instructions stored in the memory.
[0026] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the neural network training method based on distributed communication as described in any one of the first aspects.
[0027] According to a sixth aspect of the present disclosure, a training system is provided, the training system comprising the training device as described in the fourth aspect.
[0028] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
[0029] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions and advantages in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0031] Figure 1 A schematic diagram showing a neural network training based on distributed communication in a related technology according to an embodiment of the present disclosure;
[0032] Figure 2 A flowchart showing a neural network training method based on distributed communication according to an embodiment of the present disclosure is shown;
[0033] Figure 3 A schematic diagram showing a feasible structure of a deep neural network according to an embodiment of the present disclosure;
[0034] Figure 4 A schematic diagram showing the relationship between the first storage space, the second storage space and the neural network structure according to an embodiment of the present disclosure;
[0035] Figure 5A schematic diagram showing a flow chart of multiple nodes executing the above-mentioned neural network training method based on distributed communication according to an embodiment of the present disclosure;
[0036] Figure 6 A schematic flow chart of step S20 in the above-mentioned neural network training method based on distributed communication according to an embodiment of the present disclosure is shown;
[0037] Figure 7 A schematic flow chart of step S40 of the above-mentioned neural network training method based on distributed communication according to an embodiment of the present disclosure is shown;
[0038] Figure 8 A schematic flow chart of step S60 in the above-mentioned neural network training method based on distributed communication according to an embodiment of the present disclosure is shown;
[0039] Fig. 9 A block diagram of a neural network training device based on distributed communication according to an embodiment of the present disclosure is shown;
[0040] Fig.10 A block diagram of a training device according to an embodiment of the present disclosure is shown;
[0041] Fig.11 A block diagram of another training device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0044] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0045] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0046] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0047] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0048] In the actual application of related technologies, in order to improve the training speed of neural networks, the training tasks of neural networks can be distributed in parallel to multiple training nodes. Based on distributed communication, each training node can cooperate to train the neural network, thereby improving the training speed of the neural network and shortening the time it takes for the neural network to converge. Please refer to Figure 1 , which shows a schematic diagram of neural network training based on distributed communication in the related art. Figure 1 Each GPU (Graphics Processing Unit) is a training node. Figure 1 There are m training nodes in total, and the m training nodes can independently train neural networks with the same structure.
[0049] by Figure 1 Take the example of m training nodes in the network cooperating to train a neural network with n layers. During the training of the neural network at each training node, the sample data is input into the neural network, triggering the forward propagation process and the reverse propagation process in sequence. In the forward propagation process, the sample data is input into the first layer, triggering the first layer, the second layer, and finally the nth layer outputs the calculation results in sequence. In the reverse propagation process, the nth layer, the n-1th layer, and finally the first layer generate gradients in sequence.
[0050] The m training nodes communicate according to the generated gradients, and adjust the parameters of the corresponding neural network according to the communication results. Through distributed training, each training node can refer to the gradients generated by other training nodes to adjust the parameters of its corresponding neural network, thereby improving the training speed. However, the amount of data required for inter-node communication in the related technology is large and the communication frequency is also high, which makes the communication process itself consume more resources. Each training node may be forced to be in a waiting state due to frequent communication delays, thereby reducing the training speed. In this case, the communication process of distributed nodes becomes a bottleneck restricting the training efficiency.
[0051] In order to reduce the amount of communication data and the communication frequency in the distributed training process, alleviate the communication pressure, and increase the speed of distributed training of the neural network, the embodiments of the present disclosure provide a neural network training method based on distributed communication.
[0052] The neural network training method based on distributed communication provided in the embodiments of the present disclosure can be applied to any data processing device with a graphics processing unit (GPU), which can be a terminal, including a personal computer (PC), a minicomputer, a mid-size computer, a mainframe, a workstation, etc. Of course, the data processing device can also be a server. It should be noted that the data processing device can be independent or in a cluster when used to train a neural network.
[0053] The neural network training method based on distributed communication provided in the embodiment of the present disclosure can be stored in a data processing device in the form of a computer program, and the data processing device implements the neural network training method based on distributed communication in the embodiment of the present disclosure by running the computer program. The above-mentioned computer program can be an independent computer program, or a functional module, plug-in or applet integrated on other computer programs.
[0054] The following describes the neural network training method based on distributed communication according to an embodiment of the present disclosure, taking a data processing device as a training node as an example of an execution subject. Figure 2 A flowchart of a neural network training method based on distributed communication according to an embodiment of the present disclosure is shown. Figure 2 As shown, the above method includes:
[0055] S10: Train the neural network corresponding to the training node, and save the generated gradient in the first gradient sequence.
[0056] The structure of the neural network is not limited in the embodiments of the present disclosure. The neural network can be at least one of a deep neural network, a convolutional neural network, and a recurrent neural network. Figure 3 , which shows a feasible structural diagram of a deep neural network. The above-mentioned deep neural network may include a convolution layer, a pooling layer and a fully connected layer, one of the above-mentioned convolution layers is used as the input layer of the neural network, and the above-mentioned fully connected layer is used as the output layer of the above-mentioned deep neural network, and the convolution layer and the pooling layer may be arranged alternately between the above-mentioned input layer and the above-mentioned output layer.
[0057] In a feasible implementation, a first storage space (grad_buffer) and a second storage space (left_buffer) may be allocated in the training node. In one embodiment, the first storage space (grad_buffer) and the second storage space (left_buffer) are both continuous storage spaces.
[0058] Please refer to Figure 4 , which shows a schematic diagram of the relationship between the first storage space, the second storage space and the neural network structure. In the disclosed embodiment, the neural network may have multiple layers, each layer may include multiple parameters, and each parameter in each layer corresponds to a storage interval in the first storage space. The first storage space and the second storage space may have the same structure, and the first storage space and the second storage space may be the same as the storage space occupied by the parameters of the neural network. The first storage space (grad_buffer) can be used to store the gradients (first gradient sequence) generated by the training nodes during the training process, and the second storage space (left_buffer) can be used to store gradients that have not yet participated in synchronization.
[0059] Take the example of a neural network with three layers:
[0060] Layer 1 includes parameter E10, parameter E11, parameter E12 and parameter E13;
[0061] Layer 2 includes parameter E20, parameter E21, parameter E22;
[0062] Layer 3 includes parameter E30, parameter E31 and parameter E32;
[0063] Layers 1-3 include 10 parameters in total, so the first storage space and the second storage space each include 10 storage intervals. Each layer generates a gradient corresponding to each parameter in reverse during the training process. Specifically:
[0064] First, layer 3 generates a gradient T30, a gradient T31, and a parameter gradient T32 corresponding to parameter E30, parameter E31, and parameter E32, respectively;
[0065] Then, layer 2 generates gradient T20, gradient T21 and gradient T22 corresponding to parameter E20, parameter E21 and parameter E22 respectively;
[0066] Next, layer 1 generates a gradient T10 , a gradient T11 , a gradient T12 , and a gradient T13 corresponding to parameter E10 , parameter E11 , parameter E12 , and parameter E13 , respectively.
[0067] The 10 gradients can be stored in the first storage space in the order in which the gradients are generated, that is, the data in the first storage space (grad_buffer) can be {T30, T31, T32, T20, T21, T22, T10, T11, T12, T13}. The data in the first storage space (grad_buffer) {T30, T31, T32, T20, T21, T22, T10, T11, T12, T13} is the first gradient sequence.
[0068] In one embodiment, the training node can take out the minimum batch samples each time, perform minimum batch sample training, adjust the parameters of the corresponding neural network during the training process, and obtain the gradient corresponding to each parameter generated by the current batch training according to the training results, and store the gradient in the first gradient sequence. Through batch training, the first gradient sequence can be recorded after each batch training is completed, and subsequent synchronization between training nodes can be performed, while synchronization between training nodes may not be performed during each batch training process. Batch training can reduce the communication frequency and improve the training speed.
[0069] S20: Obtaining a cumulative gradient sequence according to the first gradient sequence and the second gradient sequence; the second gradient sequence is used to record gradients that have not yet participated in synchronization.
[0070] In the disclosed embodiment, the second gradient sequence can be stored in the second storage space (left_buffer). The second storage space has the same structure as the first storage space, and the storage intervals at the same position correspond to the same neural network parameters. Taking the above-mentioned neural network with three layers as an example, the 10 consecutive storage intervals in the first storage space and the 10 consecutive storage intervals in the second storage space correspond to parameters E30, E31, E32, E20, E21, E22, E10, E11 and E12 respectively. The data in the storage intervals at the same position in the first storage space and the second storage space can be added to obtain a cumulative gradient sequence, and the above-mentioned cumulative gradient sequence can be stored in the above-mentioned first storage space.
[0071] Exemplarily, taking the above-mentioned neural network with three layers as an example, any storage interval grad_buffer[i] of the first storage space and the corresponding storage interval left_buffer[i] of the second storage space can be obtained, where i takes a value of 0-9, and grad_buffer[i]+left_buffer[i] is assigned to grad_buffer[i]. The data in the first storage space after the assignment is the above-mentioned cumulative gradient sequence, that is, the above-mentioned cumulative gradient sequence and the above-mentioned first gradient sequence reuse the above-mentioned first storage space. In the actual training scenario of the neural network based on distributed communication, the number of training nodes participating in the collaborative training of the neural network is large, and because the complexity of the neural network can be very high, correspondingly, the storage space occupied by the generated gradient is often large. In each training node, the first storage space can be reused, which can greatly save storage resources.
[0072] S30: Calculate the importance index sequence according to the above cumulative gradient sequence.
[0073] Specifically, the importance index can be calculated corresponding to each cumulative gradient in the cumulative gradient sequence, and the embodiment of the present disclosure does not limit the determination method of the above importance index. Taking the above neural network with three layers as an example, an importance index sequence Im can be obtained, in which each importance index Im[i] can represent the importance of the gradient generated by the neural network parameter corresponding to grad_buffer[i] obtained above.
[0074] S40: Obtain an important gradient indication sequence, and determine important gradients in the accumulated gradient sequence according to the important gradient indication sequence.
[0075] In the embodiment of the present disclosure, each important gradient indication value in the important gradient indication sequence can indicate whether the accumulated gradient of the corresponding position is important. Taking the above-mentioned neural network with three layers as an example, an important gradient indication sequence Ip can be obtained, and each important gradient indication value Ip[i] can indicate whether the data recorded in grad_buffer[i] obtained above is an important gradient. Exemplarily, if Ip[i] is a first indication value, the data recorded in grad_buffer[i] is determined to be an important gradient, and if Ip[i] is a second indication value, the data recorded in grad_buffer[i] is determined to be a non-important gradient. In an exemplary embodiment, the important gradient indication sequence Ip can be obtained during the last inter-node synchronization process.
[0076] For example, if the values of Ip[1], Ip[3], and Ip[5] in the important gradient indication sequence Ip represent important gradients, then grad_buffer[1], grad_buffer[3], and grad_buffer[5] are important gradients, and the other gradients in grad_buffer are unimportant gradients.
[0077] In an exemplary embodiment, after determining the insignificant gradients in the accumulated gradient sequence, the second gradient sequence may be updated according to the insignificant gradients.
[0078] In the disclosed embodiment, the non-important gradient does not participate in the synchronization between the nodes this time. Therefore, the non-important gradient can be updated in the second gradient sequence, that is, stored in the second storage space (left_buffer). Updating the second gradient sequence according to the non-important gradient includes determining the position of the non-important gradient in the second gradient sequence, updating the data corresponding to the position in the second gradient sequence to the non-important gradient, and assigning 0 to the data corresponding to other positions.
[0079] Taking the above neural network with three layers as an example, grad_buffer[0], grad_buffer[2], grad_buffer[4], grad_buffer[6], grad_buffer[7], grad_buffer[8], and grad_buffer[9] are all non-important gradients. Then grad_buffer[0], grad_buffer[2], grad_buffer[4], grad_buffer[6], grad_buffer[7], grad_buffer[8], and grad_buffer[9] are assigned to left_buffer[0], left_buffer[2], left_buffer[4], left_buffer[6], left_buffer[7], left_buffer[8], and left_buffer[9] respectively, while left_buffer[1], left_buffer[3], and left_buffer[5] can all be assigned to 0.
[0080] S50: Obtain the information to be synchronized of the training node according to the important gradient and the importance index sequence.
[0081] Specifically, the important gradient and the importance index sequence may be spliced, and the splicing result may be used as the information to be synchronized. The embodiment of the present disclosure does not limit the splicing method.
[0082] S60: Perform synchronization between training nodes based on the information to be synchronized, and obtain a synchronized gradient sequence and a synchronized important gradient indication sequence.
[0083] Exemplarily, AllReduce may be used for communication, which is a general term for a class of communication methods that can efficiently communicate between distributed training nodes.
[0084] In the embodiment of the present disclosure, each training node can obtain a post-synchronization important gradient indication sequence during each synchronization, and the post-synchronization important gradient indication sequence is used as the important gradient indication sequence to be applied during the next training. Therefore, the important gradient indication sequence used in the training process of each training node is the same, that is, the important gradient positions are the same.
[0085] Taking training node A as an example, if training node A determines in step S40 that grad_buffer[1], grad_buffer[3], and grad_buffer[5] are important gradients, then other training nodes will also determine that their corresponding grad_buffer[1], grad_buffer[3], and grad_buffer[5] are important gradients. The synchronized gradient sequence can be directly calculated based on the important gradients of each node without having to consider the location of the synchronized important gradients. There is no need for additional communication between training nodes about the location information of the important gradients, thereby reducing the communication frequency.
[0086] For example, for the synchronized gradient sequence Td, the average of the important gradients grad_buffer[1], grad_buffer[3], and grad_buffer[5] in each node can be taken to obtain Td[1], Td[3], and Td[5] respectively, and the values at other positions in Td can be assigned to preset gradient values.
[0087] In one embodiment, the importance index sequence after synchronization may be obtained by taking the average of each element in the importance index sequence of each node, and the above-mentioned importance gradient indication sequence after synchronization may be obtained according to the importance index sequence after synchronization.
[0088] S70: adjusting the parameters of the neural network according to the synchronized gradient sequence, and using the synchronized important gradient indication sequence as a new important gradient indication sequence.
[0089] The step of adjusting the parameters of the neural network according to the synchronized gradient sequence includes:
[0090] S71: extracting gradients in the synchronized gradient sequence in sequence.
[0091] S72: If the above gradient is not equal to the above preset gradient value, adjust the corresponding neural network parameters according to the above gradient.
[0092] S73: If the above gradient is equal to the above preset gradient value, extract the next gradient.
[0093] In a specific implementation, the preset gradient value is used to indicate a situation where parameter adjustment is not required according to the gradient. In a feasible implementation, the preset gradient value may be 0.
[0094] In the disclosed embodiment, the elements in the synchronized gradient sequence can be stored in the above-mentioned first storage space, and the first storage space (grad_buffer) can be reused to reduce storage consumption. Taking the three-layer neural network described above as an example, grad_buffer[i] is read sequentially. If grad_buffer[i] is not the above-mentioned preset gradient value, the parameters of the neural network corresponding to the above-mentioned grad_buffer[i] are adjusted; if grad_buffer[i] is the above-mentioned preset gradient value, i is incremented by 1, and grad_buffer[i] is continued to be read. Obviously, if the currently extracted gradient is the last gradient in the above-mentioned synchronized gradient sequence, "extracting the next gradient" will fail. This situation indicates that the adjustment of the parameters of this neural network has been completed. After completion, the first storage space can be cleared so that when iteratively executing steps S10-S70, the first gradient sequence can be reused in step S10 to record it.
[0095] In the disclosed embodiment, non-important gradients are indicated by setting preset gradient values, and parameter values corresponding to non-important gradients do not need to be adjusted, thereby avoiding excessive adjustment of the parameters of the neural network and further improving the convergence speed of the neural network.
[0096] Steps S10-S70 of the disclosed embodiment show a neural network training method for a single synchronization situation. In the above-mentioned neural network training method for a single synchronization situation, both the second gradient sequence and the important gradient indication sequence can be updated. According to the update result, the above-mentioned neural network training method for a single synchronization situation can be executed again, that is, steps S10-S70 can be iteratively executed until a preset training stop condition is reached, and a trained neural network can be obtained. The disclosed embodiment does not limit the above-mentioned training stop condition. The above-mentioned training stop condition can be that the number of iterations reaches a preset iteration threshold, or that the loss generated by the above-mentioned neural network is less than a preset loss threshold.
[0097] Please refer to Figure 5 , which shows a schematic diagram of the process of multiple nodes executing the above-mentioned neural network training method based on distributed communication. Figure 5 Each training node can independently train its own neural network, update the local second gradient sequence during the training process, and obtain the information to be synchronized, which includes important gradients and importance index sequences. Each training node communicates based on the information to be synchronized to obtain a synchronized gradient sequence and a synchronized important gradient indication sequence. Each node adjusts parameters based on the synchronized gradient sequence, and determines the important gradients in the next iteration based on the synchronized important gradient indication sequence. Each training node continuously adjusts its own parameters through iteration, and communicates and collaborates between nodes during each iteration to complete the training of the neural network.
[0098] In the actual scenario of distributed training of neural networks, the number of training nodes participating in the training is large, and because the complexity of the neural network can be very high, the amount of gradient data generated is often large. Correspondingly, the amount of gradient position information data used to record the gradient position is also large. The training nodes with large amounts of data need to synchronize not only the gradient position information but also the gradient, which generates a large communication pressure and consumes more communication resources. Limited by the communication resources, more nodes may be forced to be in a state of waiting for synchronization, which reduces the training speed. Based on the above configuration, the neural network training method based on distributed communication provided by the embodiment of the present disclosure can synchronize important gradients and important gradient position information in a communication process. The synchronization result can ensure that the important gradients obtained by each training node correspond to the same neural network parameters, without the need to communicate between the training nodes on the gradient position information. This can significantly reduce the communication frequency, reduce the consumption of communication resources, reduce the time spent by the training nodes waiting for synchronization, and improve the training speed. In the scenario where there are large amounts of training nodes and the neural network has high complexity, a particularly obvious speed-up effect can be achieved.
[0099] In some feasible implementations, the speed at which the training nodes execute the above steps can be further improved based on the multi-threaded concurrency characteristics. The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0100] Please refer to Figure 6 , which shows a flow chart of step S20 in the above-mentioned distributed communication-based neural network training method, and the flow chart of obtaining the cumulative gradient sequence according to the above-mentioned first gradient sequence and the second gradient sequence includes:
[0101] S21: segmenting the first gradient sequence and the second gradient sequence respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein, if a position of the first gradient segment in the first gradient segment sequence is the same as a position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameters.
[0102] The disclosed embodiment does not limit the specific segmentation rule, as long as the first gradient sequence can be divided into a plurality of first gradient segments to form a first gradient segment sequence. The first gradient sequence and the second gradient sequence have the same structure, and the second gradient sequence can be further divided based on the segmentation rule to obtain a second gradient segment sequence, and if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameters.
[0103] Exemplarily, taking the above-mentioned neural network with three layers as an example, the first gradient sequence obtained in step S10 is stored in the first storage space (grad_buffer), and a first gradient segment Tdf[0] can be formed by the data in grad_buffer[0], grad_buffer[1], grad_buffer[2], grad_buffer[3], and grad_buffer[4], and another first gradient segment Tdf[1] can be formed by the data in grad_buffer[5], grad_buffer[6], grad_buffer[7], grad_buffer[8], and grad_buffer[9], thereby obtaining a first gradient segment sequence {Tdf[0], Tdf[1]}. Correspondingly, the second gradient sequence is stored in the second storage space (left_buffer), and the data in left_buffer[0], left_buffer[1], left_buffer[2], left_buffer[3], and left_buffer[4] form a second gradient segment Tds[0], and the data in left_buffer[5], left_buffer[6], left_buffer[7], left_buffer[8], and left_buffer[9] form another first gradient segment Tds[1], resulting in the second gradient segment sequence {Tds[0], Tds[1]}.
[0104] Taking Tdf[0] as an example, each data in Tdf[0] corresponds to parameter E30, parameter E31, parameter E32, parameter E20, and parameter E21 in sequence. Tdf[0] corresponds to Tds[0], and each data in Tds[0] also corresponds to parameter E30, parameter E31, parameter E32, parameter E20, and parameter E21 in sequence.
[0105] S22: multiple parallel computing threads are set, each of which obtains at least one first gradient segment and a second gradient segment corresponding to the first gradient segment.
[0106] Taking setting two parallel computing threads as an example, Tdf[0] and Tds[0] can be sent to computing thread A, and Tdf[1] and Tds[1] can be sent to computing thread B.
[0107] S23: Each of the above-mentioned computing threads accumulates each acquired first gradient segment with the corresponding second gradient segment to obtain a corresponding accumulated gradient segment.
[0108] Taking computing thread A as an example, the data in Tdf[0] and Tds[0] can be added element by element to obtain the corresponding cumulative gradient segment. For example, the data of each element in the first gradient segment Tdf[0] is stored in grad_buffer[0], grad_buffer[1], grad_buffer[2], grad_buffer[3], grad_buffer[4] in sequence, and the data of each element in the second gradient segment Tds[0] is stored in left_buffer[0], left_buffer[1], left_buffer[2], left_buffer[3], left_buffer[4] in sequence, then the corresponding cumulative gradient segment The data in STd[0] are grad_buffer[0]+left_buffer[0], grad_buffer[1]+left_buffer[1], grad_buffer[2]+left_buffer[2], grad_buffer[3]+left_buffer[3], grad_buffer[4]+left_buffer[4], and it is clear that the five accumulated gradients in STd[0] correspond to parameters E30, E31, E32, E20, and E21. Similarly, computing thread B can obtain STd[1] according to Tdf[1] and Tds[1]. The five accumulated gradients in STd[1] correspond to parameters E22, E10, E11, E12, and E13.
[0109] S24: Obtain a cumulative gradient sequence according to the cumulative gradient segments obtained by the above-mentioned respective computing threads.
[0110] Exemplarily, computing thread A can obtain STd[0], computing thread B can obtain STd[1], and the accumulated gradient in STd[0] and the accumulated gradient in STd[1] are arranged in sequence to obtain the above-mentioned accumulated gradient sequence. Taking the neural network with three layers described above as an example, grad_buffer[0], grad_buffer[1], grad_buffer[2], grad_buffer[3], and grad_buffer[4] can be updated respectively according to the five elements in STd[0], and grad_buffer[5], grad_buffer[6], grad_buffer[7], grad_buffer[8], and grad_buffer[9] can be updated respectively according to the five elements in STd[1]. Then, the data in the first storage space forms the above-mentioned accumulated gradient sequence. The above-mentioned accumulated gradient sequence reduces storage consumption by reusing the first storage space. Based on the above configuration, the accumulated gradient can be calculated in parallel segments to improve the calculation speed of the accumulated gradient sequence.
[0111] Correspondingly, in step S30, the importance index sequence can also be calculated with the gradient segment as the granularity, that is, for each of the above-mentioned cumulative gradient segments, the corresponding importance index is calculated; and the importance index sequence is obtained according to the importance index calculation results of each of the above-mentioned calculation threads. Exemplarily, for the cumulative gradient segment STd[0], the corresponding importance index Im[0] can be obtained; for the cumulative gradient segment STd[1], the corresponding importance index Im[1] can be obtained. Based on the above configuration, the obtained importance index can characterize the importance of the gradient segment including multiple gradients, and then in the subsequent synchronization process, it can be determined whether the gradient segment is an important gradient segment, which is convenient for determining the important gradient with the gradient segment as the granularity, thereby completing the gradient update of the gradient segment granularity, realizing sparse gradient update, further reducing the amount of communication data, and improving the training speed.
[0112] In an exemplary embodiment, taking the neural network with three layers described above as an example, two cumulative gradient segments STd[0] and STd[1] can be obtained, and correspondingly, the above-mentioned importance index sequence includes two importance indexes Im[0] and Im[1]. For each of the above-mentioned cumulative gradient segments, the statistical value of each cumulative gradient in the above-mentioned cumulative gradient segment can be calculated, and the above-mentioned statistical value is used as the above-mentioned importance index. The type of the above-mentioned statistical value is not limited in the embodiments of the present disclosure. For example, the above-mentioned statistical value can be a variance, a standard deviation or a bi-norm.
[0113] Please refer to Figure 7, which shows a flow chart of step S40 of the above-mentioned neural network training method based on distributed communication, and the flow chart of determining the important gradients in the above-mentioned accumulated gradient sequence according to the above-mentioned important gradient indication sequence, includes:
[0114] S41: For each accumulated gradient segment calculated by each of the computing threads, extract a corresponding important gradient indication value from the important gradient indication sequence.
[0115] In an exemplary embodiment, each important gradient indicator value in the above-mentioned important gradient indicator sequence also corresponds to a cumulative gradient segment in the cumulative gradient sequence. Taking the neural network with three layers described above as an example, two cumulative gradient segments STd[0] and STd[1] can be obtained, and accordingly, the above-mentioned important gradient indicator sequence also includes two important gradient indicator values Ip[0] and Ip[1].
[0116] S42: If the above-mentioned important gradient indicator value is the first indicator value, the accumulated gradients in the above-mentioned accumulated gradient segment are all determined as important gradients, and the above-mentioned accumulated gradient segment is submitted to the communication buffer of the above-mentioned training node; the above-mentioned first indicator value indicates that the accumulated gradients in the above-mentioned accumulated gradient segment are all important gradients.
[0117] S43: If the important gradient indicator value is the second indicator value, all the accumulated gradients in the accumulated gradient segment are determined to be unimportant gradients; the second indicator value indicates that all the accumulated gradients in the accumulated gradient segment are unimportant gradients.
[0118] Exemplarily, 1 can be used as the first indicator value and 0 can be used as the second indicator value. Exemplarily, if the important gradient indicator value Ip[0] is 1, it is determined that each accumulated gradient in the corresponding accumulated gradient segment STd[0] is an important gradient, and the accumulated gradient segment is submitted to the communication buffer of the training node. If the important gradient indicator value Ip[1] is 0, the five accumulated gradients in the corresponding accumulated gradient segment STd[1] are all regarded as non-important gradients.
[0119] In a feasible embodiment, the second gradient sequence may be updated according to the determined non-significant gradients. This step has been described above and will not be repeated here.
[0120] Based on the above configuration, important gradients and unimportant gradients can be quickly determined with gradient segments as the granularity, thereby improving the speed of determining important gradients.
[0121] In an exemplary embodiment, obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence includes:
[0122] S51. Obtain a gradient sequence to be synchronized according to the accumulated gradient segments in the communication buffer; wherein the position of each accumulated gradient in each accumulated gradient segment in the communication buffer in the gradient sequence to be synchronized is the same as the position of the accumulated gradient in the accumulated gradient sequence, and other positions in the sequence to be synchronized are set to preset gradient values.
[0123] Continuing with the above example, only the accumulated gradient segment STd[0] is submitted to the communication buffer, and the corresponding positions of the accumulated gradients in the accumulated gradient segment STd[0] in the accumulated gradient sequence are from the 1st to the 5th, then the 1st to the 5th in the gradient sequence to be synchronized are the five accumulated gradients in the accumulated gradient sequence STd[0], and the other positions in the gradient sequence to be synchronized are set to preset values. Exemplarily, the other positions in the gradient sequence to be synchronized can be set to zero. In the example, the gradient sequence to be synchronized includes 10 values, which correspond to parameters E30, E31, E32, E20, E21, E22, E10, E11 and E12, respectively.
[0124] S52. Concatenate the above-mentioned gradient sequence to be synchronized and the above-mentioned importance index sequence to obtain the above-mentioned information to be synchronized.
[0125] Exemplarily, the importance indicator sequence {Im[0], Im[1]} may be appended to the gradient sequence to be synchronized to obtain the information to be synchronized.
[0126] Based on the above configuration, important gradients can be quickly determined with gradient segments as the granularity to obtain a gradient sequence to be synchronized. The gradient sequence to be synchronized can record the gradient information to be synchronized corresponding to each parameter. The concatenation result of the gradient sequence to be synchronized and the importance index is used as the information to be synchronized. This allows the information to be synchronized to include the gradient, the gradient position, and the importance index sequence for determining the important gradient position in the next iteration with a smaller amount of data, thereby reducing the amount of communication data and the communication frequency, and can significantly alleviate the communication pressure generated by neural network training in a distributed communication environment.
[0127] Please refer to Figure 8 , which shows a flow chart of step S60 in the above-mentioned neural network training method based on distributed communication, and the above-mentioned flow chart of synchronizing the training nodes based on the above-mentioned information to be synchronized to obtain the synchronized gradient sequence and the synchronized important gradient indication sequence, includes:
[0128] S61. Add the gradient sequences to be synchronized in the information to be synchronized of each of the training nodes element by element to obtain a synchronized accumulated gradient sequence.
[0129] S62. Divide each synchronized accumulated gradient in the synchronized accumulated gradient sequence by the total number of the training nodes to obtain the synchronized gradient sequence.
[0130] Assume that there are three training nodes to train the above neural network with three layers, where the gradient sequence to be synchronized of training node 1 is Wt1, the gradient sequence to be synchronized of training node 2 is Wt2, and the gradient sequence to be synchronized of training node 3 is Wt3. For any sequence position i, the value of Wt1[i]+Wt2[i]+Wt3[i] is the synchronized accumulated gradient corresponding to the sequence position i in the above synchronized accumulated gradient sequence. The synchronized accumulated gradient corresponding to each sequence position in the above synchronized accumulated gradient sequence is divided by 3 to obtain the above synchronized gradient sequence.
[0131] S63. Add the importance index sequences in the information to be synchronized of each of the above training nodes element by element to obtain a cumulative importance index sequence.
[0132] S64. Divide each accumulated importance index in the accumulated importance index sequence by the total number of training nodes to obtain an average importance index sequence.
[0133] Referring to the previous example, for a neural network with three layers, the importance index sequence corresponding to each training node is {Im[0], Im[1]}. In order to distinguish training node 1, training node 2, and training node 3, the importance index sequences of training node 1, training node 2, and training node 3 are represented as {Im1[0], Im1[1]}, {Im2[0], Im2[1]}, and {Im3[0], Im3[1]}, respectively. If {AIm[0], AIm[1]} is used to represent the above average importance index sequence, then AIm[0] = (Im1[0] + Im2[0] + Im3[0]) / 3, AIm[1] = (Im1[1] + Im2[1] + Im3[1]) / 3.
[0134] S65. Calculate the important gradient indicator value corresponding to each average importance indicator in the above average importance indicator sequence to obtain the synchronized important gradient indicator sequence.
[0135] Based on the above configuration, a synchronized gradient sequence that can be accurately used to adjust the neural network parameters can be calculated, as well as a synchronized important gradient indication sequence that can be used to quickly determine important gradients, thereby improving the neural network training speed.
[0136] In a feasible implementation, the above-mentioned calculation of the importance gradient indicator value corresponding to each average importance indicator in the above-mentioned average importance indicator sequence includes:
[0137] S651. Obtain the ranking results of each average importance index in the above average importance index sequence according to the descending order of the average importance index.
[0138] Referring to the previous example, for the average importance index sequence {AIm[0], AIm[1]}, if AIm[0] is less than AIm[1], the above sorting result is {AIm[1], AIm[0]}, otherwise the above sorting result is {AIm[0], AIm[1]}.
[0139] S652. Obtain a threshold indicator, and determine an importance indicator threshold in the above ranking results according to the threshold indicator.
[0140] In a feasible implementation, the value at the position pointed to by the threshold indicator in the sorting result can be used as the importance indicator threshold. For example, if the sorting result includes 30 average importance indicators and the threshold indicator is 10, the value corresponding to the 10th average importance indicator in the sorting result can be used as the importance indicator threshold.
[0141] In a feasible implementation, the number of segments can be determined according to the above-mentioned preset segmentation rules; a preset compression rate can be obtained; and the product of the above-mentioned compression rate and the above-mentioned number of segments can be used as the above-mentioned threshold indicator. The embodiments of the present disclosure do not limit the specific data of the above-mentioned preset compression rate and the above-mentioned number of segments. Referring to the above-mentioned example, the number of segments is only two, and in actual applications, there can be far more than two segments, and the examples in the embodiments of the present disclosure do not produce specific limitations.
[0142] Based on the above configuration, a more reasonable threshold index can be obtained, thereby determining a reasonable importance index threshold, and ultimately improving the accuracy of important gradient judgment.
[0143] S653. For each average importance indicator in the above average importance indicator sequence, if the above average importance indicator is greater than the above importance indicator threshold, the corresponding importance gradient indicator value is set to the first indicator value; otherwise, the corresponding importance gradient indicator value is set to the second indicator value.
[0144] In the disclosed embodiment, the first indicator value can be used to indicate an important gradient. For example, the first indicator value can be 1. The second indicator value can be used to indicate an unimportant gradient. For example, the second indicator value can be 0. Taking the average importance indicator sequence {AIm[0], AIm[1]} as an example, the synchronized important gradient indication sequence {TIp[0], TIp[1]} can be obtained accordingly. When iteratively executing the above-mentioned neural network training method based on distributed communication, the synchronized important gradient indication sequence {TIp[0], TIp[1]} can be used as a new important gradient indication sequence {Ip[0], Ip[1]}, which is used to determine the important gradient in the next iteration. How to determine the important gradient based on the important gradient indication sequence {Ip[0], Ip[1]} has been described in the previous text and will not be repeated here.
[0145] Based on the above configuration, the important gradient indication value corresponding to each average importance index can be accurately calculated, thereby improving the accuracy of important gradient judgment.
[0146] Those skilled in the art will appreciate that, in the above method of the specific implementation, the order in which the steps are written does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0147] In a feasible embodiment, the present disclosure also provides another neural network training method based on distributed communication, which is applied to a training system including multiple training nodes. The method includes:
[0148] S100. Each training node trains the corresponding neural network and saves the generated gradient in the first gradient sequence.
[0149] S200. Obtain a cumulative gradient sequence according to the first gradient sequence and the second gradient sequence; the second gradient sequence is used to record the gradients in the training node that have not yet participated in synchronization.
[0150] S300. Calculate an importance index sequence according to the accumulated gradient sequence.
[0151] S400. Obtain an important gradient indication sequence in the training node, and determine an important gradient in the accumulated gradient sequence according to the important gradient indication sequence.
[0152] S500. Obtain the information to be synchronized of the training node according to the important gradient and the importance index sequence.
[0153] S600. Each training node performs synchronization between training nodes based on the corresponding information to be synchronized, and obtains a synchronized gradient sequence and a synchronized important gradient indication sequence.
[0154] S700. Each training node adjusts the parameters of the neural network according to the synchronized gradient sequence, and uses the synchronized important gradient indication sequence as a new important gradient indication sequence.
[0155] The steps performed by each training node in the training system described in the embodiments of the present disclosure can be referred to in the previous text and will not be repeated here. It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them.
[0156] Fig. 9 A block diagram of a neural network training device based on distributed communication according to an embodiment of the present disclosure is shown. The device is arranged at a training node and includes:
[0157] A training module 10 is used to train the neural network corresponding to the training node and save the generated gradient in the first gradient sequence;
[0158] The cumulative gradient acquisition module 20 is used to obtain a cumulative gradient sequence according to the first gradient sequence and the second gradient sequence; the second gradient sequence is used to record the gradients that have not yet participated in synchronization;
[0159] The importance index sequence calculation module 30 is used to calculate the importance index sequence according to the above-mentioned cumulative gradient sequence;
[0160] A gradient classification module 40 is used to obtain an important gradient indication sequence and determine the important gradients in the accumulated gradient sequence according to the important gradient indication sequence;
[0161] A synchronization module 50, configured to obtain the information to be synchronized of the training nodes according to the important gradients and the importance index sequence, perform synchronization between the training nodes based on the information to be synchronized, and obtain a synchronized gradient sequence and a synchronized important gradient indication sequence;
[0162] An updating module 60, configured to use the synchronized important gradient indication sequence as a new important gradient indication sequence;
[0163] The parameter adjustment module 70 is used to adjust the parameters of the neural network according to the synchronized gradient sequence.
[0164] In some possible implementations, the cumulative gradient acquisition module includes:
[0165] a segmentation unit, configured to segment the first gradient sequence and the second gradient sequence respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein if a position of the first gradient segment in the first gradient segment sequence is the same as a position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameter;
[0166] A multi-thread processing unit, used to set a plurality of parallel computing threads, each of which acquires at least one first gradient segment and a second gradient segment corresponding to the first gradient segment;
[0167] An accumulation unit, configured for each of the computing threads to accumulate each first gradient segment obtained and the corresponding second gradient segment to obtain a corresponding accumulated gradient segment;
[0168] The accumulated gradient sequence acquisition unit is used to obtain the accumulated gradient sequence according to the accumulated gradient segments obtained by the above-mentioned respective computing threads.
[0169] In some possible implementations, the importance indicator sequence calculation module includes: an importance indicator calculation unit, which is used for each of the above-mentioned computing threads to calculate the corresponding importance indicator according to the above-mentioned cumulative gradient segment obtained; an importance indicator sequence acquisition unit, which is used to obtain the importance indicator sequence according to the importance indicator calculation results of each of the above-mentioned computing threads.
[0170] In some possible implementations, the gradient classification module is used to extract the corresponding important gradient indication value in the important gradient indication sequence for each cumulative gradient segment calculated by each of the computing threads; if the important gradient indication value is the first indication value, the cumulative gradients in the cumulative gradient segment are all determined to be important gradients, and the cumulative gradient segment is submitted to the communication buffer of the training node; the first indication value indicates that the cumulative gradients in the cumulative gradient segment are all important gradients.
[0171] In some possible implementations, the synchronization module includes:
[0172] a gradient sequence to be synchronized acquiring unit, configured to obtain a gradient sequence to be synchronized according to the accumulated gradient segments in the communication buffer; wherein the position of each accumulated gradient in each accumulated gradient segment in the communication buffer in the gradient sequence to be synchronized is the same as the position of the accumulated gradient in the accumulated gradient sequence, and other positions in the sequence to be synchronized are set to preset gradient values;
[0173] The splicing unit is used to splice the above-mentioned gradient sequence to be synchronized and the above-mentioned importance index sequence to obtain the above-mentioned information to be synchronized.
[0174] In some possible implementations, the synchronization module further includes: a synchronous cumulative gradient sequence acquisition unit, configured to add the gradient sequences to be synchronized in the information to be synchronized of each of the training nodes element by element to obtain a synchronous cumulative gradient sequence;
[0175] A synchronized gradient sequence acquisition unit, configured to divide each synchronized accumulated gradient in the synchronized accumulated gradient sequence by the total number of the training nodes to obtain the synchronized gradient sequence;
[0176] A cumulative importance index sequence acquisition unit is used to add the importance index sequences in the information to be synchronized of each of the above training nodes element by element to obtain a cumulative importance index sequence;
[0177] An average importance index sequence acquisition unit is used to divide each accumulated importance index in the accumulated importance index sequence by the total number of the training nodes to obtain an average importance index sequence;
[0178] The post-synchronization important gradient indication sequence calculation unit is used to calculate the important gradient indication value corresponding to each average importance indicator in the above average importance indicator sequence to obtain the post-synchronization important gradient indication sequence.
[0179] In some possible embodiments, the above-mentioned post-synchronization important gradient indication sequence calculation unit includes: a sorting unit, used to obtain the sorting results of each average importance indicator in the above-mentioned average importance indicator sequence according to the descending order of the average importance indicator; an indicator threshold acquisition unit, used to obtain the threshold indicator, and determine the importance indicator threshold in the above-mentioned sorting result according to the above-mentioned threshold indicator; an important gradient indication value calculation unit, used to, for each average importance indicator in the above-mentioned average importance indicator sequence, if the above-mentioned average importance indicator is greater than the above-mentioned importance indicator threshold, set the corresponding important gradient indication value to the first indication value; otherwise, set the corresponding important gradient indication value to the second indication value; wherein the above-mentioned first indication value is used to indicate important gradients, and the above-mentioned second indication value is used to indicate non-important gradients.
[0180] In some possible implementations, the indicator threshold acquisition unit is used to determine the number of segments according to the preset segmentation rule; obtain a preset compression rate; and use the product of the compression rate and the number of segments as the threshold indicator.
[0181] In some possible implementations, the parameter adjustment module is used to sequentially extract the gradients in the synchronized gradient sequence; if the gradient is not equal to the preset gradient value, adjust the corresponding neural network parameters according to the gradient; if the gradient is equal to the preset gradient value, extract the next gradient.
[0182] In some possible implementations, the update module is further used to determine the non-important gradients in the cumulative gradient sequence according to the important gradient indication sequence; and update the second gradient sequence according to the non-important gradients; the device further includes an iterative control module, which is used to iteratively perform distributed communication-based neural network training until a preset training stop condition is reached.
[0183] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0184] The present disclosure also provides a computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, and when the at least one instruction or at least one program is loaded and executed by a processor, the method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.
[0185] The embodiment of the present disclosure also provides a training device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to perform the method. The training device can be provided as a terminal, a server or other forms of devices.
[0186] Fig.10 A block diagram of a training device according to an embodiment of the present disclosure is shown. For example, the training device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.
[0187] Reference Fig.10 , the training device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0188] The processing component 802 generally controls the overall operation of the training device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
[0189] The memory 804 is configured to store various types of data to support operations on the training device 800. Examples of such data include instructions for any application or method operating on the training device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0190] The power supply component 806 provides power to the various components of the training device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the training device 800.
[0191] The multimedia component 808 includes a screen that provides an output interface between the above-mentioned training device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The above-mentioned touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the above-mentioned touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the training device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.
[0192] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the training device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0193] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0194] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the training device 800. For example, the sensor assembly 814 can detect the open / closed state of the training device 800, the relative positioning of components, such as the display and keypad of the training device 800, the sensor assembly 814 can also detect the position change of the training device 800 or a component of the training device 800, the presence or absence of user contact with the training device 800, the orientation or acceleration / deceleration of the training device 800, and the temperature change of the training device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0195] The communication component 816 is configured to facilitate wired or wireless communication between the training device 800 and other devices. The training device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0196] In an exemplary embodiment, the training device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.
[0197] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the training device 800 to perform the above method.
[0198] Fig.11 1900 is a block diagram of another training device according to an embodiment of the present disclosure. For example, the training device 1900 may be provided as a server. Fig.11The training device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.
[0199] The training device 1900 may also include a power supply component 1926 configured to perform power management of the training device 1900, a wired or wireless network interface 1950 configured to connect the training device 1900 to a network, and an input / output (I / O) interface 1958. The training device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.
[0200] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the training device 1900 to perform the above method.
[0201] In an exemplary embodiment, a training system is also provided, the training system comprising a plurality of the above-mentioned training devices.
[0202] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0203] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0204] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0205] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C+, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0206] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0207] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0208] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0209] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the above-mentioned module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0210] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A neural network training method based on distributed communication, characterized in that: Applied to a training node, the method comprises: Training the neural network corresponding to the training node, and saving the generated gradient in the first gradient sequence; The first gradient sequence and the second gradient sequence are segmented respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein the second gradient sequence is used to record gradients that have not yet participated in synchronization; if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameters; Setting a plurality of parallel computing threads, each of the computing threads acquiring at least one first gradient segment and a second gradient segment corresponding to the first gradient segment; Each of the computing threads accumulates, for each first gradient segment obtained, the first gradient segment and the corresponding second gradient segment to obtain a corresponding accumulated gradient segment; Obtaining a cumulative gradient sequence according to the cumulative gradient segments obtained by each of the computing threads; Calculate the importance index sequence according to the cumulative gradient sequence; Obtain an important gradient indication sequence, and for each accumulated gradient segment calculated by each computing thread, extract a corresponding important gradient indication value from the important gradient indication sequence; if the important gradient indication value is a first indication value, determine the accumulated gradients in the accumulated gradient segment as important gradients, and submit the accumulated gradient segment to the communication buffer of the training node; the first indication value indicates that the accumulated gradients in the accumulated gradient segment are all important gradients; Obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence; Adding the gradient sequences to be synchronized in the information to be synchronized of each training node element by element to obtain a synchronized accumulated gradient sequence; Dividing each synchronized accumulated gradient in the synchronized accumulated gradient sequence by the total number of training nodes to obtain a synchronized gradient sequence; Adding the importance index sequences in the information to be synchronized of each training node element by element to obtain a cumulative importance index sequence; Dividing each accumulated importance index in the accumulated importance index sequence by the total number of training nodes to obtain an average importance index sequence; Calculating the important gradient indicator value corresponding to each average importance indicator in the average importance indicator sequence to obtain a synchronized important gradient indicator sequence; The parameters of the neural network are adjusted according to the synchronized gradient sequence, and the synchronized important gradient indication sequence is used as a new important gradient indication sequence.
2. The method according to claim 1, characterized in that The step of calculating the importance index sequence according to the cumulative gradient sequence includes: Each of the computing threads calculates a corresponding importance index according to the obtained accumulated gradient segment; An importance index sequence is obtained according to the importance index calculation results of each of the computing threads.
3. The method according to claim 1, characterized in that The obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence includes: Obtaining a gradient sequence to be synchronized according to the accumulated gradient segments in the communication buffer; wherein the position of each accumulated gradient in each accumulated gradient segment in the communication buffer in the gradient sequence to be synchronized is the same as the position of the accumulated gradient in the accumulated gradient sequence, and other positions in the sequence to be synchronized are set to preset gradient values; The to-be-synchronized gradient sequence and the importance index sequence are concatenated to obtain the to-be-synchronized information.
4. The method according to claim 3, characterized in that The calculating the important gradient indicator value corresponding to each average importance indicator in the average importance indicator sequence includes: Obtaining the sorting results of the average importance indicators in the average importance indicator sequence according to the descending order of the average importance indicators; Obtaining a threshold indicator, and determining an importance indicator threshold in the sorting result according to the threshold indicator; For each average importance indicator in the average importance indicator sequence, if the average importance indicator is greater than the importance indicator threshold, the corresponding importance gradient indicator value is set to the first indicator value; otherwise, the corresponding importance gradient indicator value is set to the second indicator value; The first indicator value is used to indicate an important gradient, and the second indicator value is used to indicate an unimportant gradient.
5. The method according to claim 4, characterized in that The obtaining of the threshold indicator comprises: determining the number of segments according to the preset segmentation rule; Get the preset compression rate; The product of the compression ratio and the number of segments is used as the threshold indicator.
6. The method according to any one of claims 1 to 5, characterized in that: The adjusting the parameters of the neural network according to the synchronized gradient sequence comprises: extracting gradients in the synchronized gradient sequence in sequence; If the gradient is not equal to a preset gradient value, adjusting the corresponding neural network parameter according to the gradient; If the gradient is equal to the preset gradient value, the next gradient is extracted.
7. The method according to claim 1, characterized in that The method further comprises: determining non-significant gradients in the accumulated gradient sequence according to the significant gradient indication sequence; updating the second gradient sequence according to the non-significant gradient; The distributed communication-based neural network training is iterated until the preset training stop condition is reached.
8. A neural network training method based on distributed communication, characterized in that: Applied to a training system including a plurality of training nodes, the method comprises: Each training node trains the corresponding neural network and saves the generated gradient in the first gradient sequence; The first gradient sequence and the second gradient sequence are segmented respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein the second gradient sequence is used to record the gradients in the training node that have not yet participated in synchronization; if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameter; multiple parallel computing threads are set, each of which obtains at least one first gradient segment and a second gradient segment corresponding to the first gradient segment; for each first gradient segment obtained, each computing thread accumulates the first gradient segment and the corresponding second gradient segment to obtain a corresponding cumulative gradient segment; according to the cumulative gradient segments obtained by each computing thread, a cumulative gradient sequence is obtained; Calculate the importance index sequence according to the cumulative gradient sequence; Obtain an important gradient indication sequence in the training node, and for each accumulated gradient segment calculated by each computing thread, extract a corresponding important gradient indication value in the important gradient indication sequence; if the important gradient indication value is a first indication value, determine the accumulated gradients in the accumulated gradient segment as important gradients, and submit the accumulated gradient segment to the communication buffer of the training node; the first indication value indicates that the accumulated gradients in the accumulated gradient segment are all important gradients; Obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence; Each training node adds the gradient sequences to be synchronized in the information to be synchronized of each training node element by element to obtain a synchronized cumulative gradient sequence; divides each synchronized cumulative gradient in the synchronized cumulative gradient sequence by the total number of the training nodes to obtain a synchronized gradient sequence; adds the importance index sequences in the information to be synchronized of each training node element by element to obtain a cumulative importance index sequence; divides each cumulative importance index in the cumulative importance index sequence by the total number of the training nodes to obtain an average importance index sequence; calculates the important gradient indication value corresponding to each average importance index in the average importance index sequence to obtain a synchronized important gradient indication sequence; Each training node adjusts the parameters of the neural network according to the synchronized gradient sequence, and uses the synchronized important gradient indication sequence as a new important gradient indication sequence.
9. A neural network training device based on distributed communication, characterized in that: The device is arranged at a training node and includes: A training module, used for training the neural network corresponding to the training node, and storing the generated gradient in the first gradient sequence; A cumulative gradient acquisition module, used for segmenting the first gradient sequence and the second gradient sequence respectively based on a preset segmentation rule to obtain a first gradient segment sequence and a second gradient segment sequence; wherein the second gradient sequence is used to record the gradient that has not yet participated in synchronization; if the position of the first gradient segment in the first gradient segment sequence is the same as the position of the second gradient segment in the second gradient segment sequence, then the first gradient segment and the second gradient segment correspond to each other and both correspond to the same neural network parameter; multiple parallel computing threads are set, each of which acquires at least one first gradient segment and a second gradient segment corresponding to the first gradient segment; each of the computing threads accumulates the first gradient segment and the corresponding second gradient segment for each acquired first gradient segment to obtain a corresponding cumulative gradient segment; a cumulative gradient sequence is obtained according to the cumulative gradient segments obtained by each of the computing threads; An importance index sequence calculation module, used to calculate the importance index sequence according to the cumulative gradient sequence; A gradient classification module is used to obtain an important gradient indication sequence, and for each accumulated gradient segment calculated by each computing thread, extract a corresponding important gradient indication value from the important gradient indication sequence; if the important gradient indication value is a first indication value, all accumulated gradients in the accumulated gradient segment are determined to be important gradients, and the accumulated gradient segment is submitted to the communication buffer of the training node; the first indication value indicates that all accumulated gradients in the accumulated gradient segment are important gradients; A synchronization module, used for obtaining the information to be synchronized of the training node according to the important gradient and the importance index sequence, adding the gradient sequences to be synchronized in the information to be synchronized of each training node element by element to obtain a synchronized cumulative gradient sequence; dividing each synchronized cumulative gradient in the synchronized cumulative gradient sequence by the total number of the training nodes to obtain a synchronized gradient sequence; adding the importance index sequences in the information to be synchronized of each training node element by element to obtain a cumulative importance index sequence; dividing each cumulative importance index in the cumulative importance index sequence by the total number of the training nodes to obtain an average importance index sequence; calculating the important gradient indication value corresponding to each average importance indicator in the average importance index sequence to obtain a synchronized important gradient indication sequence; An updating module, configured to use the synchronized important gradient indication sequence as a new important gradient indication sequence; A parameter adjustment module is used to adjust the parameters of the neural network according to the synchronized gradient sequence.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the neural network training method based on distributed communication as described in any one of claims 1-7.
11. A training device, characterized in that comprising at least one processor, and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the neural network training method based on distributed communication as described in any one of claims 1 to 7 by executing the instructions stored in the memory.
12. A training system, characterized in that: The training system comprises a plurality of training devices according to claim 11.
Citation Information
Patent Citations
Neural network linguistic model training method and device, equipment and storage medium
CN110379416A
Neural network distributed training method and device, equipment and storage medium
CN112288083A
Asynchronous gradient averaging distributed stochastic gradient descent
WO2019042571A1
