Adaptive neural network training methods, electronic devices, media, and software products

Abstract

This invention provides an adaptive neural network training method, electronic device, medium, and program product. The method includes: training a target neural network based on adaptive parameters of the current training epoch, wherein the adaptive parameters are used to determine the training workload of each training node in the target neural network; adjusting the adaptive parameters of the current training epoch based on the training time of each training node in the current training epoch, and determining the adjusted adaptive parameters as the adaptive parameters for the next training epoch. The method, electronic device, medium, and program product provided by this invention eliminate the need for manually assigning training tasks to training nodes with different performance levels, reducing labor costs and improving the training efficiency of deep learning tasks in heterogeneous clusters. Furthermore, it enables distributed training based on performance allocation, thereby further improving the training efficiency of deep learning tasks in heterogeneous clusters.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to a self-adaptive neural network training method, an electronic device, a medium and a program product. BACKGROUND

[0002] With the rapid development of deep learning technology, deep learning has been widely applied in image recognition, natural language processing, speech recognition, reinforcement learning and other fields. In order to achieve better results in practical applications, the deep learning structure represented by neural networks is becoming more and more complex, that is, the number of network layers and the number of parameters of neural networks are increasing, and at the same time, the data set for training neural networks is also getting larger, so that the training process of deep learning tasks needs to consume a large amount of computing resources. The computing power of a single machine is limited, which makes the training time too long.

[0003] At present, in order to save training time, distributed training is usually used for training. Distributed training refers to connecting multiple GPU servers through high-performance networks to jointly train a deep learning task, that is, multiple training nodes jointly complete the training task.

[0004] In the process of distributed training, each training node is usually allocated an equal amount of training tasks. However, the computing power of each training node is usually different, and the training node with poor computing power will slow down the entire training process, resulting in reduced training efficiency of deep learning tasks in a heterogeneous cluster. SUMMARY

[0005] The present application provides a self-adaptive neural network training method, an electronic device, a medium and a program product to solve the defect of low training efficiency of deep learning tasks in a heterogeneous cluster in the prior art.

[0006] The present application provides a self-adaptive neural network training method, comprising:

[0007] training the target neural network based on the adaptive parameters of the current training round, wherein the adaptive parameters are used to determine the training task amount of each training node for training the target neural network;

[0008] adjusting the adaptive parameters of the current training round based on the training time of each training node in the current training round, and determining the adjusted adaptive parameters as the adaptive parameters of the next training round.

[0009] According to the self-adaptive neural network training method provided by the present application, the target neural network is trained based on the adaptive parameters of the current training round, comprising:

[0010] determining the task amount weight of each training node based on the adaptive parameters of the current training round;

[0011] training the target neural network based on the task weight of each training node.

[0012] According to the adaptive neural network training method provided by the application, any training node trains the target neural network based on the following steps:

[0013] determining the sub-training data set of the any training node from the total training data set based on the task weight of the any training node;

[0014] training the target neural network based on the sub-training data set of the any training node to obtain gradient accumulation parameters;

[0015] updating the network parameters of the target neural network based on the gradient accumulation parameters.

[0016] According to the adaptive neural network training method provided by the application, the adaptive parameter of the current training round is adjusted based on the training time of each training node in the current training round, which includes:

[0017] determining the task change amount of each training node based on the training time of each training node in the current training round and the adaptive parameter of the current training round;

[0018] adjusting the adaptive parameter of the current training round based on the task change amount of each training node.

[0019] According to the adaptive neural network training method provided by the application, the target neural network is trained based on the adaptive parameter of the current training round, and the method further includes:

[0020] obtaining a training script representing a training task, and verifying the training script based on a preset training script writing specification;

[0021] if the training script meets the training script writing specification, adaptively packaging the training script and starting the packaged training script, and the adaptive packaging is used to add the adaptive parameter of the first training round to the training script.

[0022] According to the adaptive neural network training method provided by the application, the training script representing the training task is obtained, and the training script is verified based on the preset training script writing specification, and the method further includes:

[0023] if the training script does not meet the training script writing specification, an error prompt information is sent, and the error prompt information is used to prompt to modify the training script;

[0024] obtaining a modified training script, and checking the modified training script based on a training script writing specification;

[0025] if the modified training script meets the training script writing specification, then performing adaptive packaging on the modified training script, and starting the packaged training script;

[0026] if the modified training script does not meet the training script writing specification, then issuing an error prompt, and returning to the step of obtaining a modified training script, and checking the modified training script based on a training script writing specification, until the modified training script meets the training script writing specification.

[0027] According to the adaptive neural network training method provided by the application, the training script writing specification comprises at least one of the following: a file format specification, a file name specification, a training framework calling specification, and a key variable naming specification.

[0028] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the adaptive neural network training method according to any one of the above when executing the program.

[0029] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the adaptive neural network training method according to any one of the above.

[0030] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the steps of the adaptive neural network training method according to any one of the above.

[0031] The adaptive neural network training method, the electronic device, the medium and the program product provided by the application train the target neural network based on the adaptive parameter of the current training round, wherein the adaptive parameter is used to determine the training task amount of each training node of the target neural network, based on which, the training task can be allocated to the training nodes with different performance without manual work, the labor cost can be reduced, and the training efficiency of the deep learning task under the heterogeneous cluster can be improved; the adaptive parameter of the current training round is adjusted based on the training time of each training node in the current training round, and the adjusted adaptive parameter is determined as the adaptive parameter of the next training round, based on which, the adaptive parameter can be iteratively updated based on the training time of each training node in any training round, and finally the optimal adaptive parameter of each training node, i.e., the optimal adaptive parameter of the current heterogeneous cluster, is obtained, therefore, the distributed training mode according to the ability can be realized by the application, and the training efficiency of the deep learning task under the heterogeneous cluster can be further improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0033] Figure 1 One of the flowcharts of the adaptive neural network training method provided by the application;

[0034] Figure 2 The flowchart of the node training provided by the application;

[0035] Figure 3 The second flowchart of the adaptive neural network training method provided by the application;

[0036] Figure 4 The third flowchart of the adaptive neural network training method provided by the application;

[0037] Figure 5 The structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0038] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0039] In recent years, due to the better effect of deep learning algorithm than traditional algorithm in various tasks, deep learning has been widely applied in image recognition, natural language processing, speech recognition, reinforcement learning and other fields. In order to achieve better training effect in practical application, on the one hand, the deep learning structure represented by neural network is becoming more and more complex, that is, the number of network layers and the number of parameters of neural network are increasing, on the other hand, the data set of deep learning training is getting larger and larger, such as the training data set in some professional field will be as high as TB or even PB level. The increase of neural network parameters and the increase of training data set size will make the deep learning training process need to consume a large amount of computing resources.

[0040] Due to the limited computing power of a single machine, the training time is too long. In order to save the training time, in recent years, the training method mainly based on distributed training is more and more applied in the field of deep learning. Distributed training refers to connecting multiple GPU servers through high-performance network to train a task together. This distributed training mode breaks the original computing power limit, expands the computing scale, and thus saves the training time.

[0041] At present, distributed deep learning has made certain progress, and common deep learning frameworks have begun to support distributed training tasks, and have achieved significant acceleration effect in some large training tasks. However, due to the excessive number of training nodes, the complex neural network structure and other reasons, distributed training still faces many problems and challenges, one of which is the distributed training of heterogeneous computing resources.

[0042] When the computing performance of each training node is the same, the training efficiency of distributed training is the highest. However, in the actual production environment, because the machines of each training node are generally purchased in batches, there are usually multiple models of training nodes in the machine room, and the computing performance, storage performance, data transmission performance and other performances of these training nodes are very different.

[0043] At present, in the process of distributed training, each training node is usually allocated an equal amount of training tasks, and this method can also complete the training task, but because of the bucket effect, the whole training process will be slowed down by the training node with the worst computing performance, so the training efficiency of this method is not high, and thus the training efficiency of deep learning task under heterogeneous cluster is reduced.

[0044] To address the aforementioned issues, this invention considers the performance differences between different training nodes during training in heterogeneous clusters. Therefore, this invention employs a training method based on performance allocation, assigning more training tasks to higher-performing training nodes and fewer to lower-performing nodes. This reduces the idle time of higher-performing training nodes and helps improve the overall efficiency of distributed training.

[0045] This invention also considers that the performance differences between different training nodes are difficult to quantify. Although each training node has a performance metric, these metrics may not be very helpful in guiding specific training tasks, and they become increasingly inaccurate as the machine ages. Furthermore, allocating training tasks according to capacity requires a significant amount of coding work, which undoubtedly places a heavy burden on algorithm engineers.

[0046] Based on the above, the present invention provides an adaptive neural network training method. Figure 1 This is one of the flowcharts illustrating the adaptive neural network training method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0047] Step 110: Train the target neural network based on the adaptive parameters of the current training round, wherein the adaptive parameters are used to determine the training workload of each training node in the target neural network.

[0048] Specifically, if the current training epoch is the first training epoch, the adaptive parameters for the current training epoch can be random values ​​or the adaptive parameters finally adjusted in the previous training task. These random values ​​can be obtained through random initialization using a random algorithm or determined by pre-set initial parameters. If the current training epoch is not the first training epoch, the adaptive parameters for the current training epoch are the adaptive parameters adjusted in the previous training epoch.

[0049] Here, the adaptive parameter is used to characterize the training task volume of each training node, which can be reflected as the size of the training dataset to be allocated to each training node. This adaptive parameter can specifically be reflected as a task weight or a task proportion, etc., and this embodiment of the invention does not limit this. This embodiment of the invention uses task weight as an example for illustration.

[0050] For example, the task weight assigned to n training nodes in the k-th training round is: When k=1, the task weight of each training node can be a random value. For example, the task weight of each training node can be...

[0051] It should be noted that the entire training task may include one or more training epochs, that is, the epoch can be 1 or an integer greater than 1.

[0052] In each training round, the specific steps for training the target neural network are as follows: All training samples in the total training dataset are trained once in the target neural network; that is, all training samples in the total training dataset undergo one forward propagation and one backward propagation. Furthermore, each training node trains all training samples in its respective sub-training dataset once in the target neural network.

[0053] Furthermore, it should be noted that the adaptive neural network training method of this invention can be applied to any training node among the training nodes, or it can be applied to another computing node independent of the training nodes. This invention does not specifically limit this.

[0054] Step 120: Based on the training time of each training node in the current training round, adjust the adaptive parameters of the current training round, and determine the adjusted adaptive parameters as the adaptive parameters for the next training round.

[0055] Specifically, after each training node has completed training the target neural network, each training node records the training time for the current training round. If a training node completes its training task in advance, it can first record the training time for the current training round, and then synchronously wait for other training nodes to complete their respective training tasks before adjusting the adaptive parameters for the current training round based on the training times of all training nodes.

[0056] For example, the task weight assigned to the current training round (the kth training round) of n training nodes is: The task weight for the next training round is then...

[0057] Here, training time can be the gradient calculation time of the training node, or the overall training time for the training node to complete forward and backward propagation. The embodiments of the present invention are not specifically limited.

[0058] In one embodiment, each training node can broadcast its own training time to other training nodes and receive the training times from other training nodes. In this case, each training node contains the training times of all training nodes, and based on this, each training node can adjust the adaptive parameters of the current training round based on the training times of each training node in the current training round.

[0059] In another embodiment, each training node can broadcast its training time to a computing node, which can then adjust the adaptive parameters for the current training epoch based on the training time of each training node. This computing node can be any of the training nodes or a separate computing node independent of the training nodes; this embodiment of the invention does not specifically limit its application.

[0060] It is understandable that the adjusted adaptive parameters will affect the allocation of training tasks for each training node in the next training round, thereby affecting the training time of each training node in the next training round. In other words, this embodiment of the invention iteratively searches for suitable adaptive parameters so that each training node can maximize its performance, ensuring that no training node has idle time, thereby improving the overall training efficiency. Based on this, after step 120 above, the method further includes:

[0061] When starting the training task for the next training round, use the adaptive parameters of the next training round as the adaptive parameters of the current training round, and return to step 110 above, until the training tasks for all training rounds are completed.

[0062] In some embodiments, if in a certain training round, the adjusted adaptive parameters are basically the same as the original adaptive parameters (i.e., the adaptive parameters of the current training round), then the adjusted adaptive parameters can be used as the optimal adaptive parameters for the current cluster.

[0063] In other embodiments, if the current training epoch is the last training epoch, the adjusted adaptive parameters are determined as the adaptive parameters for the next training task, i.e., as the adaptive parameters for the first training epoch of the next training task. The adjusted adaptive parameters in the last training epoch can be used as the optimal adaptive parameters for the current cluster (the cluster to which each training node belongs).

[0064] The adaptive neural network training method provided in this invention trains the target neural network based on adaptive parameters of the current training round. These adaptive parameters determine the training workload of each training node in the target neural network. This eliminates the need for manual allocation of training tasks to nodes with different performance levels, reducing labor costs and improving the training efficiency of deep learning tasks in heterogeneous clusters. Based on the training time of each training node in the current training round, the adaptive parameters of the current training round are adjusted, and the adjusted adaptive parameters are used as the adaptive parameters for the next training round. Therefore, the adaptive parameters can be iteratively updated based on the training time of each training node in any training round, ultimately obtaining the optimal adaptive parameters for each training node, i.e., the optimal adaptive parameters for the current heterogeneous cluster. Thus, this invention can achieve a distributed training method based on available resources, further improving the training efficiency of deep learning tasks in heterogeneous clusters.

[0065] Based on the above embodiments, in this method, step 110 includes:

[0066] Based on the adaptive parameters of the current training round, the task weight of each training node is determined;

[0067] The target neural network is trained based on the task weights of each training node.

[0068] Specifically, each training node reads in the task weight of the current training round and obtains the corresponding sub-training dataset in the total training dataset based on the task weight. Then, the target neural network is trained based on the training samples in the sub-training dataset. For example, gradient accumulation training is performed based on the sample data and labels in the sub-training dataset, and the network parameters of the target neural network are updated based on the gradient accumulation parameters.

[0069] For any training node, a sub-training dataset for that training node is determined from the total training dataset based on the task weight of that training node; the target neural network is trained based on the sub-training dataset.

[0070] The adaptive neural network training method provided in this invention determines the task weights of each training node based on adaptive parameters of the current training round, allowing each training node to train the target neural network according to its corresponding task weights. Through this method, this invention can automatically determine the allocation of training tasks for training nodes with different performance levels using adaptive parameters, eliminating the need for manual allocation of training tasks to nodes with varying performance. This reduces labor costs and improves the training efficiency of deep learning tasks in heterogeneous clusters.

[0071] Based on any of the above embodiments Figure 2This is a schematic diagram of the node training process provided by the present invention, as follows: Figure 2 As shown, in this method, any training node trains the target neural network based on the following steps:

[0072] Step 210: Based on the task weight of any training node, determine the sub-training dataset of any training node from the total training dataset.

[0073] Here, the number of sub-training datasets can be obtained by multiplying the task weights by the total number of training datasets. After determining the number of sub-training datasets, training data can be selected from the total training dataset as sub-training datasets based on the number of sub-training datasets. The specific method of selecting training data can be random selection or selection according to preset rules, and this embodiment of the invention does not limit this.

[0074] Step 220: Based on the sub-training dataset of any training node, perform gradient accumulation training on the target neural network to obtain gradient accumulation parameters.

[0075] Specifically, if the training task of the target neural network is supervised training, gradient accumulation training is performed based on the sample data and labels in the sub-training dataset of any training node. If the training task of the target neural network is unsupervised training, gradient accumulation training is performed based on the sample data in the sub-training dataset of any training node.

[0076] Step 230: Update the network parameters of the target neural network based on the gradient accumulation parameters.

[0077] Specifically, after each training node obtains its gradient accumulation parameters, the network parameters of the target neural network are updated based on the gradient accumulation parameters of all training nodes. Further, after any training node pre-completes gradient accumulation training, it enters a synchronization waiting phase until all training nodes have completed gradient accumulation training, i.e., all training nodes have reached the synchronization waiting area. Then, the `all_reduce` operation is initiated to update the network parameters of the target neural network. This `all_reduce` operation is a standard technique in distributed deep learning and will not be elaborated upon here.

[0078] The adaptive neural network training method provided in this invention determines a sub-training dataset for any training node from the total training dataset based on the task weight of that training node; performs gradient accumulation training on the target neural network based on the sub-training dataset of any training node to obtain gradient accumulation parameters; and updates the network parameters of the target neural network based on the gradient accumulation parameters. Through this method, this invention can automatically determine the sub-training dataset of any training node by using task weights, i.e., automatically determine the training task of any training node, eliminating the need for manual allocation of training tasks to training nodes with different performance levels. This reduces labor costs and improves the training efficiency of deep learning tasks in heterogeneous clusters. Simultaneously, the gradient accumulation training method can enhance the training effect of the target neural network.

[0079] Based on any of the above embodiments Figure 3 The second flowchart illustrating the adaptive neural network training method provided by this invention is shown below. Figure 3 As shown, in this method, step 120 above, adjusting the adaptive parameters of the current training round based on the training time of each training node in the current training round, includes:

[0080] Step 121: Based on the training time of each training node in the current training round and the adaptive parameters of the current training round, determine the task change amount of each training node.

[0081] In one embodiment, the task change of any training node is calculated based on the following formula:

[0082]

[0083] Among them, u i Let represent the task change amount for the i-th training node. The task weight for the i-th training node in the k-th training round (i.e., the current training round is k) is represented by the adaptive parameter, which is the task weight. Let be the training time of the i-th training node in the current training round, and n be the number of training nodes.

[0084] In another embodiment, the task change of any training node is calculated based on the following formula:

[0085]

[0086] Among them, u i Let represent the task change amount for the i-th training node. The task weight for the i-th training node in the k-th training round (i.e., the current training round is k) is represented by the adaptive parameter, which is the task weight. Let be the training time of the i-th training node in the current training round, n be the number of training nodes, and round() be the rounding function.

[0087] Step 122: Adjust the adaptive parameters of the current training round based on the task change amount of each training node.

[0088] Specifically, the adjusted adaptive parameters are obtained by adding the task change amount to the adaptive parameters of the current training round.

[0089] The formula for adjusting the adaptive parameters is shown below:

[0090]

[0091] in, The adjusted task weight for the i-th training node (adaptive parameters are reflected as task weights), that is, the task weight for the i-th training node in the next training round, that is, the task weight for the i-th training node in the k+1-th training round. The task weight for the i-th training node in the k-th training round (i.e., the current training round is k) is represented by the adaptive parameter, u. i Let represent the task change amount for the i-th training node.

[0092] In one specific embodiment, if the total number of training epochs is 1, i.e., epoch = 1, the adaptive parameters for the current training epoch may not need to be adjusted.

[0093] The adaptive neural network training method provided in this invention determines the task change amount of each training node based on the training time and adaptive parameters of each training node in the current training round; and adjusts the adaptive parameters of the current training round based on the task change amount of each training node. Through this method, the adaptive parameters can be iteratively updated based on the training time of each training node in any training round, ultimately obtaining the optimal adaptive parameters for each training node, i.e., the optimal adaptive parameters for the current heterogeneous cluster. Therefore, this invention can realize a distributed training method based on energy allocation, thereby further improving the training efficiency of deep learning tasks in heterogeneous clusters.

[0094] Based on any of the above embodiments Figure 4 The third flowchart illustrating the adaptive neural network training method provided by this invention is shown below. Figure 4 As shown, prior to step 110 above, the method further includes:

[0095] Step 410: Obtain the training script for the characterization training task, and verify the training script based on the preset training script writing specifications.

[0096] Here, the training script is used to encapsulate the training task, that is, to encapsulate the overall training task of the target neural network, i.e., to encapsulate the deep learning task. This training script can be written by the user, who must follow the training script writing specifications.

[0097] Here, the training script writing guidelines are used to assist and constrain users in writing corresponding training scripts. Furthermore, users write corresponding training scripts according to the training script writing guidelines and the requirements of the training task.

[0098] In addition, after importing the training script writing specification, the specification can be used to verify the input training script. Specifically, it checks whether the training script meets the encoding requirements and whether it complies with the relevant restrictions defined in the specification.

[0099] Training script writing specifications include, but are not limited to, one or more of the following: file format specifications, file name specifications, training framework calling specifications, key variable naming specifications, etc., and the embodiments of the present invention do not impose specific limitations on these.

[0100] In one embodiment, the training script writing specifications include at least one of the following: file format specifications, file name specifications, training framework calling specifications, and key variable naming specifications.

[0101] The file format specification defines the file format of the training script. The naming specification defines the file name of the training script. The training framework calling specification defines how the training framework is called in the training script. The key variable naming specification defines the naming of key variables (important elements) in the training script.

[0102] Understandably, judging whether the input training script has problems based on the training script writing specifications makes the writing of training scripts more systematic and prevents the existence of ever-changing training scripts.

[0103] Step 420: If the training script conforms to the training script writing specifications, then the training script is adaptively encapsulated and the encapsulated training script is started. The adaptive encapsulation is used to add adaptive parameters for the first training round to the training script.

[0104] Specifically, an adaptive encapsulation program can be used to encapsulate the training script. Adaptive encapsulation refers to adding adaptive parameters to the training script while maintaining its original training functionality, thereby enabling the training script to perform adaptive distributed training and thus enabling the training task to perform adaptive distributed training.

[0105] Furthermore, adaptive encapsulation is also used to encapsulate code that adjusts adaptive parameters, as well as code that selects appropriate sub-training datasets based on task weights.

[0106] The adaptive encapsulation program can customize the encapsulation of training scripts that conform to the training script writing specifications. Specifically, the adaptive encapsulation program can be imported to encapsulate the training scripts.

[0107] Furthermore, the adaptive encapsulation program can be implemented using tools such as a Python interpreter and regular expressions; this embodiment of the invention does not specifically limit its implementation. Based on this, user-input training scripts can be encapsulated for adaptive training.

[0108] In another embodiment, adaptive encapsulation refers to adding adaptive parameters to the training script while maintaining its original training functionality, and / or fine-tuning the training logic of the training task, and / or encapsulating the training framework used in the script, so that the training script has the function of adaptive distributed training, thereby enabling the training task to have the function of adaptive distributed training.

[0109] It should be noted that starting the encapsulated training script means starting the training task and beginning the training process, so that each training node can enter the training phase.

[0110] Understandably, the challenge in designing adaptive distributed training methods lies in enabling users to efficiently and conveniently utilize these methods on heterogeneous clusters, achieving training acceleration at low cost. Based on this, this invention establishes a set of training script writing specifications to constrain the writing of training scripts; that is, the original training scripts must be written according to these specifications. These training script writing specifications are universal, meeting the requirements of most training tasks; simultaneously, they are simple, minimizing the coding work for algorithm engineers. Therefore, training scripts designed following these specifications can meet the development needs of most deep learning tasks.

[0111] The adaptive neural network training method provided in this invention obtains a training script representing the training task and verifies the training script based on a pre-defined training script writing specification. If the training script conforms to the specification, it is adaptively encapsulated and then launched. The adaptive encapsulation adds adaptive parameters for the first training round to the training script. By verifying the training script based on the specification, this ensures that only training scripts conforming to the specification are adaptively encapsulated, thus ensuring that the training task can execute normally after the training script is launched. Simultaneously, the adaptive encapsulation transforms the training script into an adaptively trainable script, allowing for iterative updates of the adaptive parameters to ultimately obtain the optimal adaptive parameters for each training node, i.e., the optimal adaptive parameters for the current heterogeneous cluster, thereby further improving the training efficiency of deep learning tasks in heterogeneous clusters.

[0112] Based on any of the above embodiments, after step 410, the method further includes:

[0113] If the training script does not conform to the training script writing specifications, an error message will be issued, which will prompt the training script to be modified.

[0114] Obtain the modified training script and verify the modified training script based on the training script writing specifications;

[0115] If the modified training script conforms to the training script writing specifications, then the modified training script is adaptively encapsulated and the encapsulated training script is started.

[0116] If the modified training script does not conform to the training script writing specifications, an error message is issued, and the process returns to obtain the modified training script and verify it based on the training script writing specifications, until the modified training script conforms to the training script writing specifications.

[0117] Here, the error message prompts the user to modify the training script. The user can use this error message to make changes. The error message can include details of any non-compliance in the training script, allowing the user to quickly identify errors and speed up the modification process. Furthermore, after the user has made the changes, the modified training script will be re-entered.

[0118] For ease of understanding, a specific embodiment is used as an example. This embodiment is as follows: First, the training script writing specification is imported. Then, the training script representing the training task is obtained. Next, the training script is checked against the imported training script writing specification. If the training script does not conform to the specification, the system reports an error, and the user needs to modify the training script to conform to the specification. If the script conforms to the specification, the adaptive encapsulation program is imported. The system will use the adaptive encapsulation program to encapsulate the training script. Next, the system will start the training script and adjust relevant parameters according to the adaptive algorithm. This adaptive algorithm is the same as the steps executed in steps 110 and 120 above, which will not be described in detail here. After the adaptive parameters stabilize, the final adjusted adaptive parameters are the optimal adaptive parameters for the current cluster. At this point, the stable adaptive parameters are output, and training continues using these parameters.

[0119] The adaptive neural network training method provided in this embodiment of the invention verifies the training script based on the training script writing specifications. If the training script does not conform to the training script writing specifications, an error message is output until a training script that conforms to the training script writing specifications is input. This ensures that the training script that conforms to the training script writing specifications is adaptively encapsulated, thereby ensuring that the training task can be executed normally after the training script is started.

[0120] The neural network training device provided by the present invention will be described below. The neural network training device described below can be referred to in correspondence with the adaptive neural network training method described above.

[0121] In this embodiment, the neural network training device includes:

[0122] The training module is used to train the target neural network based on adaptive parameters of the current training round, wherein the adaptive parameters are used to determine the training workload of each training node in the target neural network.

[0123] The adjustment module is used to adjust the adaptive parameters of the current training round based on the training time of each training node in the current training round, and to determine the adjusted adaptive parameters as the adaptive parameters for the next training round.

[0124] The neural network training device provided in this invention trains a target neural network based on adaptive parameters of the current training round. These adaptive parameters determine the training workload of each training node in the target neural network. This eliminates the need for manual allocation of training tasks to nodes with different performance levels, reducing labor costs and improving the training efficiency of deep learning tasks in heterogeneous clusters. The device adjusts the adaptive parameters of the current training round based on the training time of each training node, and uses the adjusted parameters as the adaptive parameters for the next training round. This allows for iterative updates of the adaptive parameters based on the training time of each training node in any training round, ultimately obtaining the optimal adaptive parameters for each training node, i.e., the optimal adaptive parameters for the current heterogeneous cluster. Therefore, this invention enables a distributed training method based on available resources, further improving the training efficiency of deep learning tasks in heterogeneous clusters.

[0125] Based on any of the above embodiments, the training module is further configured to:

[0126] Based on the adaptive parameters of the current training round, the task weight of each training node is determined;

[0127] The target neural network is trained based on the task weights of each training node.

[0128] Based on any of the above embodiments, any training node trains the target neural network according to the following steps:

[0129] Based on the task weight of any training node, a sub-training dataset of any training node is determined from the total training dataset.

[0130] Based on the sub-training dataset of any of the training nodes, the target neural network is trained by gradient accumulation to obtain gradient accumulation parameters;

[0131] The network parameters of the target neural network are updated based on the gradient accumulation parameters.

[0132] Based on any of the above embodiments, the adjustment module is further configured to:

[0133] Based on the training time of each training node in the current training round and the adaptive parameters of the current training round, the task change amount of each training node is determined.

[0134] Based on the task changes of each training node, the adaptive parameters of the current training round are adjusted.

[0135] Based on any of the above embodiments, the neural network training device further includes:

[0136] The first verification module is used to obtain the training script representing the training task and verify the training script based on the preset training script writing specifications.

[0137] The first encapsulation module is used to adaptively encapsulate the training script if the training script conforms to the training script writing specifications, and then start the encapsulated training script. The adaptive encapsulation is used to add adaptive parameters for the first training round to the training script.

[0138] Based on any of the above embodiments, the neural network training device further includes:

[0139] The information sending module is used to send an error message if the training script does not conform to the training script writing specifications, and the error message is used to prompt the training script to be modified.

[0140] The second verification module is used to obtain the modified training script and verify the modified training script based on the training script writing specifications.

[0141] The second encapsulation module is used to adaptively encapsulate the modified training script if the modified training script conforms to the training script writing specifications, and then start the encapsulated training script.

[0142] The step return module is used to issue an error message and return to obtain the modified training script if the modified training script does not conform to the training script writing specification, and to verify the modified training script based on the training script writing specification, until the modified training script conforms to the training script writing specification.

[0143] Based on any of the above embodiments, the training script writing specifications include at least one of the following: file format specifications, file name specifications, training framework calling specifications, and key variable naming specifications.

[0144] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an adaptive neural network training method. This method includes: training a target neural network based on adaptive parameters of the current training epoch, wherein the adaptive parameters are used to determine the training workload of each training node in the target neural network; adjusting the adaptive parameters of the current training epoch based on the training time of each training node in the current training epoch, and determining the adjusted adaptive parameters as the adaptive parameters for the next training epoch.

[0145] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the adaptive neural network training method provided by the above methods, the method comprising: training a target neural network based on adaptive parameters of the current training round, wherein the adaptive parameters are used to determine the training workload of each training node of the target neural network; adjusting the adaptive parameters of the current training round based on the training time of each training node in the current training round, and determining the adjusted adaptive parameters as the adaptive parameters of the next training round.

[0147] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an adaptive neural network training method provided by the methods described above. The method includes: training a target neural network based on adaptive parameters of the current training epoch, wherein the adaptive parameters are used to determine the training workload of each training node in the target neural network; adjusting the adaptive parameters of the current training epoch based on the training time of each training node in the current training epoch, and determining the adjusted adaptive parameters as the adaptive parameters for the next training epoch.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of training an adaptive neural network, the method comprising: The application is applied to a computing node, comprising: training a target neural network based on adaptive parameters of a current training round, the adaptive parameters being used to determine training task amounts of training nodes for training the target neural network; adjusting the adaptive parameters of the current training round based on training times of the training nodes in the current training round, and determining the adjusted adaptive parameters as adaptive parameters of a next training round; the adjusting the adaptive parameters of the current training round based on the training times of the training nodes in the current training round comprises: determining task change amounts of the training nodes based on the training times of the training nodes in the current training round and the adaptive parameters of the current training round, the adaptive parameters being task weight; adjusting the adaptive parameters of the current training round based on the task change amounts of the training nodes; the task change amount of any training node is calculated based on the following formula: ； wherein, is a task variation of the i-th training node, is a task weight of the k-th training round of the i-th training node, is a training time of the i-th training node in the current training round, is a number of training nodes, and round() is a rounding function. the training a target neural network based on adaptive parameters of a current training round further comprises: obtaining a training script representing a training task, and verifying the training script based on preset training script writing specifications; if the training script conforms to the training script writing specifications, adaptively packaging the training script by adding adaptive parameters of a first training round to the training script, and starting the packaged training script; the obtaining a training script representing a training task, and verifying the training script based on preset training script writing specifications further comprises: if the training script does not conform to the training script writing specifications, issuing an error prompt message, the error prompt message being used to prompt modification of the training script; obtaining the modified training script, and verifying the modified training script based on the training script writing specifications; if the modified training script conforms to the training script writing specifications, adaptively packaging the modified training script, and starting the packaged training script; if the modified training script does not conform to the training script writing specifications, issuing an error prompt message, and returning to the step of obtaining the modified training script, and verifying the modified training script based on the training script writing specifications until the modified training script conforms to the training script writing specifications; the training script writing specifications comprise at least one of a file format specification, a file name specification, a calling specification of a training framework, and a naming specification of key variables.

2. The adaptive neural network training method of claim 1, wherein, the training a target neural network based on adaptive parameters of a current training round comprises: determining task weight of the training nodes based on the adaptive parameters of the current training round; training the target neural network based on the task weight of the training nodes.

3. The adaptive neural network training method of claim 2, wherein, any training node trains a target neural network based on the following steps: determining a sub-training data set of the any training node from a total training data set based on the task weight of the any training node. perform gradient accumulation training on the target neural network based on the sub-training data set of any one of the training nodes to obtain gradient accumulation parameters; update network parameters of the target neural network based on the gradient accumulation parameters.

4. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the adaptive neural network training method of any one of claims 1 to 3 when executing the program.

5. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the adaptive neural network training method of any one of claims 1 to 3.

6. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the adaptive neural network training method of any one of claims 1 to 3.

Images