Model training method, device, medium and equipment for heterogeneous clusters

Abstract

This specification discloses a model training method, apparatus, medium and equipment for heterogeneous clusters. A unified training configuration is used to train a target model in a heterogeneous cluster and a homogeneous benchmark cluster, respectively. The key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are stored in the iteration process every preset number of iteration steps. For each sampling, after storing the key indicators of the current iteration process, the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is determined, and whether to continue training the target model is determined based on the difference. By comparing the accuracy of a model trained based on a heterogeneous cluster with a model trained based on a heterogeneous cluster using the same training configuration at a preset frequency during the model training process based on a heterogeneous cluster, the model training process can be monitored in real time, abnormalities in the model training process can be discovered in a timely manner, and the accuracy of the trained model can be improved.

Description

Technical Field

[0001] This specification relates to the field of computers, and in particular to a model training method, apparatus, medium, and equipment for heterogeneous clusters. Background Art

[0002] With the rapid growth of computer technology and the rapid increase in model parameters and training data, the demand for computing resources has skyrocketed. Distributed training based on heterogeneous clusters, such as those using a mix of graphics processing units (GPUs), tensor processing units (TPUs), or other chips with different architectures, has become an important solution. Verifying model training accuracy is a key step in ensuring that models achieve expected performance in real-world applications.

[0003] In the prior art, large model training is typically performed after completion, with the training effect of the large model evaluated through macro indicators such as the final loss function value. This is essentially a black-box verification. However, due to hardware differences in heterogeneous clusters, such as computing accuracy and communication delays, inconsistent parameter updates can occur, and these errors may offset each other in the loss function. Therefore, in a heterogeneous cluster environment, existing model training methods have difficulty achieving real-time perception of anomalies during large model training. Based on this, this specification provides a model training method, apparatus, medium, and equipment for heterogeneous clusters. Summary of the Invention

[0004] This specification provides a model training method, apparatus, medium and equipment for heterogeneous clusters to partially solve the above-mentioned problems existing in the prior art.

[0005] This manual adopts the following technical solutions:

[0006] A model training method for a heterogeneous cluster, comprising:

[0007] Train the target model using a unified training configuration in both the heterogeneous cluster and the homogeneous benchmark cluster.

[0008] At every preset number of iterations, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are stored after the current round of iteration, and the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is calculated; wherein the key indicators include at least the loss function value and the learning rate curve after the current round of iteration;

[0009] After calculating the difference each time, determining whether the difference exceeds a preset threshold;

[0010] If so, an abnormal accuracy warning of the target model is issued and the training of the target model is stopped;

[0011] If not, continue training the target model until the preset training termination condition is reached.

[0012] Optionally, determining the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster specifically includes:

[0013] Determine an average error between the loss function value of the heterogeneous cluster and the loss function value of the homogeneous reference cluster, an instantaneous error of the loss function value in the current iteration, and a first similarity of a learning rate curve from the start of training to the current iteration;

[0014] Determining whether the difference exceeds a preset threshold specifically includes:

[0015] Determine a first threshold value according to the number of iteration steps corresponding to the round of training;

[0016] When at least one of the following conditions exists: the average error is greater than the first threshold, the instantaneous error is greater than the second threshold, and the first similarity is less than a third threshold, it is determined that the difference exceeds the preset threshold; wherein the first threshold monotonically decreases as the number of iteration steps increases.

[0017] Optionally, the key indicator also includes a gradient norm;

[0018] The method further comprises:

[0019] Determine whether the gradient norm is within a preset range; if not, issue an abnormal target model accuracy warning.

[0020] Optionally, the key indicator also includes a validation set loss function value;

[0021] The method further comprises:

[0022] Sampling the validation set loss function values ​​of the heterogeneous cluster and the homogeneous benchmark cluster at preset time intervals, and calculating the average error of the validation set loss function values;

[0023] For each sampling of the validation set loss function value, determine the number of iteration steps corresponding to the sampling moment;

[0024] A corresponding first threshold is determined according to the number of iteration steps, and when the average error of the validation set loss function value is greater than the first threshold, an abnormal accuracy alarm of the target model is issued.

[0025] Optionally, the key indicator also includes a model weight of the target model after this round of iteration;

[0026] The method further comprises:

[0027] determining a weight error between the model weights of the heterogeneous cluster and the model weights of the homogeneous benchmark cluster;

[0028] Determining an average weight error between the heterogeneous cluster and the homogeneous reference cluster according to the number of sampling times, the number of parameters of the target model, and each weight error;

[0029] When the average weight error is greater than a fourth threshold, an abnormal target model accuracy alarm is issued.

[0030] Optionally, after a preset training termination condition is reached, the method further includes:

[0031] Determining a validation set loss function value change curve of the heterogeneous cluster according to the stored key indicators of the heterogeneous cluster;

[0032] According to the key indicators of the homogeneous benchmark cluster, a validation set loss function change curve of the homogeneous cluster is determined;

[0033] Generate a verification report for the target model based on the determined verification set loss function change curve and the training configuration of the target model.

[0034] Optionally, after a preset training termination condition is reached, the method further includes:

[0035] Utilizing the target model trained by the heterogeneous cluster and the target model trained by the homogeneous reference cluster, respectively, to execute at least one task to be executed, and determine a first output and a second output;

[0036] Calculating a second similarity between the first output and the second output, and issuing an abnormal accuracy alarm of the target model when the second similarity is less than a fifth threshold;

[0037] When the second similarity is less than a fifth threshold, a verification report of the target model is generated according to the second similarity and the training configuration of the target model.

[0038] This specification provides a model accuracy verification device, including:

[0039] The training module is used to train the target model using a unified training configuration in the heterogeneous cluster and the homogeneous benchmark cluster respectively;

[0040] a sampling module configured to store, at a preset number of iteration steps, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster during the current iteration, and calculate the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster; wherein the key indicators include at least the loss function value and the learning rate curve after the current iteration;

[0041] The judgment module is used to judge whether the difference exceeds a preset threshold after each calculation of the difference; if so, an abnormal target model accuracy alarm is issued and the training of the target model is stopped; if not, the target model is continued to be trained until the preset training termination condition is reached.

[0042] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model accuracy verification method.

[0043] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model accuracy verification method when executing the program.

[0044] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0045] In a model training method for heterogeneous clusters provided in this specification, a unified training configuration is used to train the target model in the heterogeneous cluster and the homogeneous benchmark cluster respectively. Every preset number of iteration steps, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are stored in the round of iteration. For each sampling, after storing the key indicators of the current round of iteration, the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is determined, and whether to continue training the target model is determined based on the difference.

[0046] In the above method, during the model training process based on heterogeneous clusters, the accuracy of the model trained based on the homogeneous cluster with the model trained based on the heterogeneous cluster using the same training configuration is compared with the accuracy of the model trained based on the heterogeneous cluster at a preset frequency, thereby monitoring the model training process in real time, promptly discovering abnormal situations in the model training process, and improving the accuracy of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0048] Figure 1A flowchart of a model training method for heterogeneous clusters provided in this specification;

[0049] Figure 2 A schematic diagram of a verification report provided for this instruction manual;

[0050] Figure 3 A schematic diagram of another verification report provided for this instruction manual;

[0051] Figure 4 A schematic diagram of a model training device for heterogeneous clusters provided in this specification;

[0052] Figure 5 This manual provides a corresponding Figure 1 Schematic diagram of the structure of the electronic equipment. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0054] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0055] Figure 1 A flowchart of a large model pre-training method provided in an embodiment of this specification includes the following steps:

[0056] S100: Train the target model using a unified training configuration in both the heterogeneous cluster and the homogeneous benchmark cluster.

[0057] In one or more embodiments of this specification, there is no restriction on the specific device by which the model training method is performed. It can be performed by a node in the heterogeneous cluster, or each node in the heterogeneous cluster can perform each step separately, or it can be performed by a device outside the heterogeneous cluster. This specification does not impose any restrictions on this. It also does not limit the specific device that each node is composed of, such as a mobile terminal and a server, etc. However, since the subsequent steps involve model training, accuracy detection and other steps, these steps are generally performed by the server. Therefore, this specification will also use the server as an example of a node device in the heterogeneous cluster for executing the large model pre-training method for description. The server can be a single device, or composed of multiple devices, such as a distributed server, a cloud service server, etc. This specification does not impose any restrictions on this.

[0058] With the rapid growth of computer technology, the number of model parameters and the scale of training data have increased rapidly, and the demand for computing resources has skyrocketed. Distributed training based on heterogeneous clusters has become an important solution. However, issues such as hardware differences, communication latency, and load balancing in heterogeneous environments can alter model behavior. To monitor potential interference introduced by training models in heterogeneous environments, the server can simultaneously train the target model in a heterogeneous cluster using the same training configuration, allowing for subsequent verification of the accuracy of the target model trained on the heterogeneous cluster.

[0059] Specifically, after the server determines the heterogeneous cluster and homogeneous benchmark cluster, it uses the same training configuration to train the target model on both the heterogeneous cluster and the homogeneous benchmark cluster. A heterogeneous cluster refers to a distributed computing system composed of computing nodes of different types, architectures, or performance, designed to achieve an optimal balance of cost, efficiency, or functionality through flexible resource combination. A homogeneous cluster refers to a distributed computing system composed of the same or similar hardware devices and software environments. A homogeneous benchmark cluster refers to the homogeneous cluster determined by the server from among the various homogeneous clusters used for training the target model.

[0060] It should be noted that in one or more embodiments of this specification, a unified training configuration means that when the server trains the target model, the software framework layer, hardware layer, and data layer are unified. For example, with respect to the hardware layer, the network configuration and storage device type of the homogeneous benchmark cluster and the heterogeneous cluster are the same. With respect to the software framework layer, the software framework version number, the version number of the dependent library, etc. of the homogeneous benchmark cluster and the heterogeneous cluster are the same. With respect to the data layer, the data preprocessing method, data reading order, etc. of the homogeneous benchmark cluster and the heterogeneous cluster are the same. Among them, data refers to the data set used for model training.

[0061] S102: Every preset number of iteration steps, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are stored after the current round of iteration process, and the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is calculated; wherein the key indicators include at least the loss function value and the learning rate curve after the current round of iteration process.

[0062] There are certain differences in the hardware of each node in a heterogeneous cluster. These differences often lead to dynamic instability in the heterogeneous environment. An unstable heterogeneous environment may lead to accuracy oscillation, excessive learning rate, and the risk of iteration terminal during model training. In order to prevent error accumulation and possible iteration terminal problems when training the target model based on a heterogeneous cluster, the server can store the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster after the round of iteration process at a preset number of iteration steps, so that the training accuracy of the target model based on heterogeneous cluster training can be analyzed by comparing the differences between the key indicators of the homogeneous benchmark cluster and the key indicators of the heterogeneous cluster at each preset number of iteration steps.

[0063] Specifically, the server stores the key metrics of the heterogeneous cluster and the homogeneous benchmark cluster during each iteration at a preset number of steps. The key metrics include at least the loss function value and learning rate curve after the current iteration. The server then calculates the difference between the stored key metrics of the heterogeneous cluster and the key metrics of the homogeneous benchmark cluster.

[0064] It should be noted that in one or more embodiments of this specification, there is no limit on the preset number of iteration steps set by the server. A fixed value can be selected, such as storing the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster once after every 500 iterations. The preset number of iteration steps can also be shortened round by round as the training stage of the target model progresses. It can be set according to actual needs.

[0065] In addition, there is no restriction on the specific method used by the server to determine the difference between the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster.

[0066] For example, the server can determine the difference between the loss function value of the heterogeneous cluster of the current iterative process and the loss function value of the homogeneous benchmark cluster as the instantaneous error of the loss function value, and determine the first similarity of the learning rate curve. The determined instantaneous error and first similarity are used as the difference. Of course, in order to further monitor the difference between the key indicators of the target model trained based on the heterogeneous cluster and the key indicators of the target model trained based on the homogeneous benchmark cluster, the server can also determine the average error between the loss function value of the heterogeneous cluster and the loss function value of the homogeneous benchmark cluster from the start of training to the current round of iteration, the instantaneous error of the loss function value of the current iterative process, and the first similarity of the learning rate curve when determining the difference. Since there are many optional methods, this manual does not limit this and can be set according to actual needs.

[0067] S104: After calculating the difference each time, determine whether the difference exceeds a preset threshold. If so, execute step S106; if not, execute step S108.

[0068] After determining the difference, the server can judge whether the difference exceeds a preset threshold based on the determined difference, so as to determine whether the difference between the target model trained based on the heterogeneous cluster and the target model trained based on the homogeneous benchmark cluster is too large when the target model is trained to the current iteration process, thereby realizing accuracy monitoring during the target model training process.

[0069] It should be noted that in one or more embodiments of this specification, there is no limit on the preset threshold value, nor on the method for determining the preset threshold value, and the preset threshold value can be set according to actual needs.

[0070] For example, if the determined differences include the first similarity and the instantaneous error, a second threshold and a third threshold may be set, respectively. The second threshold and the third threshold may be set as two fixed values. If the determined differences include the aforementioned average error, instantaneous error, and the first similarity, the server may determine the first threshold, the second threshold, and the third threshold when setting the preset thresholds. The first threshold is also a fixed value. Since the instantaneous error of a model trained on a heterogeneous cluster fluctuates significantly, the first threshold is generally not greater than the second threshold.

[0071] In addition, to further improve the accuracy of the target model for heterogeneous cluster training, the server can also dynamically adjust the first threshold, that is, the first threshold decreases monotonically as the number of iterations increases. Then, when determining whether the difference exceeds the preset threshold, the server first determines the first threshold based on the number of iterations corresponding to the training round.

[0072] Of course, the model training phase can also be divided. If the target model is trained for a total of T iterations, the initial training phase is [0, 0.1T], the mid-training phase is [0.1T, 0.8T], and the final training phase is [0.8T, T]. The corresponding first thresholds are 1%, 0.5%, and 0.3%, respectively. To determine whether the difference exceeds the preset threshold, the number of iterations corresponding to the training phase can be determined first, and then the training phase corresponding to that number of iterations can be determined. The first threshold can then be determined based on the training phase.

[0073] When at least one of the following conditions exists: the average error is greater than the first threshold, the instantaneous error is greater than the second threshold, and the first similarity is less than the third threshold, it is determined that the difference exceeds the preset threshold and step S10 is executed; if not, step S108 is executed.

[0074] S106: issuing an abnormal accuracy alarm for the target model and stopping training the target model.

[0075] When the difference exceeds the preset threshold, it means that the accuracy of the target model trained based on the heterogeneous cluster does not meet the training target of the model. The server can return an alarm of abnormal accuracy of the target model to the user or operation and maintenance personnel and stop training the target model.

[0076] Alternatively, an exception alert can be issued when the difference exceeds a preset threshold a certain number of times in a row. If the subsequent accuracy verification still shows a difference exceeding the preset threshold, training of the target model can be stopped. For example, if the difference exceeds the preset threshold for three consecutive iterations, a level 1 alert is triggered, meaning only logging is performed. If the cumulative number of alerts exceeds five, a level 2 alert is triggered, pausing training. This setting can be customized based on actual needs and is not restricted in this manual.

[0077] S108: Continue training the target model until a preset training termination condition is reached.

[0078] When the difference does not exceed the preset threshold, it can be determined that the model accuracy of the target model based on heterogeneous cluster training meets the training requirements, and training can continue. Of course, in one or more embodiments of this specification, the specific content of the training termination condition of the target model is not limited and can be set according to actual needs.

[0079] based on Figure 1 In the model training method for heterogeneous clusters shown, a unified training configuration is used to train the target model in the heterogeneous cluster and the homogeneous benchmark cluster respectively. The key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster in the round of iteration are stored every preset number of iteration steps. For each sampling, after storing the key indicators of the current round of iteration, the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is determined, and whether to continue training the target model is determined based on the difference.

[0080] In the above method, during the model training process based on heterogeneous clusters, the accuracy of the model trained based on the homogeneous cluster with the model trained based on the heterogeneous cluster using the same training configuration is compared with the accuracy of the model trained based on the heterogeneous cluster at a preset frequency, thereby monitoring the model training process in real time, promptly discovering abnormal situations in the model training process, and improving the accuracy of the trained model.

[0081] In addition, before starting to train the target model, the server can also compare the parameters of the homogeneous benchmark cluster and the heterogeneous cluster to eliminate the phenomenon that the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are significantly different due to different parameters in the subsequent real-time accuracy monitoring process.

[0082] Specifically, the server can first verify the distributed environment, such as checking the size of the MPI communication group and the consistency of the NCCL / RCCL communication library version. Through random control verification, compare the random seeds and shard indexes of the digital loader (Dataloader). By determining whether the dropout mechanism is closed, the differences in key indicators between homogeneous benchmark clusters and heterogeneous clusters caused by different software frameworks can be eliminated. Alternatively, single test verification can be performed on operators that may cause accuracy differences to determine whether there are differences in the accuracy of the underlying operators. Since there are many methods to choose from, this manual does not list them one by one. You can set them according to actual needs.

[0083] In addition, since the model has many key indicators, when the server determines the key indicators of the homogeneous benchmark cluster and the key indicators of the heterogeneous cluster, the key indicators can also include the gradient norm. When determining whether the difference exceeds the preset threshold, the server can also determine whether the gradient norm is within the preset range. If it is within the preset range, execute step S108; if not, execute step S106.

[0084] In addition, the loss function value determined above generally refers to the loss function value of the training set. The server can also determine the loss function value of the validation set when verifying the accuracy of the target model through the validation set. When determining whether the difference exceeds the preset threshold, the server can also sample the validation set loss function values ​​of the heterogeneous cluster and the homogeneous benchmark cluster at a preset time interval, and calculate the average error of the validation set loss function value. Then, for each sampling of the validation set loss function value, determine the number of iteration steps corresponding to the sampling moment, and determine the corresponding first threshold value based on the number of iteration steps. When the average error of the validation set loss function value is greater than the first threshold, execute step S106. When the average error of the validation set loss function value is not greater than the first threshold, execute step S108.

[0085] Furthermore, the key indicator may also include the model weight of the target model after this round of iteration. When determining whether the difference exceeds a preset threshold, the server may also determine the weight error between the model weight of the heterogeneous cluster and the model weight of the homogeneous reference cluster, and then determine the average weight error between the heterogeneous cluster and the homogeneous reference cluster based on the number of sampling times, the number of parameters of the target model, and the weight errors. When the average weight error is greater than a fourth threshold, step S106 is executed; when the average weight error is not greater than the fourth threshold, step S108 is executed.

[0086] In addition, although the loss value and the actual performance of the model have a certain correlation, the relationship is not absolute. In order to prevent the situation where the difference does not exceed the preset threshold, but there are large differences in the task execution results when the model performs a specific task, the server can also use the target model trained by the heterogeneous cluster and the target model trained by the homogeneous benchmark cluster to respectively execute at least one task to be executed, and then compare the second similarity between the outputs of the two models. The second similarity can represent the second similarity of the total output, or it can represent the second similarity of the output results for each task. This specification does not limit this. Of course, there is no limit on the specific way to calculate the second similarity. It can be cosine similarity, vector similarity, semantic similarity, etc. It can be set according to actual needs. When the second similarity is less than the fifth threshold, an abnormal accuracy alarm of the target model is issued.

[0087] It should be noted that the fourth threshold and the fifth threshold mentioned above are the same as the second threshold and the third threshold mentioned above, and can be set by the server according to actual needs, and this specification does not impose any restrictions on this.

[0088] In addition, the server can also uniformly run the target model trained by the heterogeneous cluster and the target model trained by the homogeneous benchmark cluster on the homogeneous cluster, and then use the open source evaluation framework to perform the evaluation tasks separately, such as the OpenCompass open source framework. Of course, in one or more embodiments of this specification, there is no limit on what specific tasks each target model performs, and it can be determined according to the specific purpose of the target model. For example, when the target model is a large model, it can perform Massive Multitask Language Understanding (MMLU), Truthful QA, Natural Questions (NQ), etc. separately. Then, the score difference rate of the target model trained by the heterogeneous cluster and the target model trained by the homogeneous benchmark cluster is calculated. If the difference is greater than 5% for more than three tasks, it is determined that the training accuracy does not meet the standard.

[0089] Furthermore, to verify the accuracy of the trained target model, the server can also determine the validation set loss function value change curve of the heterogeneous cluster based on the stored key indicators of the heterogeneous cluster after reaching the preset training termination condition, and determine the validation set loss function change curve of the homogeneous cluster based on the key indicators of the homogeneous benchmark cluster. Based on the determined validation set loss function change curve and the training configuration of the target model, a verification report of the target model is generated. This allows users to better analyze the accuracy changes of the target model during the training process based on the verification report. Figure 2 and Figure 3 As shown, Figure 2This is a schematic diagram of a verification report provided for this instruction manual. Figure 3 This is a schematic diagram of another verification report provided in this manual, where "A" to "L" refer to the tasks performed by the target model, and the broken line represents the score after performing the corresponding task.

[0090] It should be noted that in one or more embodiments of this specification, the specific form of the verification report is not limited. Corresponding comparison charts can be generated according to the types of key indicators obtained by the server as verification reports, such as evaluation task score comparison charts, trend curve comparison charts of training loss and verification accuracy loss, etc., which can be set according to actual needs.

[0091] Based on the same idea as the large model pre-training method provided in one or more embodiments of this specification, this specification also provides a corresponding large model pre-training device, such as Figure 4 shown.

[0092] Figure 4 A schematic diagram of a model training device for heterogeneous clusters provided in this specification, specifically including:

[0093] A training module 400 is configured to train a target model using a unified training configuration in a heterogeneous cluster and a homogeneous reference cluster.

[0094] Sampling module 401 is configured to store the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster after each predetermined number of iterations, and calculate the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster; wherein the key indicators include at least the loss function value and the learning rate curve after the current iteration;

[0095] The judgment module 402 is used to judge whether the difference exceeds a preset threshold after each calculation of the difference; if so, an abnormal target model accuracy alarm is issued and the training of the target model is stopped; if not, the target model is continued to be trained until the preset training termination condition is reached.

[0096] Optionally, the sampling module 401 is used to determine the average error between the loss function value of the heterogeneous cluster and the loss function value of the homogeneous benchmark cluster, the instantaneous error of the loss function value of the current iteration process, and the first similarity of the learning rate curve from the start of training to the current round of iteration; the judgment module 402 is used to determine a first threshold based on the number of iteration steps corresponding to this round of training; when there is at least one of the conditions that the average error is greater than the first threshold, the instantaneous error is greater than the second threshold, and the first similarity is less than the third threshold, it is determined that the difference exceeds the preset threshold; wherein, the first threshold monotonically decreases with the increase of the number of iteration steps.

[0097] Optionally, the judgment module 402 is used to judge whether the gradient norm is within a preset range, and if not, issue an abnormal target model accuracy alarm.

[0098] Optionally, the key indicators also include a validation set loss function value; the judgment module 402 is used to sample the validation set loss function values ​​of the heterogeneous cluster and the homogeneous benchmark cluster at preset time intervals, and calculate the average error of the validation set loss function values; for each sampling of the validation set loss function value, determine the number of iteration steps corresponding to the sampling moment; determine the corresponding first threshold value based on the number of iteration steps, and when the average error of the validation set loss function value is greater than the first threshold value, issue an abnormal accuracy alarm for the target model.

[0099] Optionally, the key indicator also includes the model weight of the target model after this round of iteration process; the judgment module 402 is used to determine the weight error between the model weight of the heterogeneous cluster and the model weight of the homogeneous benchmark cluster; based on the number of sampling times, the parameter amount of the target model and each weight error, determine the average weight error between the heterogeneous cluster and the homogeneous benchmark cluster; when the average weight error is greater than a fourth threshold, issue an abnormal target model accuracy alarm.

[0100] Optionally, the device also includes a generation module 403, which is used to determine the validation set loss function value change curve of the heterogeneous cluster based on the stored key indicators of the heterogeneous cluster after reaching the preset training termination condition; determine the validation set loss function change curve of the homogeneous cluster based on the key indicators of the homogeneous benchmark cluster; and generate a verification report of the target model based on the determined validation set loss function change curve and the training configuration of the target model.

[0101] Optionally, the generation module 403 is used to use the target model trained by the heterogeneous cluster and the target model trained by the homogeneous benchmark cluster to respectively execute at least one task to be executed and determine the first output and the second output; calculate the second similarity between the first output and the second output, and when the second similarity is less than a fifth threshold, issue an abnormality alarm for the target model accuracy; when the second similarity is less than the fifth threshold, generate a verification report for the target model based on the second similarity and the training configuration of the target model.

[0102] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a model training method for heterogeneous clusters.

[0103] This manual also provides Figure 5 The one shown corresponds to Figure 1 Schematic diagram of the structure of the electronic equipment. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 A model training method for heterogeneous clusters.

[0104] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0105] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a Hardware Description Language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that simply by programming a method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0106] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0107] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0108] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0109] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0115] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0117] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0119] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0120] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A model training method for heterogeneous clusters, characterized in that: include: Train the target model using a unified training configuration in both the heterogeneous cluster and the homogeneous benchmark cluster. At every preset number of iterations, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster are stored after the current round of iteration, and the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster is calculated; wherein the key indicators include at least the loss function value and the learning rate curve after the current round of iteration; After calculating the difference each time, determining whether the difference exceeds a preset threshold; If so, an abnormal accuracy warning of the target model is issued and the training of the target model is stopped; If not, continue training the target model until the preset training termination condition is reached.

2. The method according to claim 1, wherein Determining the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster specifically includes: Determine an average error between the loss function value of the heterogeneous cluster and the loss function value of the homogeneous reference cluster, an instantaneous error of the loss function value in the current iteration, and a first similarity of a learning rate curve from the start of training to the current iteration; Determining whether the difference exceeds a preset threshold specifically includes: Determine a first threshold value according to the number of iteration steps corresponding to the round of training; When at least one of the following conditions exists: the average error is greater than the first threshold, the instantaneous error is greater than the second threshold, and the first similarity is less than a third threshold, it is determined that the difference exceeds the preset threshold; wherein the first threshold monotonically decreases as the number of iteration steps increases.

3. The method according to claim 2, wherein The key indicators also include gradient norm; The method further comprises: Determine whether the gradient norm is within a preset range; if not, issue an abnormal target model accuracy warning.

4. The method according to claim 2, wherein The key indicators also include the validation set loss function value; The method further comprises: Sampling the validation set loss function values ​​of the heterogeneous cluster and the homogeneous benchmark cluster at preset time intervals, and calculating the average error of the validation set loss function values; For each sampling of the validation set loss function value, determine the number of iteration steps corresponding to the sampling moment; A corresponding first threshold is determined according to the number of iteration steps, and when the average error of the validation set loss function value is greater than the first threshold, an abnormal accuracy alarm of the target model is issued.

5. The method according to claim 1, wherein The key indicator also includes the model weight of the target model after this round of iteration; The method further comprises: determining a weight error between the model weights of the heterogeneous cluster and the model weights of the homogeneous benchmark cluster; Determining an average weight error between the heterogeneous cluster and the homogeneous reference cluster according to the number of sampling times, the number of parameters of the target model, and each weight error; When the average weight error is greater than a fourth threshold, an abnormal target model accuracy alarm is issued.

6. The method according to claim 1, wherein After the preset training termination condition is reached, the method further includes: Determining a validation set loss function value change curve of the heterogeneous cluster according to the stored key indicators of the heterogeneous cluster; Determining a validation set loss function change curve of the homogeneous benchmark cluster according to the key indicators of the homogeneous benchmark cluster; A validation report for the target model is generated according to the validation set loss function value change curve of the heterogeneous cluster, the validation set loss function change curve of the homogeneous benchmark cluster, and the training configuration of the target model.

7. The method according to claim 1, wherein After the preset training termination condition is reached, the method further includes: Utilizing the target model trained by the heterogeneous cluster and the target model trained by the homogeneous reference cluster, respectively, to execute at least one task to be executed, and determine a first output and a second output; Calculating a second similarity between the first output and the second output, and issuing an abnormal accuracy alarm of the target model when the second similarity is less than a fifth threshold; When the second similarity is less than a fifth threshold, a verification report of the target model is generated according to the second similarity and the training configuration of the target model.

8. A model training device for heterogeneous clusters, characterized in that: include: The training module is used to train the target model using a unified training configuration in the heterogeneous cluster and the homogeneous benchmark cluster respectively; a sampling module configured to store, at a preset number of iteration steps, the key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster after the current round of iteration, and calculate the difference between the stored key indicators of the heterogeneous cluster and the key indicators of the homogeneous benchmark cluster; wherein the key indicators include at least the loss function value and the learning rate curve after the current round of iteration; The judgment module is used to judge whether the difference exceeds a preset threshold after each calculation of the difference; if so, an abnormal target model accuracy alarm is issued and the training of the target model is stopped; if not, the target model is continued to be trained until the preset training termination condition is reached.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

Images