Evaluation method, device, system and equipment for model training and storage medium

Abstract

The embodiment of the invention discloses a model training evaluation method, device and system, equipment and a storage medium. The method comprises the steps of obtaining an evaluation task description file of model training; performing task scheduling and model training triggering on a model training platform according to model information, algorithm information and evaluation task information in the evaluation task description file; obtaining scheduling log data and training log data when the model training platform performs task scheduling and model training; and associating the scheduling log data with the training log data, and performing model training evaluation according to the associated scheduling log data and training log data to obtain an evaluation result. According to the method, the performance in the model training process can be actively evaluated by actively injecting the evaluation task information in the model training without waiting for completion of the model training, so that the evaluation period can be shortened, and the evaluation rate and diversity can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an evaluation method, device, system, equipment and storage medium for model training. Background Art

[0002] With the development of artificial intelligence technology, the number of parameters and training set size of network models are gradually increasing. The scale of intelligent accelerator clusters has also reached the level of 10,000 cards. Network model training also includes complex training scenarios, such as hybrid intelligent accelerator scenarios. Evaluating the model training effect on the intelligent accelerator cluster is a problem that must be solved in network model optimization.

[0003] Traditional network model training optimization uses model segmentation, network tuning, and fault tolerance. When evaluating model training, it is often necessary to wait for an abnormality to occur in the model training environment, or to wait for the model training to be completed before performing performance evaluation based on model parameters. Traditional evaluation methods result in a long model training evaluation cycle and limited evaluation dimensions.

[0004] In addition, traditional model training and evaluation methods are all processed manually or with simple scripts, and the operations are relatively scattered. When the cluster scale of the model training platform increases, the operation becomes very inconvenient, resulting in time-consuming and costly model training and evaluation. Summary of the invention

[0005] The present invention provides a model training evaluation method, device, system, equipment and storage medium to actively evaluate the performance of the model training process, shorten the evaluation cycle, and improve the evaluation rate and diversity.

[0006] According to one aspect of the present invention, a method for evaluating model training is provided, the method comprising:

[0007] Get the evaluation task description file for model training;

[0008] Performing task scheduling and model training triggering on the model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file;

[0009] Obtaining scheduling log data and training log data of the model training platform when performing task scheduling and model training;

[0010] The scheduling log data and the training log data are associated, and model training evaluation is performed based on the associated scheduling log data and training log data to obtain an evaluation result.

[0011] According to another aspect of the present invention, there is provided a model training evaluation device, the device comprising:

[0012] An evaluation task description file acquisition module is used to obtain the evaluation task description file for model training;

[0013] A model training trigger module is used to schedule tasks and trigger model training on the model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file;

[0014] A log data acquisition module is used to acquire scheduling log data and training log data of the model training platform when performing task scheduling and model training;

[0015] The evaluation result determination module is used to associate the scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

[0016] According to another aspect of the present invention, a model training evaluation system is provided, the system comprising: an evaluation task configurator, an incentive tool, a log collector, and a data analyzer; wherein:

[0017] Evaluation task configurator, used to configure the evaluation task description file for generating model training;

[0018] The incentive tool is connected to the evaluation task configurator and connected to the model training platform;

[0019] An incentive tool, used to schedule tasks on the model training platform and trigger model training according to the model information, algorithm information, and evaluation task information obtained in the evaluation task description file;

[0020] The log collector is connected to the model training platform to obtain the scheduling log data and training log data of the model training platform when performing task scheduling and model training;

[0021] The data analyzer is connected to the log collector and is used to associate the acquired scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the evaluation method of model training described in any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the evaluation method for model training described in any embodiment of the present invention when executed.

[0027] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the evaluation method for model training described in any embodiment of the present invention.

[0028] The technical solution of the embodiment of the present invention obtains an evaluation task description file for model training; performs task scheduling and model training triggering on a model training platform according to model information, algorithm information and evaluation task information in the evaluation task description file; obtains scheduling log data and training log data of the model training platform when performing task scheduling and model training; associates the scheduling log data with the training log data, and performs model training evaluation according to the associated scheduling log data and training log data to obtain an evaluation result, thereby solving the problem of performance evaluation of a network model during training. By actively injecting evaluation task information into model training, the performance of the model training process can be actively evaluated without waiting for model training to be completed, thereby shortening the evaluation cycle and improving the evaluation rate and diversity.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flow chart of a model training evaluation method provided according to Embodiment 1 of the present invention;

[0032] Figure 2 is a flowchart of generating an evaluation task workflow from an evaluation task description file according to Embodiment 1 of the present invention;

[0033] Figure 3 is a structural diagram of an evaluation system for model training provided according to Embodiment 2 of the present invention;

[0034] Figure 4 is an interactive schematic diagram of an evaluation system for model training provided according to Embodiment 2 of the present invention;

[0035] Figure 5 is a structural schematic diagram of an evaluation device for model training provided according to Embodiment 3 of the present invention;

[0036] Figure 6 It is a schematic diagram of the structure of an electronic device for implementing the evaluation method of model training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment 1

[0040] Figure 1This is a flowchart of a model training evaluation method provided according to the first embodiment of the present invention. This embodiment is applicable to the situation where the performance evaluation of the model such as automatic expansion and contraction or fault injection is performed on the model when the network model is trained through the model training platform. The method can be executed by the model training evaluation device, which can be implemented in the form of hardware and / or software. The model training evaluation device can be configured in an electronic device, which can be a mobile device such as a mobile phone, a tablet computer (PAD), a wearable device, or a personal computer (PC, Personal Computer). Figure 1 As shown, the method includes:

[0041] Step 110: Obtain the evaluation task description file for model training.

[0042] In an embodiment of the present invention, the model training platform may be a tested platform. The model training platform may include model training task files, model files, model training scheduling systems, and intelligent accelerator cluster resources. Model training task files and model files are used to determine the model structure and algorithm information for model training. The model training scheduling system is used to schedule and manage resources in model training. Intelligent accelerator cluster resources may be resources for training acceleration in model training. For example, intelligent accelerator cluster resources may be a graphics processing unit (Graphics Processing Unit Cluster, GPU) cluster, a neural network processing unit (Neural Processing Unit Cluster, NPU) cluster, a general computing task processor (General Compute Unit Cluster, GCU) cluster, or an accelerator or coprocessor (XPU, X represents a different processor architecture) cluster.

[0043] In order to actively inject cluster errors or scale up and down into model training, and then perform performance evaluation on model training, you can pre-configure an evaluation task description file. The model training platform determines the model structure and algorithm information according to the evaluation task description file for model training, and determines the resource scheduling information in model training according to the evaluation task description file.

[0044] Among them, the evaluation task description file may include model information, algorithm information and evaluation task information. Specifically, the model information may be the model file path of the training model. The algorithm information may include the algorithm file path and the startup command. The evaluation task information may include evaluation scenario information, evaluation resource information, and evaluation node scale information. The evaluation scenario information may include evaluation indicator information for model training. The evaluation resource information may include accelerator resource information used for model training. The evaluation node scale information may include the number of nodes in model training. The evaluation task information may also include: the evaluation task name, the model training platform engine, and the image path of the operating environment, etc.

[0045] Table 1 is an exemplary evaluation task description file. In Table 1, the evaluation task name is: GCU AI accelerator cluster status evaluation, the evaluation scenario is: model training scenario 1 to 4 indicators, the algorithm file path is: / opt / megatron_lm / train.gz, the startup command is: python-mtorch.distributed.run train.py, the image path of the operating environment is: megatron_lm:lastest, the model file path is: / opt / megatron_lm / checkpoint.pt, the training platform engine is: DC1_AZ1_K8S_GCU_Platform_Engine, the resources involved in the evaluation are: {cpu:xx, memMB:xx, accelerate:gcu}, and the node scale involved in the evaluation is: [min_nnodes:]nnodes*nproc_per_node.

[0046] Table 1

[0047]

[0048]

[0049] Step 120: Perform task scheduling and model training triggering on the model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file.

[0050] According to the model information, algorithm information and evaluation task information specified in the evaluation task description file, the data serialization format (YAML Ain't Markup Language, YAML) file required by the model training platform can be generated, so that the model training platform can perform model training under the specified information according to the YAML file to evaluate the model training.

[0051] In an embodiment of the present invention, the evaluation task information includes: evaluation scenario information, evaluation resource information, and evaluation node scale information; optionally, the model training platform is task scheduled and model training is triggered according to the model information, algorithm information and evaluation task information in the evaluation task description file, including: sending the model information and algorithm information in the evaluation task description file to the model training platform, so that the model training platform performs model training according to the model information and algorithm information; determining the corresponding evaluation task workflow according to the evaluation scenario information; sending the evaluation task workflow, evaluation resource information and evaluation node scale information to the model training platform, so that the model training platform performs model training according to the evaluation task information.

[0052] Among them, the model training platform can train the specified model architecture under the specified algorithm according to the model information and algorithm information. Different evaluation task workflows can be pre-formulated for different evaluation scenarios. The model training platform can perform model training according to the evaluation task workflow corresponding to the evaluation scenario information. Combined with the evaluation task workflow, evaluation resource information, and evaluation node scale information, the model can be specifically trained.

[0053] Figure 2 1 is a flowchart of generating an evaluation task workflow from an evaluation task description file according to the first embodiment of the present invention. Figure 2 As shown, for the evaluation task description file, you can first determine whether the parameters are legal; when the parameters are legal, you can generate an evaluation task identifier (such as ID); analyze the evaluation scenario information in the evaluation task and generate the corresponding workflow; according to the workflow description, you can generate the YAML file required by the model training platform; call the model training platform application programming interface (API) to upload the model training task, wait for the model training task to be completed, and when the workflow is completed, you can complete this evaluation.

[0054] Optionally, the evaluation scenario information includes at least one of the following: a linear acceleration ratio evaluation scenario for model training, a computing power utilization evaluation scenario for model training, a fault tolerance capability evaluation scenario for model training, and a scale benefit evaluation scenario for model training.

[0055] Among them, the linear speedup ratio can be that in parallel computing, when the number of processors or computing nodes is increased, the execution speed of model training can be linearly improved in proportion to the number of processors. In an embodiment of the present invention, the evaluation task workflow determined in the linear speedup ratio evaluation scenario can be: according to the node scale involved in the evaluation defined in the evaluation task description file, parse the input expression (such as [min_nnodes:]nnodes*nproc_per_node), obtain the node information required for the evaluation task, and perform node settings in the linear speedup ratio scenario according to the node information. For example, the minimum number of nodes is min_nnodes and the maximum number of nodes to be tested is nnodes. Taking min_nnodes as the base, increase it in multiples of 2 until nnodes is reached, and generate multiple YAML files required by the model training platform as the entry for the model training platform to start the training task.

[0056] Computing power utilization refers to the degree to which computing resources (such as CPU, GPU, TPU, etc.) are actually used over a period of time. In an embodiment of the present invention, the evaluation task workflow determined in the computing power utilization evaluation scenario can be: create multiple evaluation tasks, each evaluation task can generate a unique task ID, and in the generated YAML file, the task ID can be associated with the container in the scheduling system, so that the tasks generated by the model training platform can be associated with the task ID. Similarly, the log stream generated by the container in the scheduling system can be associated with the evaluation task, and then the task ID can be used as an index to associate and retrieve the indicators related to computing power utilization required for the entire evaluation process, and the computing power utilization under different models or input conditions can be evaluated.

[0057] Fault tolerance refers to the ability to continue to run correctly when the model training encounters abnormal situations such as hardware failure, software error or network problem. In an embodiment of the present invention, the evaluation task workflow determined in the fault tolerance evaluation scenario can be: generate a YAML file according to the number of nodes (nnodes) participating in the evaluation defined in the evaluation task description file, and notify the model training platform to start the model training task, and obtain the node list corresponding to the scheduling of this training task according to the obtained model training platform scheduling information. In the node list, a group of nodes are randomly selected (according to the power law of 2), and the selected nodes are powered on and off through the hardware management protocol within the specified time period until the corresponding proportion upper limit is reached. Through the evaluation of the entire process, the total time consumption and total computing amount of the model training platform are collected, and the evaluation results of the model training platform in terms of fault tolerance are determined.

[0058] In model training, economies of scale may refer to the phenomenon that as the amount of model training data, computing resources or model complexity increases, the unit training cost gradually decreases and the training efficiency and model performance are improved. In an embodiment of the present invention, the evaluation task workflow determined in the economies of scale evaluation scenario may be similar to the fault tolerance evaluation scenario. That is, multiple evaluation tasks are created to evaluate the changes in the model loss function (Loss) under different models or input conditions, and the results of multiple evaluation tasks are cross-compared to quickly discover the performance inflection point.

[0059] By determining the corresponding workflow under different evaluation scenario information, diversified performance evaluation can be achieved during the model training process. In the performance evaluation, active error injection can be performed based on the evaluation task description file, such as the fault tolerance evaluation scenario. This allows performance evaluation to be performed during the model training process without waiting for the model training to be completed, shortening the evaluation cycle and improving the evaluation rate and diversity.

[0060] Step 130: Obtain scheduling log data and training log data of the model training platform when performing task scheduling and model training.

[0061] During the model training process, the log stream of the model training platform can be extracted through the API interface or log collection tools. The log stream in the model training platform can be divided into two parts: one part is the scheduling log data corresponding to the task scheduling in the model training, and the other part is the training log data during the model training. Among them, the scheduling log data can be obtained by the scheduling container corresponding to the model training platform. The training log data can be obtained by data collection in the training container pulled up by the scheduling system in the model training platform.

[0062] In order to ensure that the collected log data is relevant to the model training performance evaluation, in an optional implementation of an embodiment of the present invention, the scheduling log data and training log data of the model training platform when performing task scheduling and model training are obtained, including: according to the preset data filtering rules in the evaluation task description file, the scheduling log data and training log data of the model training platform when performing task scheduling and model training are obtained.

[0063] The preset data filtering rules can be used to filter and clean the log data to extract useful information. For example, the scheduling log data and training log data of the model training platform during task scheduling and model training can be obtained through the preset regular expressions.

[0064] Optionally, the scheduling log data and training log data of the model training platform when performing task scheduling and model training are obtained, including: obtaining the evaluation task identification information, timestamp, task scheduling resource information, training initial parameters, and each iteration process record data of the model training platform when performing task scheduling and model training.

[0065] The acquired log data can be labeled, such as labeling each log data with data category, evaluation task identification information, and timestamp, etc. The log data can be stored, for example, the log data is stored in a time series database, and when the model training needs to be evaluated, the data can be extracted from the time series database for data analysis to obtain the evaluation results.

[0066] The initial training parameters may include the precision selected when training starts, the parallelism (data parallelism, tensor parallelism, pipeline parallelism, etc.), and the number of nodes involved in the training. The data recorded in each iteration process may be the time consumed in each iteration (iteration), batch size (batchsize), loss function (Loss), and the number of samples used in the training process.

[0067] Step 140: Associating the scheduling log data with the training log data, and performing model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

[0068] In the embodiment of the present invention, the model training process can be viewed in real time through the display page to perform real-time model training evaluation; the overall evaluation results can also be cross-compared with the previous evaluation results after the model training is completed. By extracting information based on the log stream and extracting and storing key information by timeline, cross-comparison and evaluation are facilitated.

[0069] In an optional implementation of the embodiment of the present invention, associating the scheduling log data with the training log data includes: associating the scheduling log data with the training log data according to the evaluation task identification information and the timestamp.

[0070] Logs are cleaned and filtered through rules, key information is extracted and labels are added (Labels), which are then stored in a time series database. Data analysis is displayed, and cross-comparison of historical indicators is supported to provide evaluation results. Without changing the original usage of the model training platform, the model training platform being evaluated can be seamlessly received. By real-time receiving the model training platform's own scheduling logs and the logs generated by the training tasks, the two can be associated by tasks and timelines, thereby comprehensively evaluating the functions and performance of the AI ​​accelerator cluster.

[0071] The technical solution of this embodiment obtains an evaluation task description file for model training; performs task scheduling and model training triggering on a model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file; obtains scheduling log data and training log data of the model training platform when performing task scheduling and model training; associates the scheduling log data with the training log data, and performs model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result, thereby solving the problem of performance evaluation of a network model during training. By actively injecting evaluation task information into model training, the performance of the model training process can be actively evaluated without waiting for model training to be completed, thereby shortening the evaluation cycle and improving the evaluation rate and diversity.

[0072] Embodiment 2

[0073] Figure 3 Schematic diagram of a model training evaluation system provided according to Embodiment 2 of the present invention. Figure 3 As shown, the system includes: an evaluation task configurator 310, a stimulus tool 320, a log collector 330, and a data analyzer 340; wherein:

[0074] The evaluation task configurator 310 is used to configure the evaluation task description file for generating model training. The evaluation task configurator 310 can upload and issue task files, model files, and evaluation tasks. Optionally, the evaluation task information includes: evaluation scenario information, evaluation resource information, and evaluation node scale information. The evaluation scenario information includes at least one of the following: a linear acceleration ratio evaluation scenario for model training, a computing power utilization evaluation scenario for model training, a fault tolerance evaluation scenario for model training, and a scale efficiency evaluation scenario for model training.

[0075] The incentive tool 320 is connected to the evaluation task configurator 310 and connected to the model training platform. The incentive tool 320 receives the evaluation task issued by the evaluation task configurator 310, and converts the evaluation task description file into the YAML file required by the workflow and model training platform. The incentive tool 320 is used to perform task scheduling and model training triggering on the model training platform according to the model information, algorithm information and evaluation task information in the acquired evaluation task description file. Optionally, the incentive tool 320 is specifically used to send the model information and algorithm information in the evaluation task description file to the model training platform, so that the model training platform performs model training according to the model information and algorithm information; determine the corresponding evaluation task workflow according to the evaluation scenario information; send the evaluation task workflow, evaluation resource information and evaluation node scale information to the model training platform, so that the model training platform performs model training according to the evaluation task information. The incentive tool 320 can start the evaluation task, using the task ID as the unique identifier, so that the scheduling system in the model training platform starts scheduling and starts the container environment required for model training.

[0076] The log collector 330 is connected to the model training platform to obtain the scheduling log data and training log data of the model training platform when performing task scheduling and model training. Figure 3 As shown, the evaluation system for model training also includes a filter and cleaner 350. Among them, the filter and cleaner 350 is used to filter and clean the scheduling log data and the training log data according to the preset data filtering rules in the evaluation task description file. According to the data filtering and cleaning of the filter and cleaner 350, the log collector 330 can obtain the scheduling log data and the training log data related to the model training evaluation. Optionally, the log collector 330 can obtain the evaluation task identification information, timestamp, task scheduling resource information, training initial parameters, and each iteration process record data of the model training platform when performing task scheduling and model training.

[0077] Optional, such as Figure 3 As shown, the evaluation system for model training also includes a time series database 360. The time series database 360 ​​is connected to the log collector 330 and is used to store scheduling log data and training log data.

[0078] The data analyzer 340 is connected to the log collector 330, and is used to associate the acquired scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result. Specifically, the data analyzer 340 is connected to the log collector 330 through the time series database 360. The scheduling log data and training log data collected by the log collector 330 can be stored in the time series database 360. When performing model training evaluation, the data analyzer 340 can obtain the scheduling log data and the training log data in the time series database 360, associate the scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result. Specifically, the data analyzer 340 processes the data, uses the task ID as a keyword, associates the scheduling status and the operation status during model training according to the timeline, and performs cross-comparison analysis on the large model operation indicators of the model training platform according to the task dimension or time dimension, and displays the evaluation results.

[0079] For example, the data analyzer 340 can evaluate the linear expansion of a single-type AI cluster infrastructure through a computing power utilization evaluation scenario. In the case of energy conservation and emission reduction, the data analyzer 340 can evaluate the overall usage of a hybrid cluster through a fault tolerance evaluation scenario. Based on the scaling law, the data analyzer 340 can consider cost, resource utilization, energy consumption, etc., give an evaluation to improve the efficiency of machine learning, and obtain the optimal solution for computing cost.

[0080] The technical solution of the embodiment of the present invention is that the evaluation system for model training can define multiple model training evaluation scenarios through incentive tools, such as scenarios including automatic scaling, fault injection, etc., and then trigger the model training process to understand the model training status in real time through the data analyzer; and avoid the situation where, when there is no evaluation system for model training, the user can only manually start the model training task, and only after the model training is completed can the model training results with different node scales and different parameters as input be recorded, and then evaluation can be performed.

[0081] Figure 4 FIG. 1 is an interactive schematic diagram of an evaluation system for model training according to Embodiment 2 of the present invention. Figure 4As shown, the interactive process of the evaluation task configurator, incentive tool, log collector, filter and cleaner, time series database and data analyzer in the evaluation system of model training can be: start the evaluation system of model training, and the evaluation task configurator performs function initialization; the filter and cleaner obtain the rule file, and the log collector starts the listening port to receive the log stream in real time; the user issues the evaluation task (the evaluation task is divided into multiple scenarios), and the evaluation task configurator notifies the incentive tool to analyze and process the evaluation task; the incentive tool decomposes the evaluation task into a model training task workflow diagram, and obtains the task file and model file from the evaluation task configurator, and submits the model training task in sequence through the API interface of the model training platform until the entire workflow is completed; during the operation of the model training task, the log collector receives the log stream information generated in the model training process in real time; the filter and cleaner filters and extracts useful information according to the rules (pre-set regular expressions) in real time, labels the information (Label) and stores it in the time series database; the data analyzer can obtain the associated data on the entire evaluation task timeline from the time series database; the user can view the training process in real time through the display page provided by the data analyzer, or cross-compare with the last evaluation result after the evaluation is completed.

[0082] Without changing the original usage of the model training platform, the embodiment of the present invention can associate the model training platform's own scheduling logs and the logs generated by the model training tasks in a timeline through the model training evaluation system, thereby comprehensively evaluating the functions and performance of the AI ​​accelerator cluster in various scenarios.

[0083] Embodiment 3

[0084] Figure 5 Schematic diagram of a model training evaluation device according to Embodiment 3 of the present invention. Figure 5 As shown, the device includes: an evaluation task description file acquisition module 510, a model training trigger module 520, a log data acquisition module 530 and an evaluation result determination module 540. Among them:

[0085] An evaluation task description file acquisition module 510 is used to acquire an evaluation task description file for model training;

[0086] A model training trigger module 520 is used to schedule tasks on the model training platform and trigger model training according to the model information, algorithm information and evaluation task information in the evaluation task description file;

[0087] The log data acquisition module 530 is used to acquire the scheduling log data and training log data of the model training platform when performing task scheduling and model training;

[0088] The evaluation result determination module 540 is used to associate the scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

[0089] Optionally, the evaluation task information includes: evaluation scenario information, evaluation resource information, and evaluation node scale information;

[0090] The model training trigger module 520 includes:

[0091] A first information sending unit is used to send the model information and algorithm information in the evaluation task description file to the model training platform, so that the model training platform performs model training according to the model information and algorithm information;

[0092] A workflow determination unit, used to determine a corresponding evaluation task workflow according to evaluation scenario information;

[0093] The second information sending unit is used to send the evaluation task workflow, evaluation resource information and evaluation node scale information to the model training platform, so that the model training platform performs model training according to the evaluation task information.

[0094] Optionally, the evaluation scenario information includes at least one of the following: a linear acceleration ratio evaluation scenario for model training, a computing power utilization evaluation scenario for model training, a fault tolerance capability evaluation scenario for model training, and a scale benefit evaluation scenario for model training.

[0095] Optionally, the log data acquisition module 530 includes:

[0096] The log data filtering unit is used to obtain the scheduling log data and training log data of the model training platform when performing task scheduling and model training according to the preset data filtering rules in the evaluation task description file.

[0097] Optionally, the log data acquisition module 530 includes:

[0098] The log data acquisition unit is used to obtain the evaluation task identification information, timestamp, resource information occupied by task scheduling, initial training parameters, and record data of each iterative process when the model training platform performs task scheduling and model training.

[0099] Optionally, the evaluation result determination module 540 includes:

[0100] The data association unit is used to associate the scheduling log data and the training log data according to the evaluation task identification information and the timestamp.

[0101] The model training evaluation device provided in the embodiment of the present invention can execute the model training evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0102] Embodiment 4

[0103] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the evaluation method of model training.

[0107] In some embodiments, the evaluation method of model training may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the evaluation method of model training described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the evaluation method of model training in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0113] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model training evaluation method, characterized in that: include: Get the evaluation task description file for model training; Performing task scheduling and model training triggering on the model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file; Obtaining scheduling log data and training log data of the model training platform when performing task scheduling and model training; The scheduling log data and the training log data are associated, and model training evaluation is performed based on the associated scheduling log data and training log data to obtain an evaluation result.

2. The method according to claim 1, characterized in that: The evaluation task information includes: evaluation scenario information, evaluation resource information, and evaluation node scale information; According to the model information, algorithm information and evaluation task information in the evaluation task description file, the model training platform is scheduled for tasks and model training is triggered, including: Sending the model information and algorithm information in the evaluation task description file to the model training platform, so that the model training platform performs model training according to the model information and algorithm information; Determine a corresponding assessment task workflow according to the assessment scenario information; The evaluation task workflow, the evaluation resource information and the evaluation node scale information are sent to the model training platform so that the model training platform performs model training according to the evaluation task information.

3. The method according to claim 2, characterized in that The evaluation scenario information includes at least one of the following: a linear acceleration ratio evaluation scenario for model training, a computing power utilization evaluation scenario for model training, a fault tolerance capability evaluation scenario for model training, and a scale benefit evaluation scenario for model training.

4. The method according to claim 1, characterized in that Obtaining the scheduling log data and training log data of the model training platform when performing task scheduling and model training, including: According to the preset data filtering rules in the evaluation task description file, the scheduling log data and training log data of the model training platform when performing task scheduling and model training are obtained.

5. The method according to claim 1, characterized in that Obtaining the scheduling log data and training log data of the model training platform when performing task scheduling and model training, including: The evaluation task identification information, timestamp, resource information occupied by task scheduling, initial training parameters, and record data of each iterative process of the model training platform are obtained when performing task scheduling and model training.

6. The method according to claim 1, characterized in that Associating the scheduling log data with the training log data, including: The scheduling log data and the training log data are associated according to the evaluation task identification information and the timestamp.

7. A model training evaluation device, characterized in that: include: An evaluation task description file acquisition module is used to obtain the evaluation task description file for model training; A model training trigger module is used to schedule tasks and trigger model training on the model training platform according to the model information, algorithm information and evaluation task information in the evaluation task description file; A log data acquisition module is used to acquire scheduling log data and training log data of the model training platform when performing task scheduling and model training; The evaluation result determination module is used to associate the scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

8. A model training evaluation system, characterized in that: The system includes: an evaluation task configurator, an incentive tool, a log collector, and a data analyzer; wherein: Evaluation task configurator, used to configure the evaluation task description file for generating model training; The incentive tool is connected to the evaluation task configurator and connected to the model training platform; An incentive tool, used to schedule tasks on the model training platform and trigger model training according to the model information, algorithm information, and evaluation task information obtained in the evaluation task description file; The log collector is connected to the model training platform to obtain the scheduling log data and training log data of the model training platform when performing task scheduling and model training; The data analyzer is connected to the log collector and is used to associate the acquired scheduling log data with the training log data, and perform model training evaluation based on the associated scheduling log data and training log data to obtain an evaluation result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model training evaluation method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the model training evaluation method described in any one of claims 1 to 6 when executed.

Images