A model training method and system

Abstract

The present invention discloses a model training method and system, which is applied to a model training system including an interaction module, at least one matching module and at least one training module, including: the interaction module determines the training process of the model to be trained according to the drag operation triggered by the user based on the template control; the matching module divides the training process of the model to be trained into at least one execution instruction, and determines the target training module corresponding to each execution instruction in the training module; the interaction module sends the start instruction triggered by the user based on the start control to each target training module; each target training module executes the execution instruction corresponding to each target training module to train the model to be trained and obtain the target model. In the above technical solution, the matching module determines the target training module for each execution instruction corresponding to the training process of the model to be trained, and each target training module executes each execution instruction in sequence to realize the training of the model to be trained and improve the efficiency of model training.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a model training method and system. Background Art

[0002] In the field of deep learning, researchers need to manually design specific network models, such as neural network structures, for different tasks and write a large amount of code or scripts to train the neural network structures.

[0003] The architecture of a convolutional neural network includes hyperparameters such as the size and number of convolution kernels, the size and number of pooling or downsampling, the number of network layers, and cross-layer connections. As the number of network layers gradually increases, the number of possible network structures also increases exponentially. Researchers can often only write code or scripts suitable for model training of neural network structures based on experience and repeated trials. This process is difficult, time-consuming, and labor-intensive, resulting in low model training efficiency. Summary of the Invention

[0004] The present invention provides a model training method and system, which perform model training on a to-be-trained model based on a model training system, thereby improving the model training efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a model training method, which is applied to a model training system including an interaction module, at least one matching module, and at least one training module, the method comprising:

[0006] The interaction module determines the training process of the model to be trained according to the drag operation triggered by the user based on the template control;

[0007] The matching module divides the training process of the model to be trained into at least one execution instruction, and determines a target training module corresponding to each execution instruction in the training module;

[0008] The interaction module sends the start instruction triggered by the user based on the start control to each of the target training modules;

[0009] Each of the target training modules executes the execution instruction corresponding to the target training module to train the model to be trained to obtain a target model.

[0010] The technical solution of an embodiment of the present invention provides a model training method, which is applied to a model training system including an interaction module, at least one matching module and at least one training module. The method includes: the interaction module determines the training process of the model to be trained according to the drag operation triggered by the user based on the template control; the matching module divides the training process of the model to be trained into at least one execution instruction, and determines the target training module corresponding to each execution instruction in the training module; the interaction module sends the start instruction triggered by the user based on the start control to each target training module; each target training module executes the execution instruction corresponding to each target training module to train the model to be trained and obtain a target model. According to the above technical solution, first, the interaction module of the model training system can receive the user's dragging operation on the module control displayed by the interaction module, and determine the training process of the model to be trained according to the dragged template control and the dragging order; secondly, the matching module of the model training system can screen the training modules of the model training system, and after dividing the training process of the model to be trained into at least one execution instruction, determine the corresponding target training module for the operation of each execution instruction in the screened training modules; further, the interaction module can send the start instruction triggered by the user based on the start control to each target training module, so that each execution instruction corresponding to the training process of the model to be trained is executed in turn in each target training module of the model training system, thereby realizing the training of the model to be trained and obtaining the target model corresponding to the model to be trained. There is no need to write a large amount of code to realize the training of the model to be trained, which lowers the threshold of model training and improves the efficiency of model training.

[0011] Furthermore, the execution instruction is a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series.

[0012] Furthermore, the interaction module determines the training process of the to-be-trained model according to the drag operation triggered by the user based on the template control, including:

[0013] The interaction module determines the target template according to the drag operation triggered by the user based on the template control, and connects the target templates according to the triggering order of each drag operation to obtain the training process of the model to be trained.

[0014] Furthermore, the model training system further includes a data storage module. Accordingly, before each target training module executes the execution instruction corresponding to each target training module to train the model to be trained and obtain the target model, it also includes:

[0015] The matching module determines the storage location of the training data set required for training the model to be trained;

[0016] The matching module determines the storage location of the training data required to execute each of the execution instructions;

[0017] The data storage module stores the model to be trained, the storage location of the training data set required to train the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module.

[0018] Furthermore, the matching module divides the training process of the model to be trained into at least one execution instruction, and determines the target training module corresponding to each execution instruction in the training module, including:

[0019] The matching module divides the training process into at least one execution instruction;

[0020] The matching module scores the filtered training modules selected from the training modules based on the execution instructions sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score.

[0021] Furthermore, the matching module scores the filtered training modules filtered out from the training modules based on the execution instructions sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score, including:

[0022] Determining the position status of each of the filtered training modules based on the storage location of the target training data required to execute the target execution instruction and the position information of each of the filtered training modules;

[0023] Scoring each of the screened training modules based on the location status, health status, load status and / or resource status of each of the screened training modules;

[0024] The screened training module corresponding to the highest score is determined as the target training module corresponding to the target execution instruction.

[0025] Furthermore, each of the target training modules executes the execution instructions corresponding to the target training modules to train the model to be trained to obtain a target model, including:

[0026] Each target training module obtains the execution instruction corresponding to each target training module and the storage location of the target training data required to execute the execution instruction from the data storage module;

[0027] Each of the target training modules substitutes the target training data into the corresponding execution instruction to train the model to be trained to obtain the target model.

[0028] Furthermore, the model training system further includes an interface module, and the method further includes:

[0029] When a fault occurs in the current matching module, the fault information is sent to the interface module;

[0030] The interface module determines a replacement matching module;

[0031] The replacement matching module establishes a communication connection with each of the target training modules for training the to-be-trained model determined by the current matching module, and determines the execution status of the execution instruction corresponding to each of the target training modules;

[0032] The replacement matching module monitors the target training module corresponding to the unexecuted execution instruction.

[0033] Furthermore, the method further comprises:

[0034] When the current matching module determines that any target training module for training the model to be trained fails, the current matching module scores the idle training modules in the screened training modules based on the execution instructions corresponding to the failed target training module, and replaces the failed target training module based on the idle training model corresponding to the highest score;

[0035] The current matching module sends the execution instruction corresponding to the faulty target training module and the execution status of the execution instruction to the idle training model corresponding to the highest score.

[0036] In a second aspect, an embodiment of the present invention further provides a model training system, comprising an interaction module, at least one matching module, and at least one training module;

[0037] The interaction module is used to determine the training process of the model to be trained based on the drag operation triggered by the user based on the template control; and is also used to send the start instruction triggered by the user based on the start control to each target training module;

[0038] The matching module is used to divide the training process of the model to be trained into at least one execution instruction, and determine the target training module corresponding to each execution instruction in the training module;

[0039] Each of the target training modules is used to execute the execution instructions corresponding to each of the target training modules to train the model to be trained to obtain a target model.

[0040] The description of the second aspect in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second aspect can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0041] In this application, the names of the model training system and functional modules do not limit the system or functional modules themselves. In actual implementation, the system or functional modules may appear with other names. As long as the functions of the system or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalents.

[0042] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of a model training system provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of a model training method provided by an embodiment of the present invention;

[0046] Figure 3a A schematic diagram of a front-end page for the model training system; Figure 3b A schematic diagram of a front-end page for the model training system;

[0047] Figure 4 A flowchart of another model training method provided by an embodiment of the present invention;

[0048] Figure 5 This is an execution flow chart within the Master Node in the model training system provided by an embodiment of the present invention;

[0049] Figure 6 A diagram illustrating another implementation of a model training method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0051] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0052] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0053] Furthermore, the terms "including," "having," and any variations thereof, as used in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0054] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. When its operation is completed, the process can be terminated, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the features in the embodiments of the present invention and the embodiments can be combined with each other without conflict.

[0055] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0056] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0057] Figure 1 A schematic diagram of a model training system provided by an embodiment of the present invention, such as Figure 1 As shown, the model training system includes an interaction module, at least one matching module, and at least one training module. The interaction module can be a UI, CLI (front-end interface, command line interface), etc., with the front-end interface displaying the model template as a control. The matching module can be a MasterNode, and the training module can be a WorkerNode. In actual applications, the model training system also includes an interface module (APIHUB). When the matching module and training module start, they must register a temporary node with the APIHUB for fault tolerance and heartbeat maintenance.

[0058] The following will combine various embodiments and Figure 1 The model training system shown provides a detailed description of the model training method proposed in this application.

[0059] Figure 2 This is a flow chart of a model training method provided by an embodiment of the present invention. This embodiment is applicable to situations where the efficiency of model training needs to be improved. The method can be Figure 1 The model training system shown in the figure is executed, which specifically includes the following steps:

[0060] Step 210: The interaction module determines the training process of the model to be trained according to the drag operation triggered by the user based on the template control.

[0061] The interaction module may be a front-end page or a command line interface, the template control may include a model template control, and the interaction module may determine the model template according to a drag operation triggered by the user based on the model template control.

[0062] Figure 3a This is a front-end page diagram of the model training system, such as Figure 3a As shown, the front-end page displays template controls, such as data source controls, data preprocessing controls, statistical analysis controls, time series controls, correlation analysis controls, machine learning controls, deep learning controls, etc.

[0063] Corresponding algorithms can be pre-set in the template. For example, the data source template has the corresponding data storage location pre-set, the data preprocessing template has the corresponding data preprocessing algorithm pre-set, and the deep learning template has the corresponding deep learning algorithm pre-set.

[0064] Specifically, after the user triggers the template control, the front-end page can display the sub-template corresponding to the template. For example, after the user triggers the data preprocessing control, the front-end page can display data preprocessing such as normalization, data caching, and binarization below the data preprocessing control; after the user triggers the deep learning control, the front-end page can display deep learning such as deep Tensor Flow training below the deep learning space. After the front-end page receives the drag operation that the user can trigger on any template control or the sub-template control corresponding to the template control, the template and / or sub-template dragged by the user can be displayed in the display interface in the order of dragging. Furthermore, the training process of the model to be trained can be determined based on the dragging order and the template and / or sub-template dragged by the user.

[0065] In an embodiment of the present invention, the interaction module determines the training process of the model to be trained based on the drag operation triggered by the user on the template control displayed by the interaction module, and determines the training process of the model to be trained required by the user in the form of dragging, and the interaction module can display the training process of the model to be trained in the display interface of the front-end page in the form of templates and / or sub-templates, so that the training process of the model to be trained is visualized.

[0066] Step 220: The matching module divides the training process of the model to be trained into at least one execution instruction, and determines a target training module corresponding to each execution instruction in the training module.

[0067] The executed instruction may include a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series.

[0068] Specifically, the matching module can divide the training process of the model to be trained into at least one execution instruction, which can be a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series. The matching module determines whether each training module in the model training system is idle and further screens the idle training modules. Specifically, the idle training modules can be screened based on preset rules to obtain a screened training module suitable for running the execution instruction corresponding to the training process of the model to be trained.

[0069] The matching model can also prioritize the execution instructions according to their dependencies and, based on each execution instruction, score the selected training modules in turn, obtaining a scoring result for each execution instruction against each selected training module. Furthermore, the selected training module corresponding to the maximum score for each execution instruction can be determined as the target training module for each execution instruction.

[0070] In an embodiment of the present invention, the matching module can split the training process of the model to be trained into multiple execution instructions, and assign a different training module to each execution instruction, so that when training the model to be trained, multiple different training modules can be run serially or in parallel to improve the training efficiency of the model to be trained.

[0071] Step 230: The interaction module sends the start instruction triggered by the user based on the start control to each of the target training modules.

[0072] The start instruction may include training information, such as training name, training description, training method, template name, associated tests, etc.

[0073] Specifically, the interaction module can determine and display the creation interface and start control based on the creation instruction triggered by the user based on the creation control of the training job. Figure 3b This is another front-end page diagram for the model training system, such as Figure 3bAs shown, a creation page is displayed in the front-end page, and the interactive module can also receive training information input by the user based on the creation interface, and can also send the start instruction triggered by the user based on the start control in the creation page to each target training module.

[0074] Step 240: Each of the target training modules executes the execution instruction corresponding to the target training module to train the model to be trained to obtain a target model.

[0075] Specifically, after receiving a start instruction from the interactive module, triggered by a user using a start control, each target training module can sequentially execute the corresponding execution instruction. By executing each execution instruction in each target training module according to the training process, the target model can be trained, and the target model corresponding to the target model can be obtained. This eliminates the need to write extensive code to train the target model, thereby improving model training efficiency.

[0076] In an embodiment of the present invention, each target training module executes the execution instructions corresponding to each target training module, which can realize the training of the model to be trained, that is, there is no need to write a large amount of code to realize the training of the model to be trained, thereby further improving the model training efficiency.

[0077] The model training method provided by an embodiment of the present invention is applied to a model training system including an interaction module, at least one matching module and at least one training module, including: the interaction module determines the model to be trained and the training process of the model to be trained according to the drag operation triggered by the user based on the template control; the matching module divides the training process of the model to be trained into at least one execution instruction, and determines the target training module corresponding to each execution instruction in the training module; the interaction module sends the start instruction triggered by the user based on the start control to each target training module; each target training module executes the execution instruction corresponding to each target training module to train the model to be trained and obtain the target model. According to the above technical solution, first, the interaction module of the model training system can receive the user's dragging operation on the module control displayed by the interaction module, and determine the training process of the model to be trained according to the dragged template control and the dragging order; secondly, the matching module of the model training system can screen the training modules of the model training system, and after dividing the training process of the model to be trained into at least one execution instruction, determine the corresponding target training module for the operation of each execution instruction in the screened training modules; further, the interaction module can send the start instruction triggered by the user based on the start control to each target training module, so that each execution instruction corresponding to the training process of the model to be trained is executed in turn in each target training module of the model training system, thereby realizing the training of the model to be trained and obtaining the target model corresponding to the model to be trained. There is no need to write a large amount of code to realize the training of the model to be trained, which lowers the threshold of model training and improves the efficiency of model training.

[0078] Figure 4 This is a flowchart of another model training method provided by an embodiment of the present invention. This embodiment is specific based on the above embodiment and is applied to a model training system including an interaction module, at least one matching module, at least one training module and a data storage module. Figure 4 As shown, in this embodiment, the method may further include:

[0079] Step 410: The interaction module determines a target template according to the drag operation triggered by the user based on the template control, and connects the target templates according to the triggering order of each drag operation to obtain the training process of the model to be trained.

[0080] Among them, the interactive module can be understood as a front-end page and / or a command line interface. In the present application, the interactive module can be a front-end page, i.e., a UI, which can be used to visually display template controls. The template can be understood as a program template. For example, it can be defined as an AI training JOB. By building blocks to train task nodes and establishing associations between training task nodes, the AI ​​training JOB definition can be obtained. The tasks in the AI ​​training JOB definition are assembled in the form of a PIPELINE. When the AI ​​training JOB definition is executed, each PIPELINE can be topologically traversed from the node with zero in-degree until there are no successor nodes. The model to be trained can be the running state of the program template in the computer. For example, it can be an AI training JOB instance. The AI ​​training JOB instance is an instantiation of the AI ​​training JOB definition. It can be generated by manual startup or timed scheduling. Each time the AI ​​training JOB definition is run, an AI training JOB instance can be generated.

[0081] Specifically, the user can drag and drop the template controls displayed by the UI. The UI can determine the AI ​​training JOB definition based on the template controls dragged by the user, and further connect the AI ​​training JOB definitions in a building block-based manner according to the triggering order of the user's dragging operations on each template control to obtain an AI training JOB instance.

[0082] like Figure 1 As shown, the model training system also includes an interface module, namely the API Hub. The API Hub provides access to source data and notification of changes, and offers mechanisms for authentication, authorization, access control, matching module registration, and training module registration and discovery. In practice, after the interaction module determines the training process for the model to be trained, it can send the training process to the API Hub, which then sends the training process to the matching module.

[0083] In the embodiment of the present invention, the interaction module can determine the training process of the model to be trained based on the template controls dragged by the user and the order in which the template controls are dragged, thereby realizing visualization of the training process of the model to be trained.

[0084] Step 411: The matching module determines the storage location of the training data set required to train the model to be trained.

[0085] like Figure 1 As shown, the model training system also includes a data storage module, namely Meta Data Drive.

[0086] The training dataset can be stored in the Meta Data Drive of the model training system, or in a database (ES) or server (Serve). When the training dataset required to train the model to be trained is stored in the database or server, the database or server can send the training dataset to the model training system based on the API HUB.

[0087] Specifically, the matching module may determine the storage location of the training data set in the data storage module, or the storage location in the database or server.

[0088] Step 412: The matching module divides the model to be trained into at least one execution instruction.

[0089] The execution instruction can be understood as an AI training task instance, which is an instantiation of the task node in the AI ​​training job definition and represents the specific task execution status. It can be a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series.

[0090] Specifically, the matching module can divide the training process of the model to be trained into at least one execution instruction, that is, the execution instruction can be used to divide the training process of the model to be trained into a single instruction, multiple instructions executed in parallel, and / or multiple instructions executed in series. For example, the data cache in the training process of the model to be trained can be determined as a single instruction, the binarization in the training process of the model to be trained can be determined as multiple instructions executed in parallel, and the deep learning in the training process of the model to be trained can be determined as multiple instructions executed in series. Furthermore, the training process of the model to be trained can be divided into the data cache represented by a single instruction, the binarization represented by multiple instructions executed in parallel, and the deep learning represented by multiple instructions executed in series.

[0091] For example, the Master Node divides the AI ​​training JOB instance sent by the API HUB into multiple stages, that is, the AI ​​training JOB instance can be divided into Taskset, parallel execution Tasksets and / or serial execution Tasksets.

[0092] Figure 5 This is an execution flow chart in the Master Node of the model training system provided by the embodiment of the present invention, such as Figure 5As shown, the ShedulerServ in the Master Node receives the AI ​​training job instance and sends it to the DAGSheduler in the Master Node. The DAGSheduler divides the AI ​​training job instance into multiple stages. Stages include Tasksets, parallel-executed Tasksets, and / or serial-executed Tasksets. The TaskSheduler is responsible for scheduling each task in the Tasksets to start executing tasks on the Executor of the Worker Node.

[0093] Step 413: The matching module determines the storage location of the training data required to execute each of the execution instructions.

[0094] As mentioned above, specifically, the matching module can determine the storage location of the training data set in the data storage module, or the storage location in the database or server.

[0095] Step 414: The matching module scores the filtered training modules selected from the training modules based on the execution instructions sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score.

[0096] Specifically, the matching module first filters out the filtered training modules from the training modules. Specifically, the running resources of each execution instruction corresponding to the training process of the model to be trained can be determined, and the training modules can be filtered according to the running resources, and the training modules with insufficient running resources can be filtered out to obtain the filtered training modules. Secondly, the execution instructions can be sorted according to the execution priority, and the filtered training modules filtered out from the training modules can be scored according to the execution priority sorted execution instructions. Then, the filtered training module corresponding to the highest score determined by the execution instruction can be determined as the target training module corresponding to the execution instruction.

[0097] In one implementation, step 414 may specifically include:

[0098] Based on the storage location of the target training data required to execute the target execution instruction and the location information of each of the filtered training modules, the location status of each of the filtered training modules is determined; based on the location status, health status, load condition and / or resource condition of each of the filtered training modules, each of the filtered training modules is scored; and the filtered training module corresponding to the highest score is determined as the target training module corresponding to the target execution instruction.

[0099] Specifically, for any target execution instruction that has been sorted according to the execution priority, the position status of each filtered training module can be determined first based on the storage location of the target training data required to execute the target execution instruction and the position information of each filtered training module. If the storage location of the target training data required to execute the target execution instruction is close to the position information of the filtered training module, then the position status of the filtered training module is determined to be a higher level. If the storage location of the target training data required to execute the target execution instruction is far from the position information of the filtered training module, then the position status of the filtered training module is determined to be a lower level. The specific level can be determined according to actual needs. Of course, the health status, load status and resource status of each filtered training module can also be determined. Then, the score of each filtered training module can be determined based on the weight of the position status, load status and resource status, and the filtered training module corresponding to the highest score is determined as the target training module corresponding to the target execution instruction.

[0100] For example, the Master Node uses pre-selection and optimization strategies to determine the set of Worker Nodes that meet the training requirements of the model to be trained. It then sorts the Worker Nodes in the set according to the stage's execution priority to determine the Worker Node corresponding to each stage. For example, the highest-scoring Worker Node determined by the stage with the highest execution priority can be designated as the Worker Node corresponding to the stage with the highest execution priority. Furthermore, the stages can be bound to their corresponding Worker Nodes based on the FAIR algorithm policy mechanism, and a successful binding event can be registered with the API Hub.

[0101] In practical applications, the model training system may include at least one matching module. When the model training system includes multiple matching modules, each matching module may determine the target training module for multiple models to be trained respectively. Based on the model training system, model training can be performed on multiple models to be trained at the same time.

[0102] In an embodiment of the present invention, the matching module splits the training process of the model to be trained into multiple execution instructions, and assigns a different training module to each execution instruction, so that when the model to be trained is trained, multiple different training modules can be run serially or in parallel to improve the training efficiency of the model to be trained.

[0103] Step 415, the data storage module stores the model to be trained, the storage location of the training data set required to train the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module.

[0104] The data storage module may be a Meta Data Drive.

[0105] Specifically, the Meta Data Drive can store AI training JOB instances, the storage location of Create source data, multiple Stages corresponding to the AI ​​training JOB instances, the storage location of Create source data required to execute each Stage, and the Worker Node corresponding to each Stage.

[0106] In actual applications, when the model training system replaces an AI training job instance, new Master Nodes and Worker Nodes may be added. When the Stage corresponding to the AI ​​training job instance is running on a Worker Node, if the Master Node or Worker Node crashes, or if a Stage in a Worker Node or Tasksets in a Stage experiences an anomaly, the API HUB will notify the Meta Data Drive and other Master Nodes and Worker Nodes. Of course, the data in the Meta Data Drive can also be changed synchronously. The API HUB decouples the UI, Master Node, Worker Node, and Meta Data Drive. By monitoring the status of the API UI, Master Node, Worker Node, and Meta Data Drive, and notifying other modules in a timely manner, the stability of the model training system is improved.

[0107] Step 416: The interaction module sends the start instruction triggered by the user based on the start control to each of the target training modules.

[0108] As mentioned above, the interaction module can send the start instruction triggered by the user based on the start control in the creation page to each target training module.

[0109] Step 417: Each of the target training modules executes the execution instruction corresponding to the target training module to train the model to be trained to obtain a target model.

[0110] In one implementation, step 417 may specifically include:

[0111] Each target training module obtains the execution instruction corresponding to each target training module and the storage location of the target training data required to execute the execution instruction from the data storage module; each target training module substitutes the target training data into the corresponding execution instruction to train the model to be trained to obtain the target model.

[0112] Specifically, a Worker Node retrieves the corresponding Stage and the storage location of the Create source data required to execute the Stage from the Meta Data Drive. The Worker Node then executes the corresponding Stage based on the Create source data, implementing the AI ​​training job instance and training the target model to obtain the target model. Furthermore, the Worker Node monitors the status of the corresponding bound Stage and synchronizes updates to it. Of course, the Master Server included in the Master Node can also view the Stage status through the API Hub. Specifically, the Exec Runner in the Worker Node creates task operations for each task in the Tasksets corresponding to the Stage. The Executor launches the Exec Runner to execute the corresponding task, thus executing the Stage. The ShedulerServ in the Master Node calls the API Hub to create bound tasks on the Meta Data Drivers, describing the Stages running on each Worker Node and the Tasksets corresponding to the Stages. The Exec Runner in the Worker Node periodically synchronizes the status of the bound tasks with the Meta Data Drive, automatically syncing any bound task updates to the API Hub in real time. Of course, the Master Node can also update the status of the bound Task running on the Worker Node in real time through the watch mechanism of the API HUB

[0113] Step 418: When a fault occurs in the current matching module, the fault information is sent to the interface module; the interface module determines a replacement matching module; the replacement matching module establishes a communication connection with each of the target training modules for training the model to be trained determined by the current matching module, and determines the execution status of the execution instructions corresponding to each of the target training modules; the replacement matching module monitors the target training modules corresponding to the execution instructions that have not been executed.

[0114] Specifically, when any Master Node in the AI ​​training JOB instance may fail, the fault information needs to be sent to the API HUB. The API HUB can notify other Master Nodes in the model training system to compete for the training process of the model to be trained to determine the replacement Master Node to replace the Master Node that has failed.

[0115] The replacement Master Node can establish a communication connection with each Worker Node that executes the AI ​​training JOB instance determined by the Master Node that has crashed, determine the execution status of the Stage corresponding to each Worker Node, and monitor the Worker Node corresponding to the unfinished Stage.

[0116] In an embodiment of the present invention, it is clarified that when the current matching module fails, the model training system's subsequent processing method for the training process of the model to be trained corresponding to the current matching module is clarified, so as to ensure the normal operation of the training process of the model to be trained corresponding to the current matching module when the current matching module fails.

[0117] Step 419: When the current matching module determines that any of the target training modules for training the model to be trained fails, the current matching module scores the idle training modules in the screened training modules based on the execution instructions corresponding to the failed target training module, and replaces the failed target training module based on the idle training model corresponding to the highest score; the current matching module sends the execution instructions corresponding to the failed target training module and the execution status of the execution instructions to the idle training model corresponding to the highest score.

[0118] Specifically, when any Worker Node corresponding to the AI ​​training JOB instance fails, the Master Node scores the idle WorkerNodes among the previously determined filtered Worker Nodes based on the Stage corresponding to the failed Worker Node. It can also score each idle Worker Node based on its location status, health status, load status, and / or resource status, and replace the failed Worker Node with the idle Worker Node corresponding to the highest score. The Master Node can send the Stage corresponding to the failed Worker Node and the execution status of the Stage to the idle Worker Node corresponding to the highest score, and then the idle Worker Node can execute the Stage corresponding to the failed Worker Node.

[0119] Of course, when any Worker Node fails, the failure information can be sent to the API HUB. After the MasterNode determines the idle Worker Node corresponding to the highest score to replace the failed Worker Node, the information of the idle Worker Node corresponding to the highest score can be sent to the API HUB.

[0120] In an embodiment of the present invention, the subsequent processing method of the model training system for the execution instructions corresponding to the target training module when a target training module fails is clarified, so as to ensure the normal execution of the execution instructions corresponding to the target training module when a target training module fails.

[0121] It should be noted that the training process can only repair faults three times. If the matching module and / or training module fails more than three times during the training process, manual recovery or manual rerun is required. Manual execution can be started from the failed execution instruction or from the current execution instruction. Manual rerun needs to start from the start execution instruction.

[0122] The model training method provided by an embodiment of the present invention is applied to a model training system including an interaction module, at least one matching module, at least one training module and a data storage module, including: the interaction module determines the target template according to the drag operation triggered by the user based on the template control, and connects each target template according to the triggering order of each drag operation to obtain the training process of the model to be trained; the matching module determines the storage location of the training data set required for training the model to be trained; the matching module divides the model to be trained into at least one execution instruction; the matching module determines the storage location of the training data required to execute each execution instruction; the matching module scores the filtered training modules from the training module based on each execution instruction sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score; the data storage module stores the model to be trained, the storage location of the training data set required for training the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module; Each of the target training modules executes the execution instructions corresponding to each of the target training modules to train the model to be trained and obtain the target model; when any other matching module determines that the current matching module has failed, the other matching modules that determine that the current matching module has failed determine a replacement matching module for the current matching module; the replacement matching module establishes a communication connection with each of the target training modules for training the model to be trained determined by the current matching module, and determines the execution status of the execution instructions corresponding to each of the target training modules; the replacement matching module monitors the target training modules corresponding to the execution instructions that have not been executed; when the current matching module determines that any of the target training modules for training the model to be trained has failed, the current matching module scores the other unassigned training modules based on the execution instructions corresponding to the failed target training module, and replaces the failed target training module based on the unassigned training model corresponding to the highest score; the current matching module sends the execution instructions corresponding to the failed target training module and the execution status of the execution instructions to the unassigned training model corresponding to the highest score. In an embodiment of the present invention, the interaction module can determine the training process of the model to be trained, and the matching module can determine the corresponding target training module from the execution instructions corresponding to the training process of the model to be trained in the screened training modules. After receiving the start instruction sent by the interaction module, the target training module executes each execution instruction in sequence to realize the training of the model to be trained and obtain the target model corresponding to the model to be trained. There is no need to write a large amount of code to realize the training of the model to be trained, which lowers the threshold of model training. In addition, the fault-tolerant mechanism of the model training system is clarified, which ensures the normal operation of model training and further improves the efficiency of model training.

[0123] Figure 6 Another implementation diagram of a model training method provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown in Figure 1, this implementation demonstrates the interaction between modules when the model training system executes the model training method, which includes:

[0124] Step 610: The interaction module determines the training process of the model to be trained.

[0125] Step 611: The interaction module sends the training process of the model to be trained to the matching module based on the interface module.

[0126] Step 612: The matching module determines the storage location of the training data set required to train the model to be trained.

[0127] Step 613: The matching module divides the model to be trained into at least one execution instruction.

[0128] Step 614: The matching module determines the storage location of the training data required to execute each execution instruction.

[0129] Step 615: The matching module determines the target training module corresponding to each execution instruction.

[0130] Step 616: The matching module sends the model to be trained, the storage location of the training data set required to train the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module to the data storage module based on the interface module.

[0131] Step 617: The data storage module sends the storage locations of the model to be trained, the training data set required to train the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module to the data storage module.

[0132] Step 618: The interaction module determines a start instruction according to the triggering operation of the user based on the start control triggering.

[0133] Step 619: The interaction module sends a start instruction to each target training module.

[0134] Step 620: Each target training module executes the corresponding execution instruction to train the model to be trained to obtain the target model.

[0135] The implementation method of the model training method provided in the embodiment of the present invention has the corresponding beneficial effects of executing any of the aforementioned model training methods.

[0136] The embodiment of the present invention provides a model training system, such as Figure 1As shown, it includes an interaction module, at least one matching module and at least one training module; the interaction module is used to determine the training process of the model to be trained according to the drag operation triggered by the user based on the template control; it is also used to send the start instruction triggered by the user based on the start control to each target training module; the matching module is used to divide the training process of the model to be trained into at least one execution instruction, and determine the target training module corresponding to each execution instruction in the training module; each target training module is used to execute the execution instruction corresponding to each target training module to train the model to be trained and obtain a target model.

[0137] The execution instruction is a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series.

[0138] Preferably, the interaction module is specifically used to: determine the target template according to the drag operation triggered by the user based on the template control, and connect each target template according to the triggering order of each drag operation to obtain the training process of the model to be trained.

[0139] Preferably, the model training system also includes a data storage module, which is used to store the model to be trained, the storage location of the training data set required to train the model to be trained determined by the matching module, each execution instruction, the storage location of the training data required to execute each execution instruction determined by the matching module, and the target training module.

[0140] Preferably, the matching module is specifically used to: divide the training process into at least one execution instruction; score the filtered training modules selected from the training modules based on each execution instruction sorted according to execution priority, and determine the target training module corresponding to each execution instruction according to the score.

[0141] Furthermore, scoring the filtered training modules filtered out from the training modules based on the execution instructions sorted according to the execution priority is performed in sequence, and determining the target training module corresponding to each execution instruction according to the score, including:

[0142] Based on the storage location of the target training data required to execute the target execution instruction and the location information of each of the filtered training modules, the location status of each of the filtered training modules is determined; based on the location status, health status, load condition and / or resource condition of each of the filtered training modules, each of the filtered training modules is scored; and the filtered training module corresponding to the highest score is determined as the target training module corresponding to the target execution instruction.

[0143] Preferably, the training module is specifically used for: each target training module obtains the execution instruction corresponding to each target training module and the storage location of the target training data required to execute the execution instruction from the data storage module; each target training module substitutes the target training data into the corresponding execution instruction to train the model to be trained to obtain the target model.

[0144] The model training system also includes an interface module, which is used to receive fault information sent when a fault occurs in the current matching module, and determine a replacement matching module so that the replacement matching module establishes a communication connection with each target training module determined by the current matching module to train the model to be trained, and determines the execution status of the execution instructions corresponding to each target training module; monitors the target training modules corresponding to the execution instructions that have not been executed.

[0145] Preferably, the matching module is also used to: when it is determined that any of the target training modules for training the model to be trained fails, the current matching module scores the idle training modules in the screened training modules based on the execution instructions corresponding to the failed target training module, and replaces the failed target training module based on the idle training model corresponding to the highest score; and sends the execution instructions corresponding to the failed target training module and the execution status of the execution instructions to the idle training model corresponding to the highest score.

[0146] The model training system provided by the embodiment of the present invention can execute the model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the model training method.

[0147] It is worth noting that in the embodiment of the above-mentioned model training system, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0148] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention comply with the relevant provisions of national laws and regulations.

[0149] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A model training method, characterized in that: Applied to a model training system including an interaction module, at least one matching module, and at least one training module, the method includes: The interaction module determines the training process of the to-be-trained model according to the drag operation triggered by the user based on the template control; The matching module divides the training process of the model to be trained into at least one execution instruction, and scores the filtered training modules filtered from the training modules based on each execution instruction sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score; The interaction module sends the start instruction triggered by the user based on the start control to each of the target training modules; Each of the target training modules executes the execution instruction corresponding to the target training module to train the model to be trained to obtain a target model.

2. The model training method according to claim 1, characterized in that The execution instruction is a single instruction, multiple instructions executed in parallel, or multiple instructions executed in series.

3. The model training method according to claim 1, characterized in that The interaction module determines the training process of the to-be-trained model according to the drag operation triggered by the user based on the template control, including: The interaction module determines the target template according to the drag operation triggered by the user based on the template control, and connects the target templates according to the triggering order of each drag operation to obtain the training process of the model to be trained.

4. The model training method according to claim 1, characterized in that The model training system further includes a data storage module. Accordingly, before each target training module executes the execution instruction corresponding to each target training module to train the model to be trained and obtain the target model, the system further includes: The matching module determines the storage location of the training data set required for training the model to be trained; The matching module determines the storage location of the training data required to execute each of the execution instructions; The data storage module stores the model to be trained, the storage location of the training data set required to train the model to be trained, each execution instruction, the storage location of the training data required to execute each execution instruction, and the target training module.

5. The model training method according to claim 4, characterized in that The matching module scores the filtered training modules filtered from the training modules based on the execution instructions sorted according to the execution priority, and determines the target training module corresponding to each execution instruction according to the score, including: Determining the position status of each of the filtered training modules based on the storage location of the target training data required to execute the target execution instruction and the position information of each of the filtered training modules; Scoring each of the screened training modules based on the location status, health status, load status and / or resource status of each of the screened training modules; The screened training module corresponding to the highest score is determined as the target training module corresponding to the target execution instruction.

6. The model training method according to claim 4, characterized in that Each of the target training modules executes the execution instruction corresponding to each of the target training modules to train the model to be trained to obtain a target model, including: Each target training module obtains the execution instruction corresponding to each target training module and the storage location of the target training data required to execute the execution instruction from the data storage module; Each of the target training modules substitutes the target training data into the corresponding execution instruction to train the model to be trained to obtain the target model.

7. The model training method according to claim 1, characterized in that The model training system further includes an interface module, and the method further includes: When a fault occurs in the current matching module, the fault information is sent to the interface module; The interface module determines a replacement matching module; The replacement matching module establishes a communication connection with each of the target training modules for training the to-be-trained model determined by the current matching module, and determines the execution status of the execution instruction corresponding to each of the target training modules; The replacement matching module monitors the target training module corresponding to the unexecuted execution instruction.

8. The model training method according to claim 4, characterized in that Also includes: When the current matching module determines that any target training module for training the model to be trained fails, the current matching module scores the idle training modules in the screened training modules based on the execution instructions corresponding to the failed target training module, and replaces the failed target training module with the idle training module corresponding to the highest score; The current matching module sends the execution instruction corresponding to the target training module having the fault and the execution status of the execution instruction to the idle training module corresponding to the highest score.

9. A model training system, characterized in that: comprising an interaction module, at least one matching module and at least one training module; The interaction module is used to determine the training process of the model to be trained according to the drag operation triggered by the user based on the template control; It is also used to send the start instruction triggered by the user based on the start control to each target training module; The matching module is configured to divide the training process of the model to be trained into at least one execution instruction, and score the filtered training modules filtered from the training modules based on each execution instruction sorted according to the execution priority, and determine the target training module corresponding to each execution instruction according to the score; Each of the target training modules is used to execute the execution instructions corresponding to each of the target training modules to train the model to be trained to obtain a target model.

Images