A tool system for deep learning model training

Abstract

The application discloses a tool system for deep learning model training, and belongs to the field of artificial intelligence, which comprises a core component layer, a configuration system layer, an experiment management layer and a command interaction layer; the core component layer is used for realizing low coupling degree and reusability of codes; the configuration system layer is used for setting configurable items and archiving all configurable items corresponding to experiments; the experiment management layer is used for organizing all deep learning model training experiments in a certain hierarchical structure and storing experiment information; the command interaction layer is composed of a limited number of sub-commands and is used for providing an efficient interaction mode for users to perform experiment management and start experiments; the application can efficiently manage a large amount of codes, configurable items and generated experiments, provide various start templates, and achieve a good balance between convenience and flexibility; the application has strong universality and expandability, and can quickly and intuitively set configurable items; and the application can migrate existing codes at low cost and save time and effort for managing a large number of experiments.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a tool system for deep learning model training. BACKGROUND

[0002] In recent years, artificial intelligence related research and technology has developed rapidly. Through data collection and labeling, a deep learning algorithm model is designed, implemented and trained, and finally used to solve tasks in computer vision, natural language processing, speech recognition and many other fields, which has become a common method. A deep learning algorithm model is usually composed of multiple layers of neural networks, and learns complex patterns and features in data through training. In the process of training the model, in addition to the model itself, the data that can be learned, the objective function that quantifies the effectiveness of the model, and the optimization algorithm that adjusts the parameters of the model to optimize the objective function are the most critical factors. Deep learning frameworks such as PyTorch, TensorFlow and Caffe provide basic capabilities to implement and train deep learning algorithm models. Tools such as Keras and PyTorch-Lightning provide a certain degree of abstraction and encapsulation on the basis of deep learning frameworks, saving the trouble of writing a large amount of sample code. These tools are usually implemented using the Python programming language or provide a calling interface for the Python programming language.

[0003] In the process of designing, implementing and training deep learning algorithm model, experiments are often carried out in many aspects to select the relatively optimal model. These experiments include trying different model structures, hyperparameters and training strategies, etc. Therefore, a large number of experiments will be produced, and a large number of model codes and configurable items including hyperparameters will be involved. Model code refers to the definition code of model structure, which usually needs to be implemented using a deep learning framework. The model is usually composed of multiple modules, each of which can be further divided into neural network layers. Different models may contain some same modules. Hyperparameters refer to parameters that need to be set manually rather than learned during the training process of the model, such as the number of neural network layers, batch size, initial learning rate and learning rate decay strategy, regularization coefficient, training rounds, etc. The setting of hyperparameters has an important influence on the results of model training, and usually needs to be repeatedly tried and adjusted to find the best combination. In addition to hyperparameters, other common configurable items include objective functions, optimization algorithms, saving frequency of model parameter states, evaluation indicators used when evaluating model performance, etc. It is also very important to record these configurable items completely, which is one of the key aspects to ensure the reproducibility of experiments. Here, reproducibility refers to the ability to repeatedly obtain the same or similar experimental results under the same conditions. At the same time, a large number of outputs may be generated during the training of deep learning models, such as model parameter state files, training logs, visual images, etc. In this context, it is likely that a large number of model codes, configurable items and experiments will be confused during the training of deep learning models, so that researchers and developers have to spend a lot of energy and time on sorting and comparing experimental results, rather than the algorithm itself. In addition, during the process of algorithm research and development, it is often necessary to use other people's open source algorithm models. Due to different code habits of each person, these code libraries have different organizational structures, and may be written for specific tasks or data sets without good top-level design. It is quite cumbersome to use these codes for customized experiments.

[0004] At present, the existing technical solutions related to deep learning model training mainly focus on implementing a platform for conveniently calling computing resources to implement model training, aiming to reduce the burden of users in hardware resource management, software and hardware adaptation, human-computer interaction and operation and maintenance, etc. The technical solution closest to the present invention is as follows:

[0005]

[0006]

[0007] The main shortcomings of the prior art are:

[0008] On the one hand, there is currently a lack of practical technology to consider and solve the problem of how to efficiently manage a large amount of code, configurable items and experiments involved in the process of training deep learning algorithm model. On the other hand, for the above-mentioned similar prior art, it is mainly implemented at the platform level, expecting to obtain an interactive system with simple operation to reduce the burden of users, and is not designed from the code level to provide development tools for algorithm developers who write code. And in terms of model and training process, it is largely limited to pre-designed options and logic, which lacks flexibility and scalability. At the same time, the components provided in some platform systems are only for specific task fields, and the universality is poor. SUMMARY

[0009] Therefore, in order to solve the problems existing in the prior art, the present application provides a tool system for deep learning model training. The present application realizes low coupling degree and reusability of code by abstracting key elements involved during training of deep learning algorithm model, and the present application realizes flexible setting and complete recording of a large number of configurable items by designing a configuration system, and at the same time provides an experiment management tool, so that users can efficiently organize and manage a large number of experiments, further ensuring the reproducibility of experiments. The present application can enable users to focus more time and effort on the algorithm itself, thereby significantly improving the efficiency of the deep learning model development and training process.

[0010] The technical scheme adopted by the present application to solve the technical problems is as follows:

[0011] The tool system for deep learning model training provided by the present application comprises:

[0012] The core component layer is used to realize low coupling degree and reusability of code;

[0013] The configuration system layer is used to set configurable items, and is used to archive all configurable items corresponding to experiments;

[0014] The experiment management layer is used to organize all deep learning model training experiments performed according to a certain hierarchical structure, and is used to store experiment information;

[0015] The command interaction layer is composed of a limited number of sub-commands, and is used to provide an efficient interaction way for users to perform experiment management and start experiments.

[0016] Further, the core component layer includes four core components, namely algorithm model, data module, task module and executor; the algorithm model refers to a deep learning algorithm model, which is implemented through a deep learning framework; the data module refers to a definition related to data, including the location of data, reading and preprocessing methods, and an iterable data loader; the task module refers to a definition of a deep learning task process, including a target function module, an optimization algorithm module, a learning rate regulator module, and an evaluation index module used in verification and testing; and the executor refers to related codes for executing model training, verification, testing and inference processes.

[0017] Further, the core component layer includes multiple types of startup templates; the startup templates are based on specific deep learning frameworks or high-level encapsulation frameworks thereof; the startup templates are registered on a remote server, and a user can retrieve and download the startup templates through a start command in the command interaction layer; when downloading, all files in the startup templates are downloaded from the remote server to the local in a recursive manner through a network request.

[0018] Further, the core component layer includes abstract base classes of the four core components, which are inherited and specifically implemented by a user after being imported from a software package; for a user who needs to maximize customization or whose needs cannot be met by a pre-defined startup template, the user directly inherits the abstract base classes of the four core components and provides a specific implementation that meets the needs of the user, that is, the abstract interface specified by the abstract base class is implemented on demand, and the implementation can seamlessly connect with other parts of the system.

[0019] Further, the configuration files in the configuration system layer are Python source code files that meet certain rules; each configuration file includes one or more functional configuration items, that is, each configuration item is a function, in which a class that needs to be configured is directly instantiated, and then the class is taken as a return value of the function.

[0020] Further, the core configuration items in the configurable items include an algorithm model configuration item, a data module configuration item, a task module configuration item and an executor configuration item, which are respectively used for configuring a model, data, a task and an execution process.

[0021] Further, the configuration system layer uses the decorator syntax in the Python programming language to define four kinds of decorators corresponding to the four core components, for verifying the parameters of the functional configuration items and marking the types of the configuration items; when conducting experiments, the four kinds of configuration items corresponding to the four core components are combined to form a root configuration, which is the root node of all configurable items; when performing a certain task, only the path of the file where the root node is located needs to be specified, and the configuration system layer will read and dynamically import the source code module where the root configuration is located from the path, hold the functions after parsing, and call the corresponding functions to obtain the configured required objects.

[0022] Further, the configuration system layer includes a configuration archiver for automatically archiving configuration files and combining configuration files that may exist in multiple configuration files into a single configuration file corresponding to a certain model training process, and performing merging and repeated import items during the process.

[0023] Further, the experiment information is divided into three levels: Project, Experiment, and Run:

[0024] Run is the smallest atomic unit corresponding to each run, and each Run must belong to a certain Experiment in a certain Project;

[0025] Experiment is a group of Runs with common themes, such as comparative experiments, ablation experiments, etc., and each Experiment must belong to a certain Project;

[0026] Project is composed of multiple Experiments, and refers to a specific project.

[0027] Further, the Project, Experiment, and Run have corresponding IDs, names, descriptions, creation times, and storage path information.

[0028] Further, the experiment management layer includes a runtime context, through which the current Run object can be obtained in the custom task module, so that the attribute information of the Run object can be obtained and used during the experiment.

[0029] Further, the function descriptions of the sub-commands are as follows:

[0030] Sub-command start: retrieve available start templates, download the specified start template;

[0031] Sub-command new: create a pair of new Project and Experiment;

[0032] Subcommand add-exp: add a new Experiment to the specified Project;

[0033] Subcommand ls: display Projects, Experiments and Runs in the terminal in tabular form;

[0034] Subcommand fit: perform model training on the training set;

[0035] Subcommand resume-fit: resume from an interrupted model training process;

[0036] Subcommand validate: perform model evaluation on the validation set;

[0037] Subcommand test: perform model evaluation on the test set;

[0038] Subcommand predict: use the specified model to make predictions on the specified data.

[0039] The beneficial effects of the present application are:

[0040] 1. The present application provides a system tool at the software engineering level, which enables efficient management of a large amount of code, configurable items and experiments generated when conducting deep learning algorithm research or development.

[0041] 2. The present application provides multiple start templates and can be customized based on the abstract interface of the core component, achieving a good balance between convenience and flexibility.

[0042] 3. The present application is independent of specific tasks and deep learning frameworks, and has strong universality and scalability.

[0043] 4. The unique configuration system designed by the present application enables users to quickly and intuitively set configurable items and reuse configurable items, and the accompanying configuration archiver can enhance the traceability and reproducibility of experiments.

[0044] 5. The present application does not impose rule constraints on model definition code, does not require significant modification of existing model definition code, and facilitates low-cost migration of existing code.

[0045] 6. The present application can organize experiments through a command line interface, saving time and effort in managing a large number of experiments. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A structure composition framework of a tool system for deep learning model training is provided. DETAILED DESCRIPTION

[0047] The application will be further described in detail below with reference to the accompanying drawings.

[0048] The application aims to design and implement a tool system for conveniently managing codes, configurable items and experiments during training of a deep learning algorithm model, realizing low coupling degree and reusability of the codes, flexible setting and complete recording of the configurable items, and efficient management of a large number of experiments.

[0049] The tool system for deep learning model training provided by the application is implemented based on a Python programming language and used in the form of a Python software package.

[0050] In the application, the core component layer provides abstract interfaces of four core components (algorithm model, data module, task module and executor), and the four core components are in a loose coupling relationship, facilitating reuse of the components.

[0051] In the application, the configuration system layer provides a convenient way to set a large number of configurable items, and a root configuration is a root node of all configurable items, and the root configuration contains configuration items corresponding to the four core components.

[0052] In the application, the experiment management layer organizes all deep learning model training experiments in a certain hierarchical structure, and related data of the experiments can be saved in different storage backends as needed.

[0053] In the application, the command interaction layer is composed of a limited number of sub-commands, and provides an efficient interaction mode for users to manage and start experiments.

[0054] Reference is made to Figure 1 To be specific, the tool system for deep learning model training provided by the application is composed of four layers in the overall structure:

[0055] (1) Core component layer

[0056] To achieve low coupling and reusability of the code, the core component layer summarizes the key elements involved in the process of training deep learning models into four core components, namely algorithm model, data module, task module, and executor. Algorithm model specifically refers to deep learning algorithm model, which is implemented through a certain deep learning framework (e.g. PyTorch, TensorFlow); data module specifically refers to data-related definitions, mainly including data location, reading and preprocessing methods, and iterable data loaders; task module specifically refers to the definition of the process of a certain deep learning task, which corresponds to a certain training paradigm, such as common supervised learning and self-supervised learning. Task module can be further divided into several components, such as objective function module, optimization algorithm module, learning rate adjuster module, evaluation index module used in validation and testing, etc.; executor specifically refers to the relevant code for executing model training, validation, testing, inference, etc. These four types of core components can be freely combined to replace one or more elements, reducing redundant and repetitive code. The system provides abstract base classes for these core components, which can be imported by users for inheritance and specific implementation. Taking the executor as an example, its abstract interface defines abstract methods such as fit, validate, test, and predict, which are used to perform training, validation, testing, and prediction tasks, respectively.

[0057] At the same time, the core component layer provides multiple types of starting templates for users to choose directly from. These starting templates, based on specific deep learning frameworks or high-level encapsulation frameworks, implement basic data modules, task modules, and executors, and provide a basic directory structure to facilitate users to quickly start implementing deep learning models and training, without having to write a large amount of boilerplate code. All available starting templates are registered on a remote server, allowing users to retrieve and download them through the start command in the command interaction layer. When downloading, all files in the starting template are recursively downloaded from the remote server to the local machine through a network request. For users who need to maximize customization or whose needs cannot be met by predefined starting templates, they can directly inherit the abstract base classes of these core components and provide a specific implementation that meets their needs. As long as the abstract interfaces specified by the abstract base classes are implemented as needed, they can seamlessly integrate with other parts of the system.

[0058] Among them, the separation of abstract interface and specific implementation decouples the system from specific deep learning frameworks and their high-level encapsulation frameworks, such as PyTorch or PyTorch-Lightning, which can be chosen based on actual needs when creating a project.

[0059] In the present application, the core component layer decouples the key elements involved in the process of training deep learning models, so as to facilitate code reuse. The abstract interface provides high scalability, and the predefined startup template provides out-of-the-box convenience.

[0060] (2) Configuration system layer

[0061] There are quite a few solutions for configurable items in software engineering, but for deep learning projects, configurable items have the particularity of being numerous, frequently changed, and needing to be recorded. The present application is designed for the characteristics of deep learning projects and takes into account that the essence of configuration is to provide initialization parameters for instantiating classes. A configuration system with flexibility and ease of use is designed.

[0062] Unlike common configuration files such as XML (Extensible Markup Language), JSON (JavaScript Object Notation), or YAML (YAML Ain't a Markup Language), the configuration files in the present application are Python source code files that meet certain rules. Each configuration file contains one or more functional configuration items, i.e., each configuration item is a function that directly instantiates the class to be configured in the function and then returns it as the function's return value.

[0063] Corresponding to the four core components in the core component layer, there are mainly four types of core configuration items, namely algorithm model configuration items, data module configuration items, task module configuration items, and executor configuration items, which are used to configure models, data, tasks, and execution processes, respectively. These core configuration items can be used as modular and reusable configuration components in the configuration system to reduce the redundancy of configuration files.

[0064] The configuration system layer uses the decorator syntax in Python programming language to define four corresponding decorators for the four core components, which can be used to verify the parameters of the functional configuration items and mark the types of the configuration items. When conducting experiments, the corresponding configuration items of the four core components need to be combined to form a root configuration, which is the root node of all configuration items. When performing a certain task, only the path of the file where the root node is located needs to be specified, and the configuration system will read and dynamically import the source code module where the root configuration is located from the path, hold the functions after parsing, and call the corresponding functions at the appropriate time to obtain the configured objects.

[0065] The advantage of this configuration mode is that when the configuration item is written, the code prompt function of the IDE (Integrated Development Environment) can be fully used to complete the writing of the configuration file faster. For the aforementioned configuration files in formats such as XML, JSON, etc., when writing, it is equivalent to editing a normal text file, and when the configuration items are too many, it is very tedious and prone to errors to write, and additional time is needed to check the available values when configuring, and in these configuration files, only simple and limited logic can be implemented, and for the way of using source code files as configuration files, any logic can be implemented in the configuration file by using the programming language itself. In addition, compared with the configuration mode of using a Python source code file as a configuration file but directly taking the configurable items as global variables, the function form of the configuration item can allow the configuration system to decide to call the corresponding configuration item at the appropriate time.

[0066] In the present application, the configuration system layer also contains a configuration archiver for automatically archiving the configuration file and combining the configuration files that may exist in multiple configuration files into a single configuration file corresponding to a model training process, and performing basic processing such as merging and repeating import items in this process. The archived configuration file contains the value state of all configurable items during the corresponding model training, which helps to enhance the traceability and reproducibility of the experimental results.

[0067] By writing function type configuration items in the source code file, the present application can not only take full advantage of the code prompt function of the IDE (Integrated Development Environment) to set the configuration faster, but also can realize the reuse of the configuration items. At the same time, the matching configuration archiver can automatically archive the state of all configurable items.

[0068] (3) Experiment management layer

[0069] In order to efficiently manage experiments and prevent chaos when the number of experiments expands rapidly, the system introduces an experiment management layer, which is mainly used to impose organization rules on model training, validation, testing, reasoning and other jobs. Specifically, all experiment information is divided into three levels: Project, Experiment and Run, as follows:

[0070] Run is the smallest atomic unit corresponding to each run, and each Run must belong to a certain Experiment in a certain Project;

[0071] Experiment is a group of Runs with a common theme, such as a comparative experiment, an ablation experiment, etc., and each Experiment must belong to a certain Project.

[0072] Project is composed of multiple Experiments, and refers to a specific project.

[0073] The above three types of objects also have corresponding ID, name, description, creation time, storage path and other attribute information, which correspond to the corresponding entity classes in the system.

[0074] In the experiment management layer, different storage backends can be selected to store these experiment information, such as local JSON files, relational databases MySQL, document databases MongoDB, etc. Different storage backends can share their storage methods in different ways to achieve sharing of experiment information among teams. At the same time, the experiment management layer reserves corresponding interfaces to allow users to add support for different storage backends according to their own needs.

[0075] The experiment management layer also provides a runtime context, which can be used in custom task modules to obtain the current Run object and thus obtain the attribute information of the Run object, such as the directory corresponding to the current Run, which is very useful when some file output is needed during training.

[0076] The present application designs an experiment management method, and users can organize experiments according to a specific hierarchical structure through a command line interface, and the corresponding information of the experiment can be stored persistently through a variety of storage backends.

[0077] (4) Command interaction layer

[0078] The command interaction layer provides a command line interface used in the terminal, which can realize experiment management, model training and testing, etc. in the command line terminal through a small number of sub-commands. All sub-commands and their function descriptions are shown in the following table.

[0079]

[0080]

[0081] The present application first realizes a tool system for managing code, configurable items and experiments in deep learning projects from the software engineering level, which makes up for the poor scalability and insufficient flexibility of the platform-type technical solution, and achieves a good balance between convenience and flexibility.

[0082] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A tool system for training deep learning models, implemented using the Python programming language and used as a Python package, characterized in that... include: The core component layer is used to achieve low coupling and reusability of the code; The configuration system layer is used to set configurable items and to archive all configurable items corresponding to the experiment. The experimental management layer is used to organize all deep learning model training experiments according to a certain hierarchical structure, and to store experimental information. The command interaction layer, consisting of a limited number of subcommands, provides users with an efficient way to manage and launch experiments. The core component layer comprises four core components: algorithm model, data module, task module, and executor. The algorithm model refers to a deep learning algorithm model, implemented using a deep learning framework. The data module defines data-related aspects, including data location, reading and preprocessing methods, and an iterative data loader. The task module defines the process of a deep learning task, including an objective function module, optimization algorithm module, learning rate regulator module, and evaluation metrics module used for validation and testing. The executor refers to the code used to execute the model training, validation, testing, and inference processes. The configuration files in the configuration system layer are Python source code files that conform to certain rules; each configuration file contains one or more functional configuration items, that is, each configuration item is a function, in which the class to be configured is instantiated directly and then used as the function's return value; the core configuration items among the configurable items include: algorithm model configuration items, data module configuration items, task module configuration items, and executor configuration items, which are used to configure the model, data, task, and execution process, respectively; The experimental information is divided into three levels: Project, Experiment, and Run. A Run is the smallest atomic unit corresponding to each execution, and each Run must belong to a certain Experiment in a certain Project. An Experiment is a set of Runs with a common theme, including comparative experiments and ablation experiments. Each Experiment must belong to a Project. A project consists of multiple experiments and refers to a specific project; The Project, Experiment, and Run have corresponding ID, name, description, creation time, and storage path information.

2. The tool system for training deep learning models according to claim 1, characterized in that, The core component layer includes various types of startup templates; the startup templates are based on specific deep learning frameworks or their high-level encapsulation frameworks; the startup templates are registered on remote servers, and users can retrieve and download them through the start command in the command interaction layer. During download, all files in the startup template are recursively downloaded from the remote server to the local machine via network requests.

3. The tool system for training deep learning models according to claim 2, characterized in that, The core component layer includes four abstract base classes for core components, which users can import from the software package and then inherit and implement. For users who need to maximize customization or whose needs cannot be met by the predefined startup template, they can directly inherit these four abstract base classes for core components and provide a specific implementation that meets their needs. That is, they can implement the abstract interface specified by the abstract base class as needed, which can be seamlessly connected with other parts of the system.

4. The tool system for training deep learning models according to claim 1, characterized in that, The configuration system layer uses the decorator syntax in the Python programming language to define decorators for four core components. These decorators are used to validate the parameters of functional configuration items and mark the types of configuration items. During experiments, the configuration items for these four core components are combined to form a root configuration, which is the root node of all configurable items. When performing a certain task, only the path of the file containing the root node needs to be specified. The configuration system layer will read from that path, dynamically import the source code module containing the root configuration, parse it, hold these functions, and call the corresponding functions to obtain the configured objects.

5. The tool system for training deep learning models according to claim 1, characterized in that, The configuration system layer includes a configuration archiver, which automatically archives configuration files and combines configuration files that may exist in multiple configuration files into a single configuration file corresponding to a certain model training process, and merges duplicate import items in the process.

6. The tool system for training deep learning models according to claim 1, characterized in that, The experimental management layer includes a runtime context. In the custom task module, the current Run object can be obtained through this context, and the attribute information of the Run object can be obtained and used during the experiment.

7. The tool system for training deep learning models according to claim 1, characterized in that, The function description of the subcommand is as follows: The subcommand start: retrieves available startup templates and downloads the specified startup template; The subcommand `new`: creates a new pair of Project and Experiment. The subcommand add-exp: adds a new Experiment to the specified Project; The subcommand ls displays Project, Experiment, and Run in a table format in the terminal. Subcommand fit: Train the model on the training set; The subcommand `resume-fit` resumes the training process from any interruption. The subcommand validate: performs model evaluation on the validation set; The subcommand `test` evaluates the model on the test set. The subcommand `predict`: Uses the specified model to make predictions on the specified data.

Images