Python-based machine learning management system and method

Abstract

The invention discloses a Python-based machine learning management system and method, and the method comprises the steps: creating a warehouse which comprises a main log module and a training sub-warehouse; the training is divided into times, and each execution of the training process of machine learning is regarded as one-time training; all continuous trainings of the same AI form a group of complete trainings, and operation information of the group of complete trainings is put into the same warehouse; and recording all operation information of a group of complete training on a main log module of the warehouse, and respectively storing a specific training result of each training into a corresponding training sub-warehouse. According to the method, a warehouse concept is used, one-time complete training under the same AI model is put into the same warehouse, operation information of the training is stored, training sub-warehouses are created and specific training results are recorded on the basis of single-time training, and methods for recording the training results are unified, so that different developers can read and learn codes mutually, and the training efficiency is improved. The learning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence training, and in particular to a Python-based machine learning management system and method. Background Art

[0002] Machine learning, as a core component of the field of artificial intelligence, is the process by which computer programs learn from data experience to optimize their own algorithms and generate corresponding "intelligent" suggestions and decisions. In previous machine learning programs, different software developers provided different methods and formats for recording training results, and different methods and formats for resuming training. Different formats increased the cost of reading and learning codes between different developers. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a Python-based machine learning management system and method, which can store the format and method of generating data during training.

[0004] To this end, the technical solution of the present invention is: a machine learning management method based on Python, comprising the following steps:

[0005] 1) Create a warehouse, which contains the main log module and the training sub-warehouse;

[0006] 2) Training is divided into “times”, and each execution of the machine learning training process is regarded as “1 time” of training;

[0007] All consecutive trainings of the same AI constitute a complete training set, and the running information of a complete training set is placed in the same warehouse;

[0008] 3) Before each training, you need to access the main log module, obtain the unique identifier of this training from the main log module: the run ID, and then create a training sub-warehouse with the run ID;

[0009] 4) Record all the running information of a complete training set on the main log module of the warehouse, and store the specific training results of each training in the corresponding training sub-warehouse.

[0010] On the basis of the above scheme and as a preferred scheme of the above scheme: in the step 1), the standard storage format of the warehouse, the main log module, the training sub-warehouse, and the operation ID is defined.

[0011] On the basis of the above scheme and as a preferred scheme of the above scheme: in the step 4), the user sets the training result evaluation criteria according to the screening strategy, evaluates and screens each checkpoint in the training results to determine whether the checkpoint needs to be stored in the training sub-warehouse.

[0012] On the basis of the above scheme and as the preferred scheme of the above scheme: when training is interrupted, the current training results are stored in the corresponding training sub-warehouse, and the warehouse is closed and then exited; when training is resumed, the main log module is accessed, and the running ID corresponding to the interrupted training is obtained from the main log module, and training is continued. After the training is completed, the training results are stored in the corresponding training sub-warehouse, and stored in the same training sub-warehouse as the training records before the interruption.

[0013] On the basis of the above scheme and as a preferred scheme of the above scheme: a flag is set in the main log module. When an internal abnormality occurs in training, the flag is used to record the type of internal abnormality in training, and the starting point for resuming training is determined according to the abnormality type.

[0014] On the basis of the above scheme and as the preferred scheme of the above scheme: a mutex lock mark is set in the main log module, each training sub-warehouse corresponds to a mutex lock mark, each group of training processes can access the corresponding training sub-warehouse through a specific mutex lock mark, and different training processes can be trained in parallel.

[0015] On the basis of the above scheme and as a preferred scheme of the above scheme: the internal data of the warehouse can be output according to the preset interface format and connected to the visualization platform, and the training results can be managed on the visualization platform.

[0016] The second technical solution of the present invention is: a Python-based machine learning management system, comprising:

[0017] Warehouse, used to store complete training operation information and specific training results under the same AI model;

[0018] The warehouse includes a main log module and multiple training sub-warehouses. The main log module is used to assign a running ID and record the running information of the complete training;

[0019] Multiple training sub-warehouses correspond to each run ID and store the specific training results corresponding to each run ID;

[0020] The warehouse manager, which is used to open a connection to the warehouse;

[0021] The training result evaluator is used to evaluate and filter each checkpoint in the training results to determine whether the checkpoint needs to be stored in the training sub-warehouse.

[0022] On the basis of the above scheme and as a preferred scheme of the above scheme: an interrupt processing unit is also provided in the main log module, which is used to keep the data normally after receiving the interrupt command and exit after closing the warehouse.

[0023] On the basis of the above scheme and as a preferred scheme of the above scheme: a flag is also provided in the main log module for recording the internal abnormality type of the training and determining the starting point for resuming the training according to the abnormality type.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. Use the warehouse concept to put a complete training of the same AI model into the same warehouse, which not only stores the running information of the training, but also creates a training sub-warehouse based on a single training to record the specific training results. The method of recording training results is unified to facilitate different developers to read and learn codes from each other, thereby improving learning efficiency.

[0026] 2. Users can customize the evaluation criteria of the training result analyzer according to different screening strategies, evaluate and selectively store the training results, thereby obtaining different types of data, and through standardized data structure design, ensure efficient management and traceability of training data.

[0027] 3. A processing flow for training interruption and recovery is set up to allow manual interruption, accidental interruption and other exceptions to occur, and different processing methods are used according to the exception type. The node for resuming training can also be set according to the exception type. Interruption exception processing and training recovery fragment the training process. Training can be interrupted and resumed at any time, making the training process more flexible and efficient.

[0028] 4. Set a mutex lock mark in the main log module, and use the mutex lock mark to connect the training process and the training sub-warehouse, so that different threads and processes can train and store data in parallel, ensuring the efficiency and data consistency of multi-process or multi-threaded training. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

[0030] Figure 1 It is a structural block diagram of the present invention;

[0031] Figure 2 It is a structural diagram of the complete training of the present invention;

[0032] Figure 3 This is a workflow diagram for a single training session of the present invention;

[0033] Figure 4 This is a flow chart of training interruption and recovery of the present invention;

[0034] Figure 5 This is a flowchart of multi-process or thread parallel training of the present invention. DETAILED DESCRIPTION

[0035] See the attached drawings. The Python-based machine learning management system described in this embodiment includes: a warehouse, a warehouse manager, and a training result evaluator.

[0036] A warehouse is used to store the operation information and specific training results of the complete training under the same AI model; the warehouse includes a main log module and multiple training sub-warehouses, the main log module is used to assign a runtime ID (RTID) and record the operation information of the complete training; multiple training sub-warehouses correspond to each runtime ID and store the specific training results corresponding to each runtime ID.

[0037] The main log module is also provided with an interrupt processing unit, which is used to keep data normally after receiving an interrupt command and exit after closing the warehouse. The main log module is also provided with a flag bit, which is used to record the internal abnormal type of training and determine the starting point of resuming training according to the abnormal type.

[0038] The repository manager is used to open a connection to the repository; any Python program that accesses the repository will use the repository manager to open a connection to the repository.

[0039] The training result evaluator is used to evaluate and filter each checkpoint in the training results to determine whether the checkpoint needs to be stored in the training sub-warehouse. Users only need to design a custom training result evaluator according to the provided interface format to implement different strategy filtering of training results. The training result evaluator evaluates the results of each training by the accuracy or loss function value. The evaluator will decide whether to keep the training checkpoint and is highly decoupled.

[0040] When managing training results, it is compatible with existing artificial intelligence visualization platforms, such as tensorboard, which are used to display and manage training results.

[0041] Specifically, the Python-based machine learning management method is as follows:

[0042] 1) Create a warehouse, which contains the main log module and the training sub-warehouse;

[0043] 2) Training is divided into “times”, and each execution of the machine learning training process is regarded as “1 time” of training;

[0044] The batch number, epoch number, etc. can be set to no less than 0 for each training.

[0045] All continuous training of the same AI constitutes a complete training set, and the running information of a complete training set is placed in the same warehouse.

[0046] like Figure 2As shown, the RTID of the 10 training sessions is 0000~0009, and the starting epoch number and the ending epoch number are continuous, thus forming a complete training session, which is recorded in the same warehouse, including 10 training sub-warehouses from 0000~0009.

[0047] 3) When any Python program accesses the warehouse, it will use the warehouse manager to open a connection to the warehouse. Before each training, it is necessary to access the main log module of the warehouse, obtain the running ID of this training: 0009 from the main log module, and then create and access the training sub-warehouse with the running ID: 0009. Figure 3 shown.

[0048] 4) All operation information of a complete training set is recorded on the main log module of the warehouse, including operation ID, metadata, operation parameters, status information, etc., and the specific training results of each training are stored in the corresponding training sub-warehouse. The internal data of the warehouse can be output according to the preset interface format and connected to the visualization platform to manage the training results on the visualization platform.

[0049] In a training run, not every checkpoint obtained in the training is stored in the training sub-warehouse. Set up a user-definable training result evaluator. Users can set the evaluation criteria of the evaluator according to different screening strategies. Each checkpoint in the training result is evaluated and screened by the accuracy or loss function value. The evaluator will decide whether to retain the training checkpoint and is highly decoupled to determine whether the checkpoint needs to be stored in the training sub-warehouse. Users only need to design a customized training result evaluator according to the provided interface format to implement different strategy screening of training results.

[0050] Abnormal situations such as training interruption may occur during training, such as manual interruption (such as keyboard interruption), unexpected interruption (such as program crash), and internal training exceptions.

[0051] When training is manually or unexpectedly interrupted, the Python context manager and interrupt handling mechanism will store the current training results in the corresponding training sub-warehouse after receiving an interrupt command from the keyboard or other means, and then close the warehouse before exiting. Figure 4 shown.

[0052] When training is resumed, check whether the last training was interrupted due to an exception. If so, perform recovery processing, access the main log module, obtain the running ID corresponding to the interrupted training from the main log module, and continue training. After the training is completed, store the training results in the corresponding training sub-warehouse, and store them in the same training sub-warehouse as the training records before the interruption.

[0053] A flag is set in the main log module. When an internal exception occurs during training, the flag is used to record the type of internal exception during training, and the starting point of the restored training is determined based on the exception type, thereby ensuring the integrity of the training and the safe preservation of the training data. The training process is fragmented, and the training can be interrupted and resumed at any time.

[0054] like Figure 5 As shown, a mutex lock mark is set in the main log module, and each training sub-warehouse corresponds to a mutex lock mark. Each group of training processes can access the corresponding training sub-warehouse through a specific mutex lock mark, so that different threads or processes can train and store data in parallel, ensuring the efficiency and data consistency of multi-process or multi-threaded training.

[0055] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A Python-based machine learning management method, characterized by: The following steps are involved: 1) Create a warehouse, which contains the main log module and the training sub-warehouse; 2) Training is divided into "times", and each execution of the machine learning training process is regarded as "1" training; All consecutive trainings of the same AI constitute a complete training set, and the running information of a complete training set is placed in the same warehouse; 3) Before each training, you need to access the main log module, obtain the unique identifier of this training from the main log module: the run ID, and then create a training sub-warehouse with the run ID; 4) Record all the running information of a complete training set on the main log module of the warehouse, and store the specific training results of each training in the corresponding training sub-warehouse.

2. The Python-based machine learning management method according to claim 1, characterized in that: In the step 1), the standard storage format of the warehouse, the main log module, the training sub-warehouse, and the operation ID is defined.

3. The Python-based machine learning management method according to claim 1, characterized in that: In step 4), the user sets the training result evaluation criteria according to the screening strategy, and evaluates and screens each checkpoint in the training result to determine whether the checkpoint needs to be stored in the training sub-warehouse.

4. The Python-based machine learning management method according to claim 1, characterized in that: When training is interrupted, the current training results are stored in the corresponding training sub-warehouse, and the warehouse is closed and then exited; When the training is resumed, the main log module is accessed, and the running ID corresponding to the interrupted training is obtained from the main log module. The training is continued. After the training is completed, the training results are stored in the corresponding training sub-warehouse and stored in the same training sub-warehouse as the training records before the interruption.

5. The Python-based machine learning management method according to claim 1, characterized in that: A flag is set in the main log module. When an internal exception occurs during training, the flag is used to record the type of internal exception during training, and the starting point for resuming training is determined based on the type of exception.

6. The Python-based machine learning management method according to claim 1, characterized in that: A mutex lock mark is set in the main log module. Each training sub-warehouse corresponds to a mutex lock mark. Each group of training processes can access the corresponding training sub-warehouse through a specific mutex lock mark, and different training processes can be trained in parallel.

7. The Python-based machine learning management method according to claim 1, characterized in that: The internal data of the warehouse can be output according to the preset interface format and connected to the visualization platform to manage the training results on the visualization platform.

8. Python-based machine learning management system, characterized by: include: Warehouse, used to store complete training operation information and specific training results under the same AI model; The warehouse includes a main log module and multiple training sub-warehouses. The main log module is used to assign a running ID and record the running information of the complete training; Multiple training sub-warehouses correspond to each run ID and store the specific training results corresponding to each run ID; The warehouse manager, which is used to open a connection to the warehouse; The training result evaluator is used to evaluate and filter each checkpoint in the training results to determine whether the checkpoint needs to be stored in the training sub-warehouse.

9. The Python-based machine learning management system according to claim 8, characterized in that: The main log module is also provided with an interrupt processing unit, which is used to keep data normally after receiving an interrupt command and exit after closing the warehouse.

10. The Python-based machine learning management system according to claim 8, characterized in that: The main log module is also provided with a flag bit for recording the internal abnormality type of the training and determining the starting point for resuming the training according to the abnormality type.

Images