Deep learning model training integrated management system and method
By integrating Jupyter Notebook into the Kubernetes cluster, the module management, parameter configuration, and data comparison of the deep learning model training process are automated, solving the problems of chaotic module management, scattered parameters, difficulty in data comparison, and cumbersome project reuse, thereby improving work efficiency and user experience.
Patent Information
- Application Number
- CN202511708637.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-17
AI Technical Summary
The training process of deep learning models suffers from problems such as chaotic module management, scattered parameter configuration, difficulty in comparing experimental data, weak process control, and cumbersome engineering reuse, resulting in high maintenance costs, error-proneness, and low work efficiency.
Integrating Jupyter Notebook into a Kubernetes cluster environment enables module sorting, parameter versioning, structured storage of metric data, and automated process control through module dependency management, parameter configuration center, log database, and workflow management.
It reduces manual intervention, improves user experience, solves problems such as repeated module import, messy parameter distribution, difficulty in comparing training data, and cumbersome engineering cloning, and enhances the automation and scalability of the model training system.
Smart Images
Figure CN121541850A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a deep learning model training integrated management system and method. Background Technology
[0002] With the rapid development of deep learning technology, training deep learning models based on Python has become a core part of AI product development. However, there are currently many technical problems in the deep learning model training process: First, module management is chaotic. Python imports utility functions and data processing modules through the import statement. Each file needs to be imported repeatedly and its path dependency needs to be modified, resulting in high maintenance costs and a high risk of errors. Secondly, parameter configurations are scattered, with hyperparameters and environment variables scattered throughout the project. When making changes, multiple files need to be traversed, which can easily lead to errors in editing and is not conducive to the integrity of the programming environment. Furthermore, it is difficult to compare experimental data. The indicator data generated during model training lacks unified management, making it impossible to intuitively compare the performance differences between different versions of the model during training. In addition, the process control is weak. The model training steps (data preprocessing, data preparation, training, evaluation) often require manual repetition and cannot be automated to handle unnecessary repetitive operations, which affects work efficiency. Finally, project reuse is cumbersome. Copying an existing project requires either a full clone or manually selecting files to copy, lacking an effective selection mechanism to improve the ease of reuse.
[0003] To address this, we propose an integrated management system and method for training deep learning models. Summary of the Invention
[0004] The purpose of this invention is to provide an integrated management system and method for training deep learning models, in order to solve the problems existing in the background technology.
[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: A deep learning model training integration management method, running in a Kubernetes cluster environment and integrating Jupyter Notebook as an interactive development and training platform, is characterized by the following steps: S100: In response to user instructions, selectively copy existing project files or directory structures to create new training project projects; S200. Perform module dependency management on the training project, extract module import statements from the project's Python source files, sort and deduplicate modules according to frequency of occurrence and alphabetical order, and generate intermediate Python files to be stored in the central module repository. S300. Create or clone a model training workflow for the training project, the workflow including multiple steps in data acquisition, data processing, model training and model validation. S400: For each step of the workflow, configure hyperparameters and environment variables in the parameter configuration center in the form of a versioned table. S500. Execute the workflow, and during the execution, capture and structure-store the training metric data into the log database by creating a log collection template in the training code; S600. Based on the indicator data in the log database, generate indicator comparison charts between different training rounds or different workflows.
[0006] As a further limitation of the technical solution of the present invention, step S100 specifically includes: S110, Present the file and directory structure tree of the source project; S120: Receive the user's selection instruction for the target file or directory in the structure tree; S130. Copy the target file or directory selected by the user to the new project path.
[0007] As a further limitation of the technical solution of the present invention, step S300 specifically includes: S310. Receive a user's selection instruction for the range of execution steps in the workflow, wherein the range of execution steps includes a start step and a stop step; S320. Control the training process to execute only the steps between the start step and the end step.
[0008] As a further limitation of the technical solution of the present invention, in step S400, the parameter configuration center stores a parameter versioning table, and the same column of the table stores the values of the same parameter under different versions.
[0009] As a further limitation of the technical solution of the present invention, in step S500, the structured storage of training index data specifically means storing the data in an SQLite database.
[0010] As a further limitation of the technical solution of the present invention, in step S600, the indicator comparison chart includes a comparison chart of training loss curves and a comparison chart of model accuracy.
[0011] This invention also provides a deep learning model training integrated management system, which is deployed on a Kubernetes cluster and integrates Jupyter Notebook as an interactive development and training platform. The system includes: The project cloning unit is configured to respond to user instructions by presenting the file and directory structure tree of the existing project and receiving user selection instructions to copy the target file or directory selected by the user to a new project path to create a new training project. The module import management unit is configured to extract module import statements from the Python source files of the training project, sort and deduplicate the extracted modules according to their frequency of occurrence and alphabetical order, and generate an intermediate Python file based on the results and store it in the central module repository. The process management unit is configured to create, store, and execute a model training workflow that includes multiple steps in the training project, such as data acquisition, data processing, model training, and model validation. The parameter configuration centralized management unit is configured to store and manage the hyperparameters and environment variables of each step of the workflow in the form of a versioned table; The execution log hub unit is configured to capture and structure-store training metric data to a log database by creating a log collection template in the training code, and generate metric comparison charts based on the data.
[0012] The technical solution of this invention reduces the manual intervention required during model training, greatly improves the user experience, and solves problems such as frequent import of duplicate modules, messy parameter distribution, difficulty in comparing training data logs, and cumbersome project cloning. It also cleverly utilizes the elastic scalability of Kubernetes and the ease of installation and use of Jupyter, further enhancing the inherent characteristics of the model training system. Attached Figure Description
[0013] The present invention can be further illustrated by the non-limiting embodiments given in the accompanying drawings.
[0014] Figure 1 This is a schematic diagram of the main process of the present invention; Figure 2 This is a schematic diagram illustrating the process of establishing the engineering management and workflow of the present invention. Figure 3 This is a schematic diagram of the module import management process of the present invention. Detailed Implementation
[0015] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments. Example:
[0016] I. Implementation Environment: Kubernetes cluster environment; Jupyter Notebook provides a front-end and back-end development and runtime environment; II. Implementation steps of this method: S100, Training Project Creation: The system responds to user commands by selectively copying existing project files or directory structures to create new training project projects. Specifically, during implementation: S100, in response to user instructions, selectively copies existing project files or directory structures to create new training project projects, specifically including: S110, Present the file and directory structure tree of the source project; S120: Receive user selection instructions for target files or directories in the structure tree; S130. Copy the target file or directory selected by the user to the new project path; S200. Perform module dependency management on the training project, extract module import statements from the project's Python source files, sort and deduplicate modules according to frequency of occurrence and alphabetical order, and generate intermediate Python files to be stored in the central module repository. S300. Create or clone a model training workflow for a training project. The workflow includes multiple steps in data acquisition, data processing, model training, and model validation, specifically including: S310. Receive the user's selection instruction for the scope of execution steps in the workflow. The scope of execution steps includes the start step and the end step. S320, Control the training process to execute only the steps between the start step and the end step; S400: For each step of the workflow, hyperparameters and environment variables are configured in the parameter configuration center in the form of a versioned table. The parameter configuration center stores the parameter versioned table, and the same column of the table stores the value of the same parameter under different versions. S500 executes the workflow, and during the execution process, it captures and structures the training metric data into the log database by creating a log collection template in the training code. Specifically, the structured storage of training metric data involves storing the data in an SQLite database. S600: Based on the indicator data in the log database, generate indicator comparison charts between different training rounds or different workflows. The indicator comparison charts include a comparison chart of the generated training loss curves and a comparison chart of the model accuracy. Specifically, the process of establishing project management and workflows is as follows: Figure 2 As shown, when creating a training project: ① First, based on the file and directory structure of the source project, if it is necessary to create a project based on the existing project files or directory structure, then selective copying (project cloning) is performed; otherwise, a new project is created directly to establish a model training project. ② Based on the established model training project, a workflow is created. A model training workflow is created or cloned for the training project. When cloning a workflow, the user is given the start and end steps of the execution steps in the workflow, so that the training process only executes the steps between the start and end steps. ③ Configure hyperparameters and environment variables in the parameter configuration center in the form of a versioned table. The parameter configuration center stores the parameter versioned table, and the same column of the table stores the value of the same parameter under different versions. ④ Execute the workflow, and during the execution process, capture and structure the training metric data into the log database by creating a log collection template in the training code; ⑤ Based on the metric data in the log database, generate metric comparison charts between different training rounds or different workflows.
[0017] Meanwhile, regarding the module import management process, such as Figure 2 As shown: ① For existing or newly created Python files, extract the module import statements; ② Sort and remove duplicates from the extracted modules according to their frequency of occurrence and alphabetical order, and generate an intermediate Python file by renaming the files based on the results; ③ Store the module in the central module repository; ④ When creating a new Python file, you can search for and reuse it through the central module repository.
[0018] III. Implementation System: The system is deployed on a Kubernetes cluster and integrates Jupyter Notebook as an interactive development and training platform, specifically including: The project cloning unit is configured to respond to user instructions by presenting the file and directory structure tree of the existing project and receiving user selection instructions to copy the target file or directory selected by the user to a new project path to create a new training project. The module import management unit is configured to extract module import statements from the Python source files of the training project, sort and deduplicate the extracted modules according to their frequency of occurrence and alphabetical order, and generate an intermediate Python file based on the results and store it in the central module repository. The workflow management unit is configured to create, store, and execute model training workflows that include multiple steps such as data acquisition, data processing, model training, and model validation within a training project. The parameter configuration centralized management unit is configured to store and manage hyperparameters and environment variables of each step of the workflow in the form of versioned tables; The execution log hub unit is configured to capture and structure-store training metric data to a log database by creating a log collection template in the training code, and generate metric comparison charts based on the data.
[0019] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.
Claims
1. A deep learning model training integration management method, running in a Kubernetes cluster environment and integrating Jupyter Notebook as an interactive development and training platform, characterized in that: The method includes the following steps: S100: In response to user instructions, selectively copy existing project files or directory structures to create new training project projects; S200. Perform module dependency management on the training project, extract module import statements from the project's Python source files, sort and deduplicate modules according to frequency of occurrence and alphabetical order, and generate intermediate Python files to be stored in the central module repository. S300. Create or clone a model training workflow for the training project, the workflow including multiple steps in data acquisition, data processing, model training and model validation. S400: For each step of the workflow, configure hyperparameters and environment variables in the parameter configuration center in the form of a versioned table. S500. Execute the workflow, and during the execution, capture and structure-store the training metric data into the log database by creating a log collection template in the training code; S600. Based on the indicator data in the log database, generate indicator comparison charts between different training rounds or different workflows.
2. The deep learning model training ensemble management method according to claim 1, characterized in that: Step S100 specifically includes: S110, Present the file and directory structure tree of the source project; S120: Receive the user's selection instruction for the target file or directory in the structure tree; S130. Copy the target file or directory selected by the user to the new project path.
3. The deep learning model training ensemble management method according to claim 1, characterized in that: Step S300 specifically includes: S310. Receive a user's selection instruction for the range of execution steps in the workflow, wherein the range of execution steps includes a start step and a stop step; S320. Control the training process to execute only the steps between the start step and the end step.
4. The deep learning model training ensemble management method according to claim 1, characterized in that: In step S400, the parameter configuration center stores a parameter versioning table, and the same column of the table stores the values of the same parameter under different versions.