Large model training system and method, electronic equipment and storage medium

By introducing standardized data processing, model training and log processing specifications into the large model training system, combined with the scheduling function of the plug-in management module, the problems of poor flexibility and scalability in the existing technology are solved, customized model training is realized, and user experience is improved.

CN120181259APending Publication Date: 2025-06-20合肥智能语音创新发展有限公司

Patent Information

Application Number
CN202510116033.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-01-24
Publication Date
2025-06-20

AI Technical Summary

Technical Problem

The prior art develops adaptation modules for each training framework, or builds independent training systems for different frameworks, resulting in poor flexibility and scalability, making it difficult to implement customized model training, affecting the user experience.

Method used

It provides a large model training system, including data set preprocessing module, training module, log management module and plug-in management module. Through standardized data processing flow, model training process and log processing specifications, combined with user input configuration files, these modules are scheduled to realize customized model training.

Benefits of technology

It realizes a flexible and scalable model training system, which can be customized training according to user needs, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120181259A_ABST
    Figure CN120181259A_ABST
Patent Text Reader

Abstract

The invention provides a large model training system and method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a configuration file inputted by a user, and analyzing the configuration file; based on the analyzed configuration file, configuring the target plug-in; scheduling a data set preprocessing module, a training module and a log management module based on the configured target plug-in, and training a target large model based on the called module; the data set preprocessing module is used for preprocessing the training data set according to a preset standardized data processing flow specification; the training module is used for training a to-be-trained target large model based on a standardized training data set according to a preset standardized model training process specification; and the log management module is used for processing log data of the large model training system according to a preset standardized log processing specification. The method is high in flexibility and expandability, customized model training can be achieved, and user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a large model training system, method, electronic device, and storage medium. Background Art

[0002] In recent years, with the continuous growth of the scale and complexity of deep learning models, developers' demand for efficient and flexible training tools has been increasing. Existing technologies usually develop adaptation modules separately for each training framework or build independent training systems for different frameworks to meet specific task requirements, with high costs, poor flexibility and scalability. The existing one-stop training platforms also have the same problems and are difficult to achieve customized model training according to different user needs, affecting the user experience. Summary of the Invention

[0003] The present invention provides a large model training system, method, electronic device, and storage medium to solve the problems in the prior art of separately developing adaptation modules for each training framework or building independent training systems for different frameworks to meet specific task requirements, or using one-stop training platforms for model training, with poor flexibility and scalability.

[0004] The present invention provides a large model training system, including: A dataset preprocessing module for preprocessing a training dataset according to a preset standardized data processing flow specification to obtain a standardized training dataset; A training module for training a target large model to be trained based on the standardized training dataset according to a preset standardized model training flow specification to obtain a trained target large model; A log management module for processing log data of the large model training system according to a preset standardized log processing specification to obtain standardized log data; A plugin management module for scheduling the dataset preprocessing module, the training module, and the log management module based on a configuration file input by a user to achieve customized training of the target large model.

[0005] In some embodiments, the plugin management module is further configured to: Determine a modified target module, where the modified target module is obtained by adjusting and / or expanding the interface of the target module based on user requirements; the target module is selected from the dataset preprocessing module, the training module, and the log management module; Schedule the modified target module based on a configuration file input by a user to achieve customized training of the target large model.

[0006] In some embodiments, the plugin management module is further configured to: Obtain the current configuration file input by the user in real time; Dynamically adjust the training parameters of the target large model based on the current configuration file; Pass the training parameters to the called module.

[0007] In some embodiments, the dataset preprocessing module includes a plurality of first interfaces, and the plurality of first interfaces follow the standardized data processing flow specification; the training module includes a plurality of second interfaces, and the plurality of second interfaces follow the standardized model training flow specification; the log management module includes a plurality of third interfaces, and the plurality of third interfaces follow the standardized log processing specification; Correspondingly, scheduling the dataset preprocessing module, the training module, and the log management module includes: Scheduling the plurality of first interfaces, the plurality of second interfaces, and the plurality of third interfaces.

[0008] In some embodiments, the plurality of second interfaces include a dataset processing interface, an initialization interface, a start training interface, a training interface, an evaluation interface, a model saving interface, a progress reporting interface, and an error reporting interface; wherein: The dataset processing interface is used to annotate the standardized training dataset to obtain an annotated training dataset; the initialization interface is used to initialize the target large model; the start training interface is used to start the training process; the training interface is used to train the target large model based on the annotated training dataset; the evaluation interface is used to evaluate the trained target large model; the model saving interface is used to save the trained target large model; the progress reporting interface is used to feedback the current training progress; the error reporting interface is used to report training errors.

[0009] In some embodiments, the standardized data processing flow specification includes: supporting multiple data formats and multiple field mapping modes; the standardized log processing specification includes: managing log data through a log interface, supporting a custom log callback mechanism, the log data being structured log data, and the log data including context information related to the training process of the target large model.

[0010] The present invention also provides a large model training method, including: Obtain the configuration file input by the user, parse the configuration file to obtain the parsed configuration file; Configure the target plugin based on the parsed configuration file to obtain the configured target plugin; Schedule the dataset preprocessing module, training module, and log management module based on the configured target plugin, and train the target large model to be trained based on the called modules to obtain the trained target large model; Among them, the dataset preprocessing module is used to preprocess the training dataset according to the preset standardized data processing flow specification; the training module is used to train the target large model to be trained based on the standardized training dataset according to the preset standardized model training flow specification; the log management module is used to process the log data of the large model training system according to the preset standardized log processing specification.

[0011] In some embodiments, after configuring the target plugin based on the parsed configuration file to obtain the configured target plugin, it further includes: Schedule the modified target module based on the configured target plugin, and train the target large model based on the called module to obtain the trained target large model; Among them, the modified target module is obtained by adjusting and / or expanding the interface of the target module based on user requirements; the target module is selected from the dataset preprocessing module, training module, and log management module.

[0012] In some embodiments, the method further includes: Dynamically adjust the training parameters of the target large model based on the parsed configuration file; Pass the training parameters to the called module.

[0013] In some embodiments, the dataset preprocessing module includes multiple first interfaces, and the multiple first interfaces follow the standardized data processing flow specification; the training module includes multiple second interfaces, and the multiple second interfaces follow the standardized model training flow specification; the log management module includes multiple third interfaces, and the multiple third interfaces follow the standardized log processing specification; Correspondingly, the scheduling of the dataset preprocessing module, training module, and log management module includes: Schedule the multiple first interfaces, multiple second interfaces, and multiple third interfaces.

[0014] In some embodiments, the standardized data processing flow specification includes: supporting multiple data formats and multiple field mapping modes; the standardized log processing specification includes: managing log data through a log interface, supporting a custom log callback mechanism, the log data being structured log data, and the log data including context information related to the training process of the target large model.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the large model training method described in any of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the large model training method described in any of the above is implemented.

[0017] The large model training system, method, electronic device, and storage medium provided by the present invention parse the configuration file obtained from user input; configure the target plug-in based on the parsed configuration file; schedule the data set preprocessing module, training module, and log management module that follow the preset standardized specifications based on the configured target plug-in, and train the target large model based on the called modules. It has strong flexibility and scalability, can achieve customized model training, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic structural diagram of the large model training system provided by the embodiments of the present invention.

[0020] Figure 2 It is one of the schematic flowcharts of the large model training method provided by the embodiments of the present invention.

[0021] Figure 3 It is another schematic flowchart of the large model training method provided by the embodiments of the present invention.

[0022] Figure 4 It is yet another schematic flowchart of the large model training method provided by the embodiments of the present invention.

[0023] Figure 5 It is a schematic structural diagram of the electronic device provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, the "and / or" in the present invention represents at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0026] Currently, the mainstream open-source frameworks on the market have implemented independent parameter management logics, data processing flows, training flows, optimization methods, and log management methods, but lack a unified abstraction for the user layer. This differentiated design causes users to be deeply bound after selecting a framework and makes it difficult to flexibly switch between different frameworks. Even worse, they may need to repeat the development and debugging of code for different frameworks. If users want to collaborate across frameworks or switch frameworks according to different task requirements, they often need to pay a high learning and migration cost.

[0027] Existing solutions separately adapt each framework or build independent training systems for different frameworks to meet specific task requirements, and have the following defects: (1) High adaptation cost and long update cycle: Each framework has a unique parameter management logic, data format, and training process, resulting in developers having to adapt each framework separately, which not only consumes a large amount of human resources but also increases the complexity of the system. It is necessary to continuously update the adaptation code to follow the upgrade and changes of the framework version, increasing the maintenance cost.

[0028] (2) Difficulty in supporting diverse model training tasks: Different tasks (such as fine-tuning, reinforcement learning, self-supervised learning) have different requirements for models and training frameworks. To adapt to different tasks, additional development and debugging are required, and the workload is huge. With the rapid development of large models and the emergence of new frameworks, the adaptation work has increased exponentially, making it difficult to continuously support diverse model training tasks.

[0029] (3) Repeated development and resource waste: Each independent system often repeats the development of similar functional modules, such as data loading modules, optimizer setting modules, logging modules, etc., resulting in waste of resources; modules or general components cannot be shared among systems, reducing the code reuse rate.

[0030] (4) Poor flexibility and scalability: Independent systems are usually built around a specific framework. Once the framework is selected, it is difficult for users to flexibly switch or expand to other frameworks, restricting the flexibility and scalability of technologies.

[0031] (5) Poor user experience: Users need to learn different interfaces and operation processes for different frameworks, increasing the learning cost; when collaborating or migrating tasks across frameworks, they need to reconfigure the training environment and toolchain, resulting in reduced efficiency.

[0032] (6) Ecosystem limitations: During the training and fine-tuning of large models, different frameworks and platforms have formed their own technical ecosystems. This ecological closure and differential design bring many restrictions to users when using and switching frameworks.

[0033] Existing solutions use a one-stop training platform for large model training, which has the following defects: (1) Lack of flexibility: Users can only use the functions provided by the platform and cannot customize the training process or expand new functions according to specific needs. The platform often provides a standardized training pipeline, which executes in fixed steps from data preprocessing, model initialization to optimizer setting and model evaluation. If users want to change the process (such as introducing custom intermediate steps, special data augmentation methods or specific model pruning strategies), it is usually difficult to achieve.

[0034] Some platforms do not support users to freely insert custom modules or plugins, and users can only use the modules and tools provided by the official. For example, if users need to introduce a specific optimization algorithm (such as the latest adaptive optimizer) or special loss function, they need to wait for the platform to be updated.

[0035] In addition, the platform has strict restrictions on the configuration of training parameters and resources (such as memory allocation, graphics processor selection, distributed training strategy, etc.), and users cannot flexibly adjust the resource usage strategy.

[0036] (2) Platform lock-in risk: Users are tied to a specific platform, making it difficult to migrate or collaborate with other tools. Platforms typically use proprietary data formats and storage methods. For example, some cloud platforms store data in specific cloud storage servers, and these data formats or application programming interfaces (APIs) can only be used on that platform. When users migrate data, complex data conversion and download operations are required, and some metadata or training configurations may even be lost. Some platforms use specific model export formats, making it difficult to directly load and use them on other frameworks or platforms. For example, after a model is trained on a certain platform, it cannot be exported in a common format.

[0037] To this end, the embodiments of the present invention provide a large model training system, method, electronic device, and storage medium. By obtaining a configuration file input by a user, parsing the configuration file; based on the parsed configuration file, configuring a target plug-in; based on the configured target plug-in, scheduling a data set preprocessing module, a training module, and a log management module that follow a preset standardization specification, and training a target large model based on the called modules, which has strong flexibility and scalability, can achieve customized model training, and improve the user experience.

[0038] Figure 1 It is a schematic structural diagram of the large model training system provided by the embodiments of the present invention. As Figure 1 shown, the large model training system 100 includes: A data set preprocessing module 110, configured to preprocess a training data set according to a preset standard data processing flow specification to obtain a standardized training data set; A training module 120, configured to train a target large model to be trained based on the standardized training data set according to a preset standard model training flow specification to obtain a trained target large model; A log management module 130, configured to process the log data of the large model training system according to a preset standard log processing specification to obtain standardized log data; A plug-in management module 140, configured to schedule the data set preprocessing module 110, the training module 120, and the log management module 130 based on a configuration file input by a user to achieve customized training of a target large model.

[0039] Among them, the standard data processing flow specification defines that the function of the data set preprocessing module 110 is to preprocess the training data set, the input is the training data set, and the output is the standardized training data set.

[0040] Among them, the standardized model training process specification defines the function of the training module 120 as executing one training step, with the input being a single batch of standardized training data and the output being the training loss and metrics.

[0041] Among them, the standardized log processing specification provides a general log collection mechanism that can span different deep learning frameworks and training processes, while meeting the requirements of high efficiency, scalability, and ease of use, ensuring that the training process is observable, traceable, and debuggable.

[0042] It should be noted that the standardized data processing process specification, the standardized model training process specification, and the standardized log processing specification are formulated based on the core processes abstracted from the general training process.

[0043] Optionally, the core processes abstracted from the general training process at least include: Preprocessing stage: cleaning, transforming, and standardizing the data; Standard actions: data loading, data augmentation, data splitting, data standardization.

[0044] Model construction and initialization: defining the model, initializing the parameters, and constructing the model; Standard actions: model loading, model parameter initialization, loss function definition, optimizer configuration.

[0045] Training stage: training the model, updating the parameters, and calculating the loss; Standard actions: starting training, model training (iterative optimization), model evaluation (such as accuracy calculation).

[0046] Tuning stage: adjusting the training process based on the evaluation results, such as hyperparameter tuning, regularization, etc.; Standard actions: hyperparameter tuning, pruning, training strategy adjustment (such as learning rate decay).

[0047] Model verification and export: verifying the model performance through the validation set and finally exporting the model for inference or deployment; Standard actions: model verification, model saving, model export.

[0048] Among them, the configuration file includes, but is not limited to, information such as training tasks, training parameters, log levels, log storage paths, etc.

[0049] It should be noted that by abstracting the general training process and defining a series of standardized training actions and auxiliary capabilities, the functions of different training tools can be unified and standardized.

[0050] Optionally, the dataset preprocessing module 110 is used to preprocess the training dataset, such as cleaning, transforming, and standardizing it, according to the preset standardized data processing process specification, to obtain a standardized training dataset.

[0051] In some embodiments, the standardized data processing flow specification includes: supporting multiple data formats and multiple field mapping modes; the standardized log processing specification includes: managing log data through a log interface, supporting a custom log callback mechanism, the log data being structured log data, and the log data including context information related to the training process of the target large model.

[0052] Optionally, the standardized data processing flow specification supports multiple file formats and field formats.

[0053] Optionally, the standardized data processing flow specification supports selecting a field mapping method, such as manual mapping, AI-assisted mapping, or no mapping.

[0054] Among them, the structured log data includes multiple fields such as a timestamp, log level (such as INFO, WARNING, ERROR), component name (i.e., identifying the log source module), training metrics (such as loss, accuracy), etc.

[0055] For example, the following is the log data generated during the training phase: “{ "timestamp": "2024-11-27T10:00:00Z", "level": "INFO", "module": "train_step", "metrics": { "loss": 0.25, "accuracy": 0.95 }}”。

[0056] Optionally, the standardized log processing specification further includes: supporting outputting logs to the console, file system, or remote log service terminal; users can specify the log level and storage path through a configuration file; supporting real-time log recording and analysis during the training process, and supporting asynchronous recording and batch processing.

[0057] For example, the training framework can asynchronously write log data to the log system at the end of each epoch or batch, and at the same time push important training events (such as model accuracy, loss function, etc.) to the monitoring system through real-time streaming logs.

[0058] It should be noted that each training framework no longer directly interacts with the file system or the cloud. Instead, it only needs to output logs by calling the logging interface without caring about the underlying implementation. The logs are structured and can capture context information related to large model training, such as the large model name, training stage, hyperparameters, metrics, etc. Structured log data can ensure the readability and searchability of the log content, facilitating subsequent analysis and processing, especially in complex distributed training environments. By adopting a custom log callback mechanism, log records or alerts can be triggered based on specific events.

[0059] It can be understood that through structured logs, real-time processing, context awareness, and multi-level configuration, the observability and transparency of the training process can be enhanced. It can efficiently collect and store logs while ensuring scalability, security, and ease of use, which helps to quickly diagnose problems and optimize the training process in large-scale, distributed training tasks.

[0060] In some embodiments, the plugin management module 140 is further configured to: Determine the modified target module, where the modified target module is obtained by adjusting and / or expanding the interfaces of the target module based on user requirements. The target module is selected from the dataset preprocessing module 110, the training module 120, and the log management module 130; Schedule the modified target module based on the configuration file input by the user to achieve customized training of the target large model.

[0061] Among them, the modified target module can be a modified dataset preprocessing module, a modified training module, or a modified log management module.

[0062] Optionally, the modified target module includes existing interfaces and modified interfaces. For example, if the target module is the training module 120, the modified training module includes existing dataset processing interfaces, initialization interfaces, start training interfaces, training interfaces, as well as modified evaluation interfaces, model saving interfaces, progress reporting interfaces, and error reporting interfaces.

[0063] Optionally, the plugin management module 140 is further configured to schedule the existing interfaces and modified interfaces of the modified target module based on the configuration file input by the user to achieve customized training of the target large model.

[0064] For example, call the existing initialization interface, start training interface, training interface, as well as the modified evaluation interface and model saving interface in the modified training module to train the target large model.

[0065] Optionally, the plugin management module 140 is further configured to schedule the modified target module and other unmodified modules based on the configuration file input by the user, so as to implement the customized training of the target large model. For example, call the modified training module, the dataset preprocessing module 110, and the log management module 130, or call the training module 120 and the modified log management module.

[0066] In some embodiments, the plugin management module 140 is further configured to: Obtain the current configuration file input by the user in real time; Dynamically adjust the training parameters of the target large model based on the current configuration file; Pass the training parameters to the called modules.

[0067] It should be noted that the plugin management module 140 supports flexible parameter configuration and passing methods to ensure that the parameters are reusable and easy to manage; it supports dynamically adjusting parameters during the training process, such as the learning rate, gradient clipping threshold, etc.

[0068] For example, based on the current configuration file, the following parameter configuration information is obtained: “pre_trained_model: type: "soark_tiny" hidden_size: 768 training: batch_size: 4 learning_rate: 0.0016”.

[0069] In some embodiments, the dataset preprocessing module 110 includes multiple first interfaces, and the multiple first interfaces follow the standardized data processing flow specification; the training module 120 includes multiple second interfaces, and the multiple second interfaces follow the standardized model training flow specification; the log management module 130 includes multiple third interfaces, and the multiple third interfaces follow the standardized log processing specification; Correspondingly, scheduling the dataset preprocessing module 110, the training module 120, and the log management module 130 includes: Scheduling the multiple first interfaces, the multiple second interfaces, and the multiple third interfaces.

[0070] Optionally, call multiple target interfaces among the multiple first interfaces, the multiple second interfaces, and the multiple third interfaces for large model training.

[0071] Optionally, based on multiple tasks, perform parallel scheduling on the multiple first interfaces, the multiple second interfaces, and the multiple third interfaces.

[0072] Optionally, determine the priorities of the multitasks, and schedule the multiple first interfaces, multiple second interfaces, and multiple third interfaces according to the priorities of the multitasks.

[0073] Optionally, monitor the multitasks based on the task monitoring interface, and provide real-time feedback on the task status and progress.

[0074] In some embodiments, the multiple second interfaces include a dataset processing interface, an initialization interface, a start training interface, a training interface, an evaluation interface, a model saving interface, a progress reporting interface, and an error reporting interface; where: The dataset processing interface is used to annotate the standardized training dataset to obtain the annotated training dataset; the initialization interface is used to initialize the target large model; the start training interface is used to start the training process; the training interface is used to train the target large model based on the annotated training dataset; the evaluation interface is used to evaluate the trained target large model; the model saving interface is used to save the trained target large model; the progress reporting interface is used to provide feedback on the current training progress; the error reporting interface is used to report training errors.

[0075] Among them, the function of the dataset processing interface is to obtain the labels of the standardized training dataset, with the input being the name of the standardized training dataset and the output being the labels of the standardized training dataset; the function of the initialization interface is to initialize the large model and configuration parameters, with the input being the large model object, configuration file path, dictionary, log level, etc., and the output being the initialized large model object; the function of the start training interface is to start the training; the function of the training interface is to execute one training step, with the input being a single batch of training data and the output being the training loss and metrics; the function of the evaluation interface is to evaluate the performance of the large model, with the input being the data loader and the output being the evaluation metrics (such as accuracy, F1 score, etc.); the function of the model saving interface is to save the large model to the specified path, with the input being the file path and the output being the large model file; the function of the progress reporting interface is to report the training progress and provide feedback on the current training progress, with the input being the progress reporting callback function; the function of the error reporting interface is to report training errors, with the input being the error reporting callback.

[0076] In the embodiments of the present invention, through modular and standardized methods, the compatibility, scalability, and reusability of the training tools and frameworks are improved; through strict specification constraints, it is ensured that the training process can be executed stably, efficiently, and repeatedly in different tools and frameworks, and flexible and customizable model training can be achieved.

[0077] Figure 2 This is one of the flow diagrams of the large model training method provided by the embodiments of the present invention. As Figure 2 shown, a large model training method is provided, and this large model training method is applied to Figure 1Taking the large model training system as an example, it includes the following steps: Step 210, Step 220, and Step 230. The process steps of this method are only a possible implementation manner of the present invention.

[0078] Step 210: Obtain the configuration file input by the user, parse the configuration file, and obtain the parsed configuration file; Step 220: Configure the target plugin based on the parsed configuration file to obtain the configured target plugin; Step 230: Schedule the dataset preprocessing module 110, the training module 120, and the log management module 130 based on the configured target plugin, and train the target large model to be trained based on the called modules to obtain the trained target large model; Among them, the dataset preprocessing module 110 is used to preprocess the training dataset according to the preset standardized data processing process specification; the training module 120 is used to train the target large model to be trained based on the standardized training dataset according to the preset standardized model training process specification; the log management module 130 is used to process the log data of the large model training system according to the preset standardized log processing specification.

[0079] Among them, the target plugin includes module call logic code, and each target plugin represents a training framework; by standardizing the unique functions of different frameworks, seamless switching between multiple frameworks can be achieved.

[0080] It should be noted that the large model training system adopts a plug-in design architecture, supports dynamic extension of frameworks and tools through plug-in hot loading; through the plug-in mechanism, tools can flexibly load new functions or support new frameworks at runtime, avoiding the refactoring cost of traditional tools.

[0081] Optionally, the above method further includes registering and managing the target plugin, such as plugin initialization, plugin scheduling, and resource management.

[0082] Optionally, only the training module 120 is called to train the target large model.

[0083] Optionally, the training module 120 and the dataset preprocessing module 110 are called, or the training module 120 and the log management module 130 are called, or the dataset preprocessing module 110, the training module 120, and the log management module 130 are called to train the target large model.

[0084] In some embodiments, the standardized data processing flow specification includes: supporting multiple data formats and multiple field mapping modes; the standardized log processing specification includes: managing log data through a log interface, supporting a custom log callback mechanism, the log data being structured log data, and the log data including context information related to the training process of the target large model.

[0085] It can be understood that through the standardized data processing flow specification, one-key loading of different data formats is achieved, the complexity of data preprocessing is reduced, and diverse task requirements can be met; through the adoption of the standardized log processing specification, cross-frame consistency is achieved, facilitating the tracking and analysis of the training process, and facilitating the visualization and performance tuning of large-scale model training.

[0086] Figure 3 This is the second flow diagram of the large model training method provided by the embodiments of the present invention. As Figure 3 shown, the large model training method includes: configuration parsing, plugin registration, startup, dataset preprocessing, startup training, post-training processing, evaluation, and training end.

[0087] Optionally, multiple mechanisms such as error callback, progress callback, log callback, and label callback are adopted to optimize the training of the large model.

[0088] In some embodiments, after configuring the target plugin based on the parsed configuration file to obtain the configured target plugin, it further includes: Scheduling the modified target module based on the configured target plugin, and training the target large model based on the called module to obtain the trained target large model; wherein, the modified target module is obtained by adjusting and / or expanding the interface of the target module based on user requirements; the target module is selected from a dataset preprocessing module, a training module, and a log management module.

[0089] Optionally, the modified training module is called to train the target large model, or the modified training module, the dataset preprocessing module 110, and / or the log management module 130 are called to train the target large model.

[0090] Optionally, the modified training module, the modified dataset preprocessing module, and / or the modified log management module are called to train the target large model.

[0091] It should be noted that a mechanism for inserting custom nodes can be adopted in the modified target module to flexibly adjust or expand the operations at each stage.

[0092] In some embodiments, the above method further includes: Dynamically adjust the training parameters of the target large model based on the parsed configuration file; Pass the training parameters to the called module.

[0093] It can be understood that dynamically adjusting the training parameters of the target large model, such as the learning rate, batch size, etc., based on the parsed configuration file can adapt to different task requirements, significantly improve the training efficiency, optimize the training process, reduce unnecessary consumption of computing resources, and thus accelerate model convergence.

[0094] In some embodiments, the dataset preprocessing module 110 includes multiple first interfaces, and the multiple first interfaces follow the standardized data processing flow specification; the training module 120 includes multiple second interfaces, and the multiple second interfaces follow the standardized model training flow specification; the log management module 130 includes multiple third interfaces, and the multiple third interfaces follow the standardized log processing specification; Correspondingly, scheduling the dataset preprocessing module 110, the training module 120, and the log management module 130 includes: Schedule the multiple first interfaces, the multiple second interfaces, and the multiple third interfaces.

[0095] Optionally, the multiple second interfaces include a dataset processing interface, an initialization interface, a start training interface, a training interface, an evaluation interface, a model saving interface, a progress reporting interface, and an error reporting interface.

[0096] Among them, the dataset processing interface is used to label the standardized training dataset to obtain the labeled training dataset; the initialization interface is used to initialize the target large model; the start training interface is used to start the training process; the training interface is used to train the target large model based on the labeled training dataset; the evaluation interface is used to evaluate the trained target large model; the model saving interface is used to save the trained target large model; the progress reporting interface is used to feedback the current training progress; the error reporting interface is used to report training errors.

[0097] Figure 4 This is the third schematic diagram of the process of the large model training method provided by the embodiments of the present invention. As Figure 4As shown, the training module 120 includes: a data source interface (Data Source), a training interface (Train Preprocess), a dataset processing interface (Get Token), a start training interface (Start Train), a progress reporting interface (Report Progress), a post-training processing interface (Train Postprocess), an evaluation interface (Evaluate), and a final report interface (Final Report); the large model training method includes: uploading a training dataset based on the data source interface, training a target large model based on the training interface, annotating the training dataset based on the dataset processing interface, starting the training of the target large model based on the start training interface, evaluating the training process based on the evaluation interface and interacting with the start training interface, performing post-training processing based on the post-training processing interface, evaluating the performance of the target large model based on the evaluation interface, and obtaining a final model training report based on the final report interface.

[0098] In an embodiment of the present invention, by obtaining a configuration file input by a user, parsing the configuration file; based on the parsed configuration file, configuring a target plug-in; based on the configured target plug-in, scheduling a dataset preprocessing module, a training module, and a log management module that follow a preset standardized specification, and training a target large model based on the called modules, which has strong flexibility and scalability, can achieve customized model training, and improve the user experience.

[0099] Figure 5 The structural schematic diagram of the electronic device provided by the embodiment of the present invention is as Figure 5As shown in the figure, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communications interface 520, and the memory 530 complete communication with each other through the communication bus 540. The processor 510 may call the logical instructions in the memory 530 to execute the large model training method, which includes: obtaining a configuration file input by a user, parsing the configuration file to obtain a parsed configuration file; configuring a target plug-in based on the parsed configuration file to obtain a configured target plug-in; scheduling a data set preprocessing module, a training module, and a log management module based on the configured target plug-in, and training a target large model to be trained based on the called modules to obtain a trained target large model; where the data set preprocessing module is used to preprocess a training data set according to a preset standardized data processing flow specification; the training module is used to train the target large model to be trained based on a standardized training data set according to a preset standardized model training flow specification; the log management module is used to process the log data of the large model training system according to a preset standardized log processing specification.

[0100] In addition, when the logical instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the large model training method provided by the above-mentioned various methods. The method includes: obtaining a configuration file input by a user, parsing the configuration file to obtain a parsed configuration file; configuring a target plugin based on the parsed configuration file to obtain a configured target plugin; scheduling a data set preprocessing module, a training module, and a log management module based on the configured target plugin, and training a target large model to be trained based on the called modules to obtain a trained target large model; wherein, the data set preprocessing module is used to preprocess the training data set according to a preset standardized data processing flow specification; the training module is used to train the target large model to be trained based on the standardized training data set according to a preset standardized model training flow specification; the log management module is used to process the log data of the large model training system according to a preset standardized log processing specification.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large model training system, characterized in that: include: The data set preprocessing module is used to preprocess the training data set according to the preset standardized data processing flow specification to obtain a standardized training data set; The training module is used to train the target large model to be trained based on the standardized training data set according to the preset standardized model training process specification to obtain the trained target large model; The log management module is used to process the log data of the large model training system according to the preset standardized log processing specifications to obtain standardized log data; The plug-in management module is used to schedule the data set preprocessing module, training module and log management module based on the configuration file input by the user to achieve customized training of the target large model.

2. The large model training system according to claim 1, characterized in that: The plug-in management module is also used for: Determine a modified target module, wherein the modified target module is obtained by adjusting and / or expanding an interface of the target module based on user requirements; The target module is selected from the data set preprocessing module, the training module and the log management module; Based on the configuration file input by the user, the modified target module is scheduled to achieve customized training of the target large model.

3. The large model training system according to claim 1, characterized in that: The plug-in management module is also used for: Get the current configuration file entered by the user in real time; Based on the current configuration file, dynamically adjust the training parameters of the target large model; The training parameters are passed to the called module.

4. The large model training system according to claim 1, characterized in that: The data set preprocessing module includes a plurality of first interfaces, and the plurality of first interfaces follow the standardized data processing process specification; the training module includes a plurality of second interfaces, and the plurality of second interfaces follow the standardized model training process specification; the log management module includes a plurality of third interfaces, and the plurality of third interfaces follow the standardized log processing specification; Correspondingly, the scheduling of the data set preprocessing module, the training module and the log management module includes: The multiple first interfaces, the multiple second interfaces, and the multiple third interfaces are scheduled.

5. The large model training system according to claim 4, characterized in that: The plurality of second interfaces include a data set processing interface, an initialization interface, a training startup interface, a training interface, an evaluation interface, a model saving interface, a progress reporting interface, and an error reporting interface; wherein: The data set processing interface is used to label the standardized training data set to obtain the labeled training data set; the initialization interface is used to initialize the target large model; the start training interface is used to start the training process; the training interface is used to train the target large model based on the labeled training data set; the evaluation interface is used to evaluate the trained target large model; the model saving interface is used to save the trained target large model; the progress reporting interface is used to feedback the current training progress; the error reporting interface is used to report training errors.

6. The large model training system according to claim 1, characterized in that: The standardized data processing flow specifications include: supporting multiple data formats and supporting multiple field mapping modes; the standardized log processing specifications include: managing log data through a log interface and supporting a customized log callback mechanism. The log data is structured log data and includes contextual information related to the training process of the target large model.

7. A large model training method, characterized in that: include: Obtaining a configuration file input by a user, parsing the configuration file, and obtaining a parsed configuration file; Based on the parsed configuration file, configure the target plug-in to obtain the configured target plug-in; Based on the configured target plug-in, the data set preprocessing module, the training module and the log management module are scheduled, and the target large model to be trained is trained based on the called module to obtain the trained target large model; Among them, the data set preprocessing module is used to preprocess the training data set according to the preset standardized data processing process specifications; the training module is used to train the target large model to be trained based on the standardized training data set according to the preset standardized model training process specifications; the log management module is used to process the log data of the large model training system according to the preset standardized log processing specifications.

8. The large model training method according to claim 7, characterized in that: After configuring the target plug-in based on the parsed configuration file to obtain the configured target plug-in, the method further includes: Based on the configured target plug-in, the modified target module is scheduled, and based on the called module, the target large model is trained to obtain a trained target large model; The modified target module is obtained by adjusting and / or expanding the interface of the target module based on user needs; the target module is selected from the data set preprocessing module, training module and log management module.

9. The large model training method according to claim 7, characterized in that: The method further comprises: Based on the parsed configuration file, dynamically adjust the training parameters of the target large model; The training parameters are passed to the called module.

10. The large model training method according to claim 7, characterized in that: The data set preprocessing module includes a plurality of first interfaces, and the plurality of first interfaces follow the standardized data processing process specification; the training module includes a plurality of second interfaces, and the plurality of second interfaces follow the standardized model training process specification; the log management module includes a plurality of third interfaces, and the plurality of third interfaces follow the standardized log processing specification; Correspondingly, the scheduling of the data set preprocessing module, the training module and the log management module includes: The multiple first interfaces, the multiple second interfaces, and the multiple third interfaces are scheduled.

11. The large model training method according to claim 7, characterized in that: The standardized data processing flow specifications include: supporting multiple data formats and supporting multiple field mapping modes; the standardized log processing specifications include: managing log data through a log interface and supporting a customized log callback mechanism. The log data is structured log data and includes contextual information related to the training process of the target large model.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the large model training method as described in any one of claims 7 to 11.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the large model training method as described in any one of claims 7 to 11.

Citation Information

Patent Citations

  • Neural network interactive automatic training system and method

    CN112529167A

  • Deep learning model standardized training method, management system thereof and processing terminal

    CN113592017A

  • Visual large model fine tuning method and device

    CN117785177A

  • Management method and device of AI training platform, electronic equipment and storage medium

    CN118333122A

  • Artificial intelligence fraud management solution

    US20150032589A1

Cited By

  • Automatic training method and device of video AI algorithm model based on byte code enhancement

    CN120807515A

  • Automatic training method and device of video AI algorithm model based on bytecode enhancement

    CN120807515B

  • Large model training method and device based on segment scheduling, medium and electronic equipment

    CN121119175A