Large model elastic training method and system

Abstract

This application relates to a method and system for large-scale model elastic training. The method includes: real-time acquisition of training logs and cluster resource idle rate in a shared directory; extraction of checkpoint data from the training logs; determination of a candidate node set based on the checkpoint data and preset hardware resource allocation constraints; determination of a target resource allocation strategy based on the cluster resource idle rate and a preset threshold; determination of a target node set based on the target resource allocation strategy and the candidate node set; and initiation of a large-scale model training job based on the target node set. This application solves the problem of low cluster resource utilization caused by static resource allocation in related technologies, thereby improving the utilization rate of cluster resources.

Description

Technical Field

[0001] This application relates to the field of large model training, and in particular to methods and systems for elastic training of large models. Background Technology

[0002] As the scale of training parameters for large models grows ever larger, unprecedented challenges are being posed to the resource allocation of the underlying computing infrastructure. Against this backdrop, an efficient and reliable elastic training technique for large models will directly impact the utilization rate of cluster resources during the training process.

[0003] Existing distributed training methods for large models employ a static strategy, assuming that the number of nodes participating in the computation remains constant throughout the training period. The drawback of this approach is that any node change necessitates a restart from the most recent checkpoint, resulting in significant wasted computing power and low cluster resource utilization.

[0004] There is currently no effective solution to the problem of low cluster resource utilization caused by static resource allocation in related technologies. Summary of the Invention

[0005] This embodiment provides a large-scale model elastic training method and system to solve the problem of low cluster resource utilization caused by static resource allocation in related technologies.

[0006] Firstly, this embodiment provides a large-model elastic training method applicable to a large-model elastic training system; the system includes a pre-built shared directory, a sidecar container connected to the shared directory, a job manager connected to the sidecar container, a job controller connected to the job manager, and a job scheduler connected to the job controller. The method is applicable to the large-model elastic training system and includes:

[0007] Real-time acquisition of training logs and cluster resource idle rate in the shared directory; and extraction of checkpoint data from the training logs;

[0008] Based on the checkpoint data and preset hardware resource allocation constraints, a set of candidate nodes is determined;

[0009] Based on the cluster resource idle rate and the preset threshold, determine the target resource allocation strategy;

[0010] Based on the target resource allocation strategy and the candidate node set, a target node set is determined; and based on the target node set, a large model training job is started.

[0011] In some embodiments, the real-time acquisition of training logs in the shared directory includes:

[0012] The training logs are written to the shared directory using a pre-built main container;

[0013] The sidecar container obtains the training logs in real time by monitoring changes in the log files in the shared directory.

[0014] In some embodiments, determining the candidate node set based on the checkpoint data and preset hardware resource allocation constraints includes:

[0015] The total parallel scale is determined based on the parallel configuration parameters in the checkpoint data;

[0016] The candidate node set is determined based on the total parallel scale and the hardware resource allocation constraints.

[0017] In some embodiments, the hardware resource allocation constraints include hardware resource divisibility conditions and batch consistency conditions.

[0018] In some embodiments, determining the target resource allocation strategy based on the cluster resource idle rate and a preset threshold includes:

[0019] If the cluster resource idle rate is higher than the first threshold among the thresholds, the target resource allocation strategy is determined to be an expansion strategy.

[0020] If the cluster resource idle rate is lower than the second threshold among the thresholds, the target resource allocation strategy is determined to be a scaling-down strategy; wherein the first threshold is greater than the second threshold.

[0021] In some embodiments, determining the target node set based on the target resource allocation strategy and the candidate node set includes:

[0022] If the target resource allocation strategy is an expansion strategy, then the target node set is determined based on the pre-configured maximum node size;

[0023] If the target resource allocation strategy is a scaling-down strategy, then the target node set is determined based on the pre-configured minimum node size.

[0024] In some embodiments, initiating a large model training job based on the target node set includes:

[0025] The job controller assigns predefined labels to the target nodes corresponding to the target node set.

[0026] The current training job is terminated by the job scheduler, and the large model training job is started with the target node set.

[0027] In some embodiments, the large model training task is a training task for a personalized teaching large model;

[0028] The personalized teaching model is used for intelligent lesson preparation recommendations, automatic homework grading, or real-time Q&A dialogues.

[0029] Secondly, this embodiment provides a large model elastic training system, including a pre-built shared directory, a sidecar container connected to the shared directory, a job manager connected to the sidecar container, a job controller connected to the job manager, and a job scheduler connected to the job controller; the system is configured to implement the large model elastic training method described in the first aspect above.

[0030] In some embodiments, the system also includes a main container;

[0031] The main container is connected to the shared directory and is used to write training logs into the shared directory.

[0032] Compared with related technologies, the large-scale model elastic training method and system provided in this embodiment obtains training logs and cluster resource idle rates in a shared directory in real time; extracts checkpoint data from the training logs; determines a candidate node set based on the checkpoint data and preset hardware resource allocation constraints; determines a target resource allocation strategy based on the cluster resource idle rate and preset thresholds; determines a target node set based on the target resource allocation strategy and the candidate node set; and starts the large-scale model training job based on the target node set. By using the sidecar container to perceive the training status in real time and making intelligent scaling decisions based on the cluster resource idle rate, the method solves the problem of low cluster resource utilization caused by static resource allocation in related technologies, thereby improving the utilization rate of cluster resources.

[0033] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a structural block diagram of a large model elastic training system provided in one embodiment of this application;

[0036] Figure 2 This is a flowchart of a large model elastic training method provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating the determination of a target resource allocation strategy according to an embodiment of this application;

[0038] Figure 4 This is a flowchart illustrating the determination of a target node set according to an embodiment of this application;

[0039] Figure 5 This is a flowchart illustrating a method for selecting flexible jobs by a job manager according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of a large model elastic training system provided in an embodiment of this application.

[0041] In the diagram: 102, Shared Directory; 104, Sidecar Container; 106, Job Manager; 108, Job Controller; 110, Job Scheduler. Detailed Implementation

[0042] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0044] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Figure 1This is a structural block diagram of the large model elastic training system in this embodiment. For example... Figure 1 As shown, the terminal may include a shared directory 102, a sidecar container 104, a job manager 106, a job controller 108, and a job scheduler 110. The shared directory 102 provides training logs to the sidecar container 104. The sidecar container 104 receives the training logs, extracts checkpoint data, determines a candidate node set, and uploads the candidate node set to the job manager 106. The job manager 106 receives the candidate node set, determines the target node set, and sends a training job to the job controller 108. The job controller 108 receives the training job, tags the target nodes with resource preemption tags, and controls the job scheduler 110 to start a large model training job based on the target nodes. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0045] This embodiment provides a method for elastic training of large models. Figure 2 This is a flowchart of the large model elastic training method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0046] Step S210: Obtain the training logs and cluster resource idle rate in the shared directory in real time; and extract checkpoint data from the training logs.

[0047] Specifically, training logs can be obtained by the sidecar container monitoring file system event changes in the shared directory; alternatively, they can be forwarded to the sidecar container in real time for parsing via a log collection agent; or the training framework can actively write checkpoint events to a message queue for the sidecar container to subscribe to and consume. Cluster resource idle rate can be obtained through real-time collection via API calls; periodic queries can be performed by integrating a third-party monitoring system; or periodic reporting can be done through a custom resource collector. No restrictions are placed on the specific implementation methods for obtaining training logs and resource idle rates.

[0048] Step S220: Determine the candidate node set based on the checkpoint data and preset hardware resource allocation constraints.

[0049] Specifically, the candidate node set can be determined by traversing checkpoint data and verifying hardware resource constraints one by one; alternatively, it can be based on the node's GPU configuration, directly calculating and generating all candidate values ​​that satisfy the hardware resource allocation constraints; or it can combine historical scaling records and performance profiles to quickly filter within the constraint space using heuristic algorithms. Here, a node refers to each independently running computer or server in a cluster system. Each node can independently run its own service and collaborate with other nodes through a high-speed communication network. The specific method for deriving the candidate node set based on checkpoint data and hardware resource allocation constraints is not restricted here.

[0050] Step S230: Determine the target resource allocation strategy based on the cluster resource idle rate and the preset threshold.

[0051] Specifically, the target resource allocation strategy can be determined by directly comparing the current cluster resource idle rate with a preset static threshold; it can also be achieved by introducing a hysteresis mechanism or a state machine model; or it can be achieved by combining historical resource utilization trends and forecast information, dynamically adjusting the threshold, or using a machine learning model for intelligent judgment. No restrictions are placed on the specific method for making target resource allocation strategy decisions based on the cluster resource idle rate and threshold.

[0052] Step S240: Based on the target resource allocation strategy and the candidate node set, determine the target node set; and based on the target node set, start the large model training job.

[0053] Specifically, the target node set can be determined by directly selecting boundary values ​​from the candidate node set according to the target resource allocation strategy; alternatively, it can be determined by combining the real-time load and topology of each node in the cluster and calculating the node combination with the lowest overall cost through a cost model; or it can be determined by introducing priority or fairness strategies to arbitrate allocation among multiple candidate jobs. There are no restrictions on the specific method for selecting the target node set.

[0054] Through the above steps, training logs are collected in real time, and checkpoint data containing node server running status information is extracted. Based on preset hardware resource allocation constraints, a set of candidate nodes that meet these constraints is selected. The idle rate of key hardware resources such as GPUs and memory in the cluster is continuously monitored. Based on the idle rate, a scaling strategy for node servers is determined, and the final target node size is selected from the corresponding node servers based on the candidate node set. Finally, the job is restarted based on this size and the target node set, thereby achieving a precise match between resources and training requirements. This solves the problem of low cluster resource utilization caused by static resource allocation in related technologies, and improves the utilization rate of cluster resources.

[0055] The above steps are explained in detail below:

[0056] In some embodiments, the real-time acquisition of training logs in the shared directory in step S210 includes the following steps:

[0057] Training logs are written to a shared directory using a pre-built main container;

[0058] The sidecar container obtains training logs in real time by monitoring changes in log files in the shared directory.

[0059] Specifically, the main container, acting as the execution entity for the training task, continuously writes the state information generated during training to a shared directory in log form through its internal training framework. The sidecar containers, on the other hand, capture real-time updates to the log files via a file system event listening mechanism deployed on the shared directory, and extract key information according to predefined parsing rules. The shared directory, main container, and sidecar containers all run on node servers containing both CPUs and GPUs. The state information can include the loss value, evaluation metrics, and checkpoint storage paths for each iteration; the type of state information is not restricted.

[0060] This embodiment ensures the stability of the training process and provides real-time, reliable state input for flexible decision-making, thereby reducing system coupling and laying the foundation for the coherence and efficiency of the overall flexible training process.

[0061] In some embodiments, step S220, determining the candidate node set based on checkpoint data and preset hardware resource allocation constraints, includes the following steps:

[0062] Determine the total parallel scale based on the parallel configuration parameters in the checkpoint data;

[0063] The candidate node set is determined based on the total parallel scale and hardware resource allocation constraints.

[0064] Specifically, after each training iteration, the sidecar container extracts parallel configuration parameters from the checkpoint data of the training job, including pipeline model parallel size (PP), tensor model parallel size (TP), and context parallel size (CP), and combines them with expert parallel size (EP) to calculate the total parallel scale. .

[0065] There are multiple ways to calculate the total parallelism scale based on parallel configuration parameters. If a mathematical formula is used, the formula for calculating the total parallelism scale is:

[0066] ;

[0067] in, For the total parallel scale, For pipeline parallelism, For tensor parallelism, For context parallelism, For expert parallelism.

[0068] Within the minimum and maximum number of nodes configured for the job [ , Within ], the number of any candidate nodes A node must meet certain hardware resource allocation constraints to be included in the candidate node set.

[0069] This embodiment can automatically calculate the total parallel scale based on the parallel configuration in the checkpoints, and strictly screen out a set of candidate nodes that meet the hardware resource allocation constraints within a preset number of nodes, providing a reliable and feasible scale option for subsequent flexible decision-making.

[0070] In some embodiments, the hardware resource allocation constraints in step S220 include hardware resource divisibility conditions and batch consistency conditions.

[0071] Specifically, the hardware resource divisibility condition is the total number of GPUs in the candidate nodes. It must be divisible by the total parallel size. The expression for the divisibility condition of hardware resources is:

[0072] ;

[0073] in, The number of candidate nodes. This indicates the number of GPUs per node. This represents the total parallel scale.

[0074] The batch consistency condition is that the Global Batch Size (GBS) must be divisible by the product of the current data parallelism granularity and the Micro Batch Size (MBS). The expression for the batch consistency condition is:

[0075] ;

[0076] Where GBS stands for Global Batch Size, DP for Data Parallelism, and MBS for Micro Batch Size. The formula for calculating Data Parallelism (DP) is:

[0077] ;

[0078] in, The number of candidate nodes. This indicates the number of GPUs per node. For expert parallelism, This represents the total parallel scale.

[0079] Substituting the formula for calculating data parallelism into the expression for batch processing consistency conditions, we can expand to obtain:

[0080] ;

[0081] The range of the number of nodes traversed by the sidecar container [ , Within the [process], all nodes that simultaneously satisfy the hardware resource divisibility condition and the batch processing consistency condition are selected, forming a candidate node set that satisfies the parallel strategy and hardware resource allocation constraints. The checkpoint data and the corresponding candidate node set are then reported to the job manager.

[0082] This embodiment establishes a screening mechanism based on hardware resource allocation constraints, providing a high-quality candidate set for subsequent resource utilization-based decisions. This ensures that while improving resource utilization, elastic scaling always maintains a solid guarantee of training correctness.

[0083] In some embodiments, step S230 determines the target resource allocation strategy based on the cluster resource idle rate and a preset threshold, such as... Figure 3 As shown, it includes the following steps:

[0084] Step S231: If the cluster resource idle rate is higher than the first threshold in the threshold, determine the target resource allocation strategy as the expansion strategy.

[0085] Step S232: If the cluster resource idle rate is lower than the second threshold in the threshold, the target resource allocation strategy is determined to be a shrinkage strategy; wherein, the first threshold is greater than the second threshold.

[0086] Specifically, when the cluster resource idle rate consistently exceeds the first threshold (e.g., 10%), it indicates that the cluster has sufficient redundant computing power, and the system adopts an expansion strategy to improve training throughput. When the cluster resource idle rate consistently falls below the second threshold (e.g., 5%), it indicates that cluster resources are becoming strained, and the system activates a shrinkage strategy to prioritize resource supply for high-priority services. The first and second thresholds are used to determine the current state of cluster resources. When the idle rate exceeds the first threshold, it indicates that resource utilization is low and expansion is needed; when the idle rate falls below the second threshold, it indicates that cluster resources are currently strained and shrinkage is needed.

[0087] It is understood that the first and second thresholds can be adjusted according to the actual application scenario and are not limited to the listed fixed values. The job manager can determine the target resource allocation strategy based on the cluster resource idle rate, CPU usage, or GPU usage. There are no restrictions on the data types used to determine the target resource allocation strategy.

[0088] There are several ways to obtain the cluster resource idle rate. If a mathematical formula is used, the ratio of remaining resources to total resources is calculated. The formula for calculating the cluster resource idle rate is as follows:

[0089] ;

[0090] Where S is the cluster resource idle rate, B is the remaining resource amount, and A is the total resource amount.

[0091] In this embodiment, the system maps continuous cluster resource states to discrete, explicit scaling decisions, achieving a threshold-based bistable triggering mechanism. This balances the stability and efficiency of resource allocation in a dynamic environment.

[0092] In some embodiments, step S240 determines the target node set based on the target resource allocation strategy and the candidate node set, such as... Figure 4 As shown, it includes the following steps:

[0093] Step S241: If the target resource allocation strategy is an expansion strategy, then determine the target node set based on the pre-configured maximum node size;

[0094] Step S242: If the target resource allocation strategy is a scaling-down strategy, then determine the target node set based on the pre-configured minimum node size.

[0095] Specifically, first, the job manager receives the checkpoint data and candidate node set reported by the sidecar container and persists it to the database.

[0096] After that, as Figure 5As shown, when the target resource allocation strategy is a scaling strategy, the job manager prioritizes selecting a job that meets the following two conditions for scaling: First, the number of nodes currently used by the job has not reached its configured maximum number of nodes. Second, the number of nodes in the candidate node set is not less than the optimal number of nodes for scaling up the job. When a training job meets the above two conditions, it is scaled up.

[0097] When the target resource allocation strategy is a scaling-down strategy, the job manager selects the number of nodes equal to the pre-configured minimum number of nodes from the candidate node set as the target size, and selects the job that will release the most resources after scaling down from the currently running training jobs, and performs the scaling-down operation on it. This ensures that training tasks can still run with the minimum available resources when resources are scarce, while releasing as many computing resources as possible for high-priority services.

[0098] This embodiment ensures that all elastic scheduling decisions are based on the candidate node set, thereby avoiding training errors or performance degradation that may be introduced by arbitrarily changing the number of nodes while dynamically adjusting the resource scale, and achieving a balance between resource utilization and training process robustness.

[0099] In some embodiments, step S240, which involves initiating a large model training job based on the target node set, includes the following steps:

[0100] The job controller assigns predefined labels to the target nodes corresponding to the target node set.

[0101] The current training job is terminated using the job scheduler, and a large model training job is started with the target node set.

[0102] Specifically, such as Figure 6 As shown, firstly, the job controller calls the API server of an open-source container orchestration system (such as Kube-apiserver in a Kubernetes cluster) to assign a predefined resource tag to each node in the target node set. This tag identifies that these nodes have been preempted by the current training job, preventing other tasks from being scheduled to the same nodes, thus ensuring resource availability when the target node set restarts. Open-source container orchestration systems include, but are not limited to, Kubernetes, Docker Swarm, or Apache Mesos; the type of open-source container orchestration system is not restricted here.

[0103] Next, the job scheduler terminates the currently running training job and, based on the target number of nodes and the node label selection strategy determined by the job manager, starts a new set of training jobs. It configures these jobs to load model parameters and optimizer states from the latest checkpoint in shared storage and resumes training iterations. The job manager, job controller, and job scheduler run on node servers containing only CPUs.

[0104] This embodiment achieves decoupling and coordination between resource reservation and job reconstruction. The job controller uses a tag mechanism to statically bind and isolate resources, while the job scheduler is responsible for the dynamic start, stop, and state recovery of training instances. The two are coordinated through the job manager, avoiding training interruptions, repeated iterations, or convergence deviations caused by job restarts.

[0105] In some embodiments, the large model training job in step S240 is a training job for a personalized teaching large model.

[0106] A personalized teaching model is used for intelligent lesson preparation recommendations, automatic homework grading, or real-time Q&A dialogues.

[0107] Specifically, in the smart education platform, the personalized teaching model needs to support online inference services and offline model training tasks. Online inference services include intelligent lesson preparation recommendations, automatic homework grading, and real-time Q&A dialogues, while offline model training tasks are used to continuously optimize the model's ability to understand student learning, reason about knowledge graphs, and generate exercises.

[0108] This platform is typically deployed on a unified GPU cluster, with training and inference sharing computing resources. Due to the significant diurnal cycle of teaching activities, peak teaching hours occur during the day, with a large number of teachers and students concurrently accessing AI services, leading to high strain on GPU memory, computing units, and memory bandwidth. Conversely, nighttime is a low-demand period, with a sharp decrease in online requests, resulting in a large amount of idle computing power in the cluster. Therefore, the platform uses a sidecar container to obtain real-time training logs and cluster resource idle rates for the personalized teaching model, extracting checkpoint data for training the model from the logs. Based on the checkpoint data and preset hardware resource allocation constraints, a candidate node set for training the personalized teaching model is dynamically determined. Then, based on a comparison of the cluster resource idle rate with a preset threshold, a target resource allocation strategy (expansion or reduction) is determined. Finally, based on this strategy and the candidate node set, the target node size is determined, and the job is restarted accordingly. Therefore, during high-load periods during the day, the system automatically triggers training scaling down to reduce the number of training nodes and free up critical resources such as GPUs, ensuring the response latency and stability of online education services; during low-load periods at night, it automatically triggers training scaling up to increase the number of training nodes and make full use of idle computing power to improve training speed and overall cluster resource utilization.

[0109] In city-level video surveillance systems, AI platforms need to simultaneously run high-concurrency real-time video analysis and inference tasks as well as large-scale model training tasks in the background. The analysis and inference tasks include facial recognition, abnormal behavior detection, crowd density estimation, and cross-camera tracking. The large-scale model training task in the background is used to continuously optimize the visual model's ability to identify new types of security incidents.

[0110] This system typically employs an "edge-central cloud" collaborative architecture, with the central training cluster and video analytics service sharing a GPU resource pool. Due to the strong diurnal variation in pedestrian activity, daytime is peak time for security operations, with numerous cameras concurrently uploading high-definition video streams, consuming significant GPU memory and computing resources for inference services. At night, pedestrian traffic is sparse, video traffic drops by over 70%, inference load decreases dramatically, and the cluster experiences substantial idle computing power. Therefore, the system uses edge containers to collect real-time training logs and cluster resource idle rates of the large-scale video surveillance model, extracting checkpoint data from the logs for training the model. Based on this, a candidate node set for training the model is generated under hardware constraints. A scaling-up / scaling strategy is determined by combining resource idle rate and threshold values. After determining the target node size based on the strategy and candidate set, the training job is restarted. This enables automatic scaling down of training tasks during high-load daytime periods, proactively releasing computing resources and prioritizing the service level agreement (SLA) for real-time video analytics. During low-load periods at night, the training scale is automatically expanded to make full use of idle computing power for model fine-tuning and accelerate the convergence process. This significantly improves training efficiency and overall cluster resource utilization while ensuring the performance of critical business operations.

[0111] It is understood that large-scale model training tasks include, but are not limited to, training tasks for personalized teaching large-scale models, city-level video surveillance backend large-scale models, or content recommendation and generation models. There are no restrictions on the type of large-scale model training tasks here.

[0112] This embodiment effectively solves the common problem of resource contention and large utilization fluctuations between training and inference tasks under traditional static resource allocation, providing a general, automatic, and efficient resource management paradigm for various large model training scenarios with similar load characteristics. It ensures that the system is suitable for large-scale, long-running large model training tasks, effectively reducing operational complexity.

[0113] This embodiment also provides a large model elastic training system, including a pre-built shared directory, a sidecar container connected to the shared directory, a job manager connected to the sidecar container, a job controller connected to the job manager, and a job scheduler connected to the job controller; the system is configured to implement any one of the large model elastic training methods in the above embodiments.

[0114] In some embodiments, the large model elastic training system also includes a main container;

[0115] The main container, connected to the shared directory, is used to write training logs to the shared directory.

[0116] Specifically, the main container, as the core execution unit of the training task, is responsible for executing the training process. During training iterations, the main container writes the log files generated during training to a shared directory. The shared directory is mounted to both the main container and the sidecar container, ensuring bidirectional visibility of log content and read-write separation. The shared directory can be a persistent volume, a network file system, or a host path; there are no restrictions on the implementation method of the shared directory.

[0117] In this embodiment, the main container focuses solely on model training and log generation, without being aware of the scaling decision process. This avoids modifying the training code or the framework itself, significantly improving the system's maintainability and scalability.

[0118] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.

[0119] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0120] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0121] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for elastic training of large models, characterized in that, This method is applicable to large-scale model elastic training systems. The system includes a pre-built shared directory, a sidecar container connected to the shared directory, a job manager connected to the sidecar container, a job controller connected to the job manager, and a job scheduler connected to the job controller. Real-time acquisition of training logs and cluster resource idle rate in the shared directory; and extraction of checkpoint data from the training logs; Based on the checkpoint data and preset hardware resource allocation constraints, a set of candidate nodes is determined; The step of determining the candidate node set based on the checkpoint data and preset hardware resource allocation constraints includes: The total parallel scale is determined based on the parallel configuration parameters in the checkpoint data; The candidate node set is determined based on the total parallel scale and the hardware resource allocation constraints; Based on the cluster resource idle rate and the preset threshold, determine the target resource allocation strategy; Based on the target resource allocation strategy and the candidate node set, a target node set is determined; and based on the target node set, a large model training job is started.

2. The large-model elastic training method according to claim 1, characterized in that, The real-time acquisition of training logs in the shared directory includes: The training logs are written to the shared directory using a pre-built main container; The sidecar container obtains the training logs in real time by monitoring changes in the log files in the shared directory.

3. The large-model elastic training method according to claim 1, characterized in that, The hardware resource allocation constraints include hardware resource divisibility conditions and batch processing consistency conditions.

4. The large-model elastic training method according to claim 1, characterized in that, The step of determining the target resource allocation strategy based on the cluster resource idle rate and a preset threshold includes: If the cluster resource idle rate is higher than the first threshold among the thresholds, the target resource allocation strategy is determined to be an expansion strategy. If the cluster resource idle rate is lower than the second threshold among the thresholds, the target resource allocation strategy is determined to be a scaling-down strategy; wherein the first threshold is greater than the second threshold.

5. The large-model elastic training method according to claim 1, characterized in that, The step of determining the target node set based on the target resource allocation strategy and the candidate node set includes: If the target resource allocation strategy is an expansion strategy, then the target node set is determined based on the pre-configured maximum node size; If the target resource allocation strategy is a scaling-down strategy, then the target node set is determined based on the pre-configured minimum node size.

6. The large model elastic training method according to claim 1, characterized in that, The step of initiating a large model training job based on the target node set includes: The job controller assigns predefined labels to the target nodes corresponding to the target node set. The current training job is terminated by the job scheduler, and the large model training job is started with the target node set.

7. The large model elastic training method according to claim 1, characterized in that, The large model training task is a training task for a personalized teaching large model; The personalized teaching model is used for intelligent lesson preparation recommendations, automatic homework grading, or real-time Q&A dialogues.

8. A large-scale model elastic training system, comprising a pre-built shared directory, a sidecar container connected to the shared directory, a job manager connected to the sidecar container, a job controller connected to the job manager, and a job scheduler connected to the job controller; characterized in that, The system is configured to perform the steps of the large model elastic training method according to any one of claims 1 to 7.

9. The large-model elastic training system according to claim 8, characterized in that, The system also includes a main container; The main container is connected to the shared directory and is used to write training logs into the shared directory.

Images