Dynamic computing power scheduling optimization method for AGI large model distributed training

Abstract

The application discloses a dynamic computing power scheduling optimization method for AGI large model distributed training, which comprises the following steps: obtaining computing power state parameters of each training subtask during AGI large model distributed training, and binding the computing power state parameters with corresponding computing power nodes to construct a memory resident computing network state atlas. The memory resident degree of each training subtask is processed by the memory resident computing network state atlas to divide the memory resident boundary. The migration feasibility of the training subtask to be scheduled is determined according to the memory resident boundary to determine the strong resident subtask, and the strong resident subtask is subjected to replacement scheduling constraint. The training iteration process of the AGI large model is subjected to boundary keeping correction, and a preheating copy is established for a specified model slice to generate a boundary keeping scheduling strategy. The AGI large model distributed training is executed through the boundary keeping scheduling strategy, and the original resident inheritance relationship is kept in the training iteration process to output a dynamic computing power scheduling optimization result.

Description

Technical Field

[0001] This invention belongs to the field of AGI large model technology, and more specifically, relates to a dynamic computing power scheduling optimization method for distributed training of AGI large models. Background Technology

[0002] In scenarios where trillion-level AGI large models are trained jointly using tensor parallelism, pipeline parallelism, and parameter sharding, some training subtasks are not independent tasks that can be transferred at any time. Their corresponding model shards, activation caches, optimizer states, and gradient caches have been residing in the video memory of a specific graphics processor for a long time.

[0003] Currently, while existing technical solutions propose performing task migration or training batch resharding when nodes are overloaded, they do not address the memory residency boundary issue. Therefore, in practical engineering, once a forced migration is performed, it may be necessary to reload model shards, rebuild activation caches, and synchronize optimizer states, which can easily cause training pause times to exceed the original node overload time.

[0004] Therefore, existing technologies address the technical problem of model sharding, activation cache, optimizer state, and gradient cache residing in specific GPU memory for extended periods during the joint training of trillion-level AGI models using tensor parallelism, pipeline parallelism, and parameter sharding, thus forming a memory residency boundary. Their dynamic computing power scheduling methods directly perform task migration or training batch resharding when nodes are overloaded, which easily leads to model shard reloading, activation cache reconstruction, optimizer state resynchronization, and gradient cache rebinding, resulting in training pause times exceeding the original node overload duration. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to resolve the aforementioned deficiencies and propose a dynamic computing power scheduling optimization method for distributed training of large AGI models.

[0006] The present invention adopts the following technical solution.

[0007] The first aspect of this invention discloses a dynamic computing power scheduling optimization method for distributed training of large AGI models, the method comprising: Obtain the computing power status parameters of each training subtask during distributed training of the AGI large model, and bind the computing power status parameters to the corresponding computing power nodes to construct a state map of the memory-resident computing network. The memory residency dependency of each training subtask is classified and processed by the memory residency network state map to delineate the memory residency boundary. Based on the memory residency boundary, the migration feasibility of the training subtasks to be scheduled is determined to identify the strongly residency subtasks, and alternative scheduling constraints are applied to the strongly residency subtasks. Based on the results of the alternative scheduling constraints on the strongly resident subtasks, the training iteration process of the AGI large model is corrected by boundary preservation, and a warm-up copy is established for the specified model slices to generate a boundary preservation scheduling strategy. The boundary-preserving scheduling strategy is used to perform distributed training of AGI large models and maintain the original resident inheritance relationship during the training iteration process to output dynamic computing power scheduling optimization results.

[0008] Furthermore, the step of obtaining the computing power state parameters of each training subtask during the distributed training of the AGI large model, and binding the computing power state parameters with the corresponding computing power nodes to construct a memory-resident computing network state graph, includes: Obtain the execution order of training iterations in the distributed training of the AGI large model, extract multiple training subtasks according to the execution order of the training iterations, and record the tensor parallel group, pipeline parallel segment and parameter sharding group of each training subtask. Collect the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask, and determine the memory dwell intensity based on the memory occupied by each dwell position. The computing power status parameters include the tensor parallel group, pipeline parallel segment, parameter sharding group, model sharding residence position, activation cache residence position, optimizer state residence position, gradient cache residence position, and the memory occupied by each residence position.

[0009] Furthermore, the step of obtaining the computing power state parameters of each training subtask during the distributed training of the AGI large model, and binding the computing power state parameters with the corresponding computing power nodes to construct a memory-resident computing network state graph, also includes: Calculate the node load pressure of the computing nodes corresponding to the training subtasks, and construct the memory-resident computing network load state set by combining the memory occupancy intensity. Based on the memory-resident computing network bearer state set, the tensor parallel group boundary, pipeline boundary and parameter partition boundary corresponding to each training subtask are recorded, and the number of cross-boundary associations corresponding to the tensor parallel group boundary, pipeline boundary and parameter partition boundary is counted. The memory-resident computing network state set is merged according to the sequential numbering of the training subtasks, and combined with the cross-boundary association quantity to generate the memory-resident computing network state map.

[0010] Furthermore, the step of classifying the memory residency dependency of each training subtask using the memory residency network state map to delineate memory residency boundaries includes: Read the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask in the memory-resident network state map, and determine whether the training subtask is attached to each dwell position to obtain the memory-resident object attachment result. Based on the memory-resident object dependency results, combined with the memory-resident occupancy intensity and the number of cross-boundary associations, the residency dependency intensity of each training subtask is determined, and the training subtask is graded according to the residency dependency intensity. Based on the results of the level labeling of the training subtasks, the strongly resident subtasks are determined, and the migration pause risk corresponding to the strongly resident subtasks is calculated, so as to delineate the memory residence boundary according to the migration pause risk.

[0011] Furthermore, the step of determining the migration feasibility of the training subtasks to be scheduled based on the memory residency boundary to identify strongly resident subtasks, and imposing alternative scheduling constraints on the strongly resident subtasks, includes: The training subtasks to be scheduled are determined based on the memory residency boundary, and candidate receiving nodes corresponding to the training subtasks to be scheduled are established, so as to calculate the migration adaptation strength corresponding to the weak residency subtasks in the candidate receiving nodes. Based on the migration adaptation strength corresponding to the weakly resident subtask, the restricted migration inheritance strength of the resident subtask is calculated, and the alternative scheduling relief strength of the strongly resident subtask is determined according to the restricted migration inheritance strength. The alternative scheduling relief strength is used to determine whether the strongly resident subtask is prohibited from migration, and when the strongly resident subtask is determined to be prohibited from migration, an alternative scheduling constraint is applied to the strongly resident subtask.

[0012] Furthermore, based on the result of the alternative scheduling constraints applied to the strongly resident subtasks, the boundary-preserving correction is performed on the training iteration process of the AGI large model, and preheating copies are established for specified model slices to generate a boundary-preserving scheduling strategy, including: Obtain the graphics processor utilization, link congestion duration, and node energy consumption increase in the current training iteration, and perform different migration processes on the weak-resident subtask and medium-resident subtask respectively based on the graphics processor utilization, link congestion duration, and node energy consumption increase. Based on the results of different transfer processing on the weakly resident subtasks and the mediumly resident subtasks, the training batch carrying capacity corresponding to the strongly resident subtasks is corrected, and prediction replicas are established for the specified model shards according to the corrected training batch carrying capacity. The boundary-preserving scheduling strategy is obtained by uniformly solidifying the results of different migration processes performed on the weakly resident subtasks and the mediumly resident subtasks, as well as the corrected training batch carrying capacity and the preheating replicas.

[0013] Furthermore, the step of performing distributed training of the AGI large model through the boundary-preserving scheduling strategy, and maintaining the original resident inheritance relationship during the training iteration to output dynamic computing power scheduling optimization results, includes: AGI large model distributed training is performed according to the boundary preservation scheduling strategy, and after each training iteration, it is determined whether the model shards, activation cache, optimizer state and gradient cache maintain the original resident inheritance relationship to obtain the resident inheritance confirmation result. Extract the preheated copy of the model fragment corresponding to the strongly resident subtask from the resident inheritance confirmation result, determine whether the preheated copy has the smooth acceptance condition, and output the preheated copy acceptance determination result. The preheated copy acceptance determination result includes determining whether the preheated copy has smooth acceptance conditions and determining whether the preheated copy does not have smooth acceptance conditions.

[0014] Furthermore, the step of performing distributed training of the AGI large model through the boundary-preserving scheduling strategy, and maintaining the original resident inheritance relationship during the training iteration to output dynamic computing power scheduling optimization results, also includes: When the result of the preheating copy acceptance determination is that the preheating copy has smooth acceptance conditions, the strong-resident subtask is changed from prohibited migration to restricted migration, and the model fragment with the preheating copy is inherited in the corresponding acceptance node. If the result of the preheating copy acceptance determination is that the preheating copy does not have the smooth acceptance condition, then the boundary preservation correction will continue to be performed on the training iteration process of the AGI large model.

[0015] The second aspect of this invention discloses a dynamic computing power scheduling optimization device for distributed training of AGI large models, used to implement the dynamic computing power scheduling optimization method for distributed training of AGI large models as described in any one of the first aspects, the device comprising: The state graph construction module is used to obtain the computing power state parameters of each training subtask during the distributed training of the AGI large model, and bind the computing power state parameters with the corresponding computing power nodes to construct the state graph of the memory-resident computing network. The memory residency boundary delineation module is used to classify the memory residency dependency of each training subtask through the memory residency network state map, so as to delineate the memory residency boundary. The scheduling constraint module is used to determine the migration feasibility of the training subtasks to be scheduled based on the memory residency boundary, so as to identify the strongly resident subtasks and apply alternative scheduling constraints to the strongly resident subtasks. The scheduling strategy generation module is used to perform boundary preservation correction on the training iteration process of the AGI large model based on the result of the alternative scheduling constraints on the strongly resident subtasks, and to establish a warm-up copy for the specified model slices in order to generate a boundary preservation scheduling strategy. The optimization result output module is used to perform distributed training of AGI large models through the boundary-preserving scheduling strategy, and maintain the original resident inheritance relationship during the training iteration process to output dynamic computing power scheduling optimization results.

[0016] A third aspect of the present invention discloses a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described in the first aspect.

[0017] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0018] The beneficial effects of the present invention are as follows: Compared with the prior art, the present invention has the following advantages: (1) This invention obtains the computing power state parameters of each training subtask during distributed training of the AGI large model and binds the computing power state parameters to the corresponding computing power nodes to construct a memory-resident computing network state map. Then, the memory-resident computing network state map is used to classify the memory-resident dependency of each training subtask to delineate the memory-resident boundary. Subsequently, the migration feasibility of the training subtask to be scheduled is determined according to the memory-resident boundary to identify the strongly resident subtask and to apply alternative scheduling constraints to the strongly resident subtask. This increases the identification of the memory-resident boundary and the migration feasibility determination, fundamentally eliminating the technical defects of existing dynamic computing power scheduling methods that directly execute task migration or training batch resharding when nodes are overloaded, resulting in model shard reloading, activation cache reconstruction, optimizer state resynchronization and gradient cache rebinding, which in turn makes the training pause time longer than the original node overload duration.

[0019] (2) Based on identifying the memory residency boundary, this invention performs boundary-preserving correction on the training iteration process of the AGI large model and establishes a preheating copy for the specified model fragment to generate a boundary-preserving scheduling strategy. Finally, the distributed training of the AGI large model is executed through the boundary-preserving scheduling strategy, and the original residency inheritance relationship is maintained during the training iteration process to output the dynamic computing power scheduling optimization result. In the dynamic scheduling of computing and network integration, the in-boundary load reduction, inter-boundary slow migration and residency state inheritance mechanism are also added, so that the distributed training task of the AGI large model can maintain the collaborative optimization of computing power, network and energy consumption, while further avoiding the destruction of the existing memory residency relationship, and realizing dynamic computing power scheduling optimization without causing additional training pauses. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 This is a flowchart illustrating the dynamic computing power scheduling optimization method for distributed training of large AGI models provided by this invention.

[0022] Figure 2 This is a schematic diagram of the dynamic computing power scheduling optimization device for distributed training of AGI large models provided by the present invention. Detailed Implementation

[0023] The present application will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and should not be construed as limiting the scope of protection of the present application.

[0024] like Figure 1 As shown in one embodiment, a dynamic computing power scheduling optimization method for distributed training of AGI large models includes the following steps: Step S110: Obtain the computing power status parameters of each training subtask during the distributed training of the AGI large model, and bind the computing power status parameters with the corresponding computing power nodes to construct the memory-resident computing network state map.

[0025] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes the following steps in step S110: Step S111: Obtain the execution order of training iterations in the distributed training of the AGI large model, extract multiple training subtasks according to the execution order of training iterations, and record the tensor parallel group, pipeline parallel segment, and parameter sharding group of each training subtask.

[0026] Step S112: Collect the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask, and determine the memory dwell intensity based on the memory occupied by each dwell position.

[0027] The computing power status parameters include tensor parallel group, pipeline parallel segment, parameter sharding group, model sharding residence position, activation cache residence position, optimizer state residence position, gradient cache residence position, and the memory occupied by each residence position.

[0028] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention further includes the following steps in step S110: Step S113: Calculate the node load pressure of the computing nodes corresponding to the training subtasks, and construct the memory-resident computing network load state set by combining the memory occupancy intensity.

[0029] Step S114: Based on the memory-resident computing network bearer state set, record the tensor parallel group boundary, pipeline boundary, and parameter partition boundary corresponding to each training subtask, and count the number of cross-boundary associations corresponding to the tensor parallel group boundary, pipeline boundary, and parameter partition boundary.

[0030] Step S115: Merge the state set of the memory-resident computing network according to the order number of the training subtasks, and combine it with the number of cross-boundary associations to generate the state map of the memory-resident computing network.

[0031] In a specific embodiment, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes steps 1 to 5: Step 1: Construct the state graph of the AGI training memory-resident computation network.

[0032] The process of collecting data during the distributed training of AGI large-scale models includes tensor parallel groups, pipeline parallel segments, parameter shard groups, training subtask types, model shard residing locations, activation cache residing locations, optimizer state residing locations, gradient cache residing locations, GPU memory remaining capacity, computing node load status, network link carrying capacity status, and node energy consumption status. Each training subtask is then bound to its corresponding model shard, activation cache, optimizer state, gradient cache, and the computing node it resides in. Furthermore, the boundaries of the parallel group, pipeline segment, and parameter shard boundaries where these binding relationships are located are recorded, forming an AGI training GPU memory residing network state map. This includes the following sub-steps: Sub-step 1.1: Collect the training parallel structure and the basic objects of the sub-task.

[0033] Specifically, following the execution order of each training iteration, the following subtasks are extracted: forward computation subtask, backward gradient computation subtask, gradient synchronization subtask, parameter aggregation subtask, and checkpoint saving subtask. The tensor parallel group, pipeline parallel segment, and parameter slice group to which each training subtask belongs are recorded. For the same training subtask, its current execution node number, graphics processor number, training batch affiliation, and iteration round affiliation are also recorded, thus forming the basic object set of training subtasks.

[0034] Sub-step 1.2: Establish the binding relationship between the training subtask and the memory-resident object.

[0035] Specifically, based on the basic object set of the training subtasks, we continue to collect the model slice dwell positions, activation cache dwell positions, optimizer state dwell positions, and gradient cache dwell positions corresponding to each training subtask, and calculate their memory dwell intensity, expressed as: ; In the formula, Indicates the first The memory resident intensity of each training subtask, dimensionless; This indicates the memory usage of the model fragments, in gigabytes. This indicates that the active cache is using video memory, in gigabytes. This indicates the memory usage of the optimizer, in gigabytes. This indicates the video memory used by the gradient cache, in gigabytes. This indicates the remaining video memory capacity of the corresponding graphics processor, in gigabytes. This is a smoothing parameter for video memory availability, measured in gigabytes, with a value ranging from 0.5 to 1.0. It is used to avoid abnormally amplified calculation results when the remaining video memory capacity is close to zero, while not masking the true resident memory pressure. This formula characterizes the degree of video memory dependency of the subtask on the current graphics processor by the ratio between the total amount of resident video memory objects and the remaining video memory that can continue to be used.

[0036] Sub-step 1.3: Integrate computing power nodes, network links, and energy consumption carrying capacity status.

[0037] Specifically, based on the memory resident binding relationship set, the system collects the graphics processor utilization, task queue length, real-time power consumption, and temperature of the corresponding computing nodes, as well as the bandwidth margin, transmission latency, packet loss status, and congestion status of the corresponding network links. The overall load capacity of the nodes is then calculated, expressed as: ; In the formula, Indicates the first The overall load-bearing pressure of the nodes and links where each training subtask is located, dimensionless; This represents the graphics processor utilization rate; it is dimensionless and ranges from 0 to 1. Indicates the task queue length, in units of tasks; This indicates the transmission delay of the corresponding link, in milliseconds. This indicates the node's real-time power consumption, in watts. The utilization smoothing parameter is dimensionless and ranges from 0.03 to 0.05. This is a smoothing parameter for queue length, measured in units of queue lengths, with a value ranging from 2 to 4. This is a delay smoothing parameter, in milliseconds, with a value of 5-10. This is a power consumption smoothing parameter, measured in watts, with a value ranging from 50 to 80. This formula is used to convert computing power congestion, task waiting, link latency, and node power consumption into a single dimensionless stress result, used to determine whether the current location is approaching an unbearable state.

[0038] Sub-step 1.4: Define the boundaries of the parallel group, the pipeline section, and the parameter partitioning.

[0039] Specifically, based on the memory-resident computing network state set, the boundaries of the tensor parallel group, pipeline, and parameter sharding for each training subtask are recorded, and the number of cross-boundary connections is counted. For the same training subtask, if its model shards, activation cache, optimizer state, and gradient cache are all located within the same GPU or the same parallel group, its boundary connection is low; if its optimizer state, gradient cache, or parameter aggregation link spans multiple parallel groups or pipelines, its boundary connection is high. The stronger the memory residency, the lower the load, and the fewer the cross-boundary connections, the more stable the residency boundary; conversely, it indicates that although the subtask has residency relationships, it is close to a state requiring subsequent scheduling intervention.

[0040] Sub-step 1.5 generates the state map of the AGI training memory-resident computational network.

[0041] Specifically, the boundary-marked memory-resident network state set is merged according to the training subtask number to form a state graph that includes training subtask type, tensor parallel group affiliation, pipeline parallel segment affiliation, parameter slice group affiliation, model slice residence location, activation cache residence location, optimizer state residence location, gradient cache residence location, memory residence intensity, overall load pressure, and residence boundary stability intensity. For each training subtask, a continuous binding relationship is established between "training subtask, memory-resident object, computing node, network link, energy consumption status, and parallel boundary".

[0042] Step S120: The memory residency dependency of each training subtask is classified by memory residency network state map to delineate memory residency boundaries.

[0043] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes the following steps in step S120: Step S121: Read the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask in the memory-resident computation network state map, and determine whether the training subtask is attached to each dwell position, so as to obtain the memory-resident object attachment result.

[0044] Step S122: Based on the memory-resident object dependency results, combined with the memory-resident occupancy intensity and the number of cross-boundary associations, determine the residency dependency intensity of each training subtask, and label the training subtasks according to the residency dependency intensity.

[0045] Step S123: Based on the results of the level labeling of the training subtasks, determine the strongly resident subtasks and calculate the migration pause risk corresponding to the strongly resident subtasks, so as to delineate the memory residence boundary according to the migration pause risk.

[0046] In a specific embodiment, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention, step 2, generates memory residency boundary classification results. Based on the AGI training memory residency network state map obtained in step 1, the memory residency dependence of each training subtask is classified. Subtasks that only depend on the current training batch and do not involve long-term model shards are marked as weakly resided subtasks; subtasks that depend on model shards and activation caches but do not involve optimizer state reconstruction are marked as medium-residence subtasks; subtasks that simultaneously depend on model shards, activation caches, optimizer states, and gradient caches are marked as strongly resided subtasks. Subsequently, the node, parallel group, pipeline section, and parameter shard group where the strongly resided subtask is located are defined as memory residency boundaries, obtaining the memory residency boundary classification results, including the following sub-steps: Sub-step 2.1: Extract the dependencies of the memory-resident objects.

[0047] Specifically, using the AGI training memory-resident network state graph as input, the system sequentially reads the model slice residence location, activation cache residence location, optimizer state residence location, and gradient cache residence location corresponding to each training subtask. It then confirms whether the training subtask is only attached to the current training batch, whether it is attached to long-term model slices, whether it needs to maintain optimizer state continuity, and whether it needs to maintain gradient cache continuity. Subsequently, based on the occupancy ratio of the four types of memory-resident objects, different memory-resident objects are converted into a unified residence dependency strength to distinguish whether the training subtask has a long-term memory binding relationship.

[0048] Sub-step 2.2: Calculate the residency and attachment determination strength of the training subtask.

[0049] Specifically, based on the memory-resident object dependency structure, the memory-resident occupancy strength, resident boundary stability strength, and boundary entanglement number obtained in step 1 are used. When the training subtask has strong memory-resident object dependency, high memory-resident occupancy, relatively stable resident boundary, and many cross-boundary entanglement, it is not suitable to be directly transferred, and the resident dependency determination strength increases accordingly.

[0050] In this embodiment, the expression for the dwell attachment determination strength is: ; In the formula, Indicates the first The dwell dependency determination strength of each training subtask, dimensionless; This indicates the strength of memory resident dependency structure; Indicates the intensity of video memory resident usage, dimensionless; Indicates the stability strength of the stationary boundary, dimensionless; This indicates the number of connections that cross the boundaries of tensor parallel groups, pipeline parallel segments, and parameter piecewise groups, expressed in units of 1; This represents the amplification factor for video memory usage; it is dimensionless and ranges from 0.4 to 0.6. This represents the dwell stability amplification factor, which is dimensionless and ranges from 0.5 to 0.7. This represents the boundary entrainment amplification factor, which is dimensionless and ranges from 0.6 to 0.8. This represents the boundary entrainment smoothing parameter, expressed in units of 1-2.

[0051] Sub-step 2.3 performs weak, medium, and strong residency classification marking.

[0052] Specifically, each training subtask is categorized into levels based on its residency determination. If a training subtask only depends on the current training batch and does not involve the continuous maintenance of long-term model shards, optimizer states, and gradient caches, it is marked as a weakly resided subtask. If a training subtask depends on model shards and activation caches but does not involve optimizer state reconstruction, and the gradient cache can be inherited within the same parallel group, it is marked as a moderately resided subtask. If a training subtask depends on model shards, activation caches, optimizer states, and gradient caches simultaneously, and requires reloading, reconstruction, or synchronization after crossing tensor parallel groups, pipeline parallel segments, or parameter shard groups, it is marked as a strongly resided subtask. During execution, training subtasks with a dwell dependency determination strength below 0.35 are classified as weakly dwelling subtasks, those with a dwell dependency determination strength between 0.35 and 0.65 are classified as medium-dwelling subtasks, and those with a dwell dependency determination strength above 0.65 are classified as strongly dwelling subtasks. When a training subtask explicitly contains cross-node continuous dependencies of optimizer state and gradient cache, it is promoted to a strongly dwelling subtask even if its dwell dependency determination strength does not exceed 0.65.

[0053] Sub-step 2.4: Calculate the risk of paused migration for strongly resident subtasks.

[0054] Specifically, for objects marked as strongly resident subtasks, the potential pause risk when they are forcibly migrated is further calculated. The pause risk consists of model fragment reloading, activation cache reconstruction, optimizer state resynchronization, and gradient cache rebinding. The capacity of the GPU object that needs to be migrated or reconstructed is divided by the effective carrying bandwidth, and the reconstruction correction overhead of different objects is added to obtain the amount of time that can be directly used to determine whether to prohibit migration.

[0055] Sub-step 2.5: Define the memory residency boundary and generate the hierarchical results.

[0056] Specifically, based on the pause risk results of strongly resident subtasks, the GPU, compute node, tensor parallel group, pipeline parallel section, and parameter shard group where the strongly resident subtask resides are defined as memory resident boundaries. For strongly resident subtasks whose expected migration pause time is higher than the average overload duration of the current training iteration, their boundaries are marked as prohibited direct migration boundaries; for strongly resident subtasks whose expected migration pause time is lower than the average overload duration of the current training iteration but still involves optimizer state or gradient cache inheritance, their boundaries are marked as restricted migration boundaries; for boundaries of medium-resident subtasks, they are marked as migration-allowed boundaries within the same group; and for boundaries of weakly resident subtasks, they are marked as directly migration-allowed boundaries. The final result is a memory resident boundary classification that includes the training subtask number, resident level, memory resident object, migration pause risk, node, parallel group, pipeline section, parameter shard group, and boundary migration permission type.

[0057] Step S130: Based on the memory residency boundary, determine the migration feasibility of the training subtasks to be scheduled, so as to identify the strongly resident subtasks and apply alternative scheduling constraints to the strongly resident subtasks.

[0058] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes the following steps in step S130: Step S131: Determine the training subtasks to be scheduled based on the memory residency boundary, and establish candidate receiving nodes corresponding to the training subtasks to be scheduled, so as to calculate the migration adaptation strength corresponding to the weak residency subtasks in the candidate receiving nodes.

[0059] Step S132: Based on the migration adaptation strength corresponding to the weakly resident subtask, calculate the restricted migration inheritance strength of the resident subtask, and determine the alternative scheduling buffer strength of the strongly resident subtask according to the restricted migration inheritance strength.

[0060] Step S133: Determine whether the strong-resident subtask is prohibited from migrating by using alternative scheduling to mitigate the intensity, and apply alternative scheduling constraints to the strong-resident subtask when it is determined that the strong-resident subtask is prohibited from migrating.

[0061] In a specific embodiment, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention, step 3, generates migration feasibility and alternative scheduling constraint results. Based on the memory residency boundary classification results obtained in step 2, the migration feasibility of the training subtasks to be scheduled is judged. For weakly residency subtasks, direct migration is allowed between candidate nodes that meet the constraints of computing power load, link capacity, and energy consumption; for medium-residency subtasks, migration is only allowed between candidate nodes with the same model shard replicas, the same pipeline segment affiliation, or inheritable activation cache; for strongly residency subtasks, direct migration across memory residency boundaries is prohibited, and alternative scheduling is preferentially adopted by boundary-based load reduction, local reduction of training batches, acceptance by idle nodes in the same group, delay of gradient synchronization rhythm, or adjustment of parameter aggregation order. The migration feasibility and alternative scheduling constraint results are thus formed, including the following sub-steps: Sub-step 3.1: Extract the training subtasks to be scheduled and the set of candidate receiving nodes.

[0062] Specifically, based on the memory residency boundary classification results, the scheduled training subtasks under conditions of node overload, link congestion, abnormal energy consumption growth, or training iteration blockage are extracted, and their residency level, memory residency object, migration pause risk, node, tensor parallel group, pipeline parallel segment, parameter shard group, and boundary migration permission type are read. Subsequently, candidate receiving nodes are selected within the same training cluster. Candidate receiving nodes should at least have remaining GPU memory capacity, computing power load margin, link capacity margin, and energy consumption capacity margin. For weakly resided subtasks, candidate receiving nodes can be selected across nodes; for moderately resided subtasks, candidate receiving nodes must continue to retain model shard copies, pipeline segment ownership, or activation cache inheritance conditions; for strongly resided subtasks, candidate receiving nodes are only used as in-boundary receiving or preheating copy receiving objects, not as direct cross-boundary migration objects.

[0063] Sub-step 3.2: Calculate the direct migration adaptation strength of weakly residing subtasks.

[0064] Specifically, for objects marked as weakly resident subtasks, the direct migration adaptation strength is calculated among candidate accepting nodes. This direct migration adaptation strength simultaneously considers state transition time, graphics processor utilization, task queuing length, link bandwidth margin, and energy consumption margin. The expression is: ; In the formula, Indicates the first The weak-retention subtasks have been migrated to the first... The direct migration adaptation strength of each candidate receiving node, dimensionless; Indicates the first The weak-retention subtasks have been migrated to the first... The estimated state transition time for each candidate node is in milliseconds. Indicates the first The graphics processor utilization of each candidate node, dimensionless, with a value of 0-1; Indicates the first The bandwidth margin of the link corresponding to each candidate receiving node, in gigabytes per second; Indicates the first The energy consumption margin of each candidate receiving node, in watts; Indicates the first The task queuing length of each candidate node, in units of; This is the state transition time decay parameter, in milliseconds, with a value of 80-120. This is a bandwidth margin smoothing parameter, in gigabytes per second, with a value of 8-12. This is a smoothing parameter for energy consumption margin, in watts, with a value of 120-160. The queue length smoothing parameter is in units of 2-4. This formula indicates that weakly resident subtasks have a low dependence on long-term memory objects, so the feasibility of their direct migration is mainly determined by the idle computing power of the receiving node, the link's carrying capacity, energy consumption margin, and state transition time.

[0065] During execution, if the direct migration adaptation strength is higher than 0.55, a direct migration permit is allowed to be generated; if it is lower than 0.55, direct migration will not be executed for the time being, and subsequent restricted migration or alternative scheduling judgment will be entered. The above threshold is set around 0.55 to avoid candidate nodes being mistakenly selected when they are only better in a certain indicator.

[0066] Sub-step 3.3 calculates the restricted migration and inheritance strength of resident subtasks.

[0067] Specifically, based on the direct migration permission results for weakly resident subtasks, objects marked as medium-resident subtasks are further processed. For medium-resident subtasks, idle nodes are not used as migration targets. Instead, the migration target is determined by checking whether candidate receiving nodes have copies of the same model shards, belong to the same pipeline segment, can inherit the activation cache, and are within the synchronizable range of shard groups with the same parameters. The restricted migration inheritance strength is then calculated using the following expression: ; In the formula, Indicates the first The resident sub-tasks have been migrated to the first... The constrained migration and inheritance strength of each candidate successor node, dimensionless; Indicates the first Does the candidate accepting node have the first... The value of the identical model fragment copy required for the resident subtask is 0 or 1. Indicates the first Can the candidate node inherit the first...? The activation cache for the resident subtasks can be either 0 or 1. Indicates the first Whether each candidate receiving node is within the synchronization range of the same pipeline segment or the same parameter shard group, the value is 0 or 1; Indicates the first The resident sub-tasks have been migrated to the first... The number of boundaries that need to be crossed when selecting a candidate node, in units of ; Indicates the first The resident sub-tasks have been migrated to the first... The estimated migration pause time for each candidate node, in seconds; To activate the cache inheritance weight, a dimensionless weight with a value ranging from 0.7 to 0.9; Inherit weights for pipeline segments or parameter partitions; these weights are dimensionless and range from 0.8 to 1.0. The boundary crossing penalty coefficient is given per unit and ranges from 0.6 to 0.8. The pause time decay parameter, in seconds, ranges from 1.5 to 2.0. This formula indicates that while resident subtasks are allowed to migrate, it must ensure that model shard copies, activation caches, and pipeline ownership are inherited. The more boundaries crossed and the longer the pause time, the lower the restricted migration inheritance strength. During execution, if the restricted migration inheritance strength is higher than 0.60, a restricted migration permission is generated; if it is lower than 0.60, migration is prohibited, and the process proceeds to alternative scheduling constraint judgment.

[0068] Sub-step 3.4, based on the restricted migration permission results for medium-resident subtasks, continues processing objects marked as strongly resident subtasks. For strongly resident subtasks, firstly, direct migration across memory residency boundaries is prohibited; then, within their original memory residency boundaries, the de-loading capacity, the capacity that can be reduced in training batches, the capacity that can be taken over by idle nodes in the same group, the gradient synchronization delay time, and the parameter aggregation adjustment time are checked respectively, and the alternative scheduling buffer strength is calculated, expressed as: ; In the formula, Indicates the first The alternative scheduling and mitigation intensity of a strong stationary subtask is dimensionless. This indicates the deloadable computing power capacity within the boundary, measured in trillion floating-point operations per second. This indicates that the training batch size can be reduced, in units of the number of samples. This indicates the capacity that the idle nodes in the same group can handle, measured in trillions of floating-point operations per second. This indicates the possible delay time for gradient synchronization, in milliseconds. This indicates that the parameter aggregation time is adjustable, in milliseconds. The load reduction capacity smoothing parameter is in trillions of floating-point operations per second and has a value of 20-30. This is the batch reduction smoothing parameter, in units of sample size, with a value ranging from 16 to 32. This is a smoothing parameter for the capacity of the same group, in units of trillion floating-point operations per second, with a value of 20-30. This is the gradient synchronization delay smoothing parameter, in milliseconds, with a value of 50-80. The parameter aggregation adjusts the smoothing parameter, in milliseconds, with a value of 50-80. , , , , All are dimensionless weight parameters, ranging from 0.18 to 0.24, and have been normalized. This is because overload relief for strongly residing subtasks cannot rely on a single method; it must be mitigated through a combination of in-boundary load reduction, local batch reduction, group assignment, and synchronization rhythm adjustment. This formula is used to uniformly transform the alternative scheduling resources available when strongly residing subtasks cannot be migrated into dimensionless mitigation capabilities.

[0069] Next, the strength of the no-migration decision for the strongly resident subtask is calculated, expressed as: ; In the formula, Indicates the first The intensity of the prohibition on migration determination for each strong stationing sub-task is dimensionless. This indicates the estimated migration pause time, in seconds. This indicates the duration of overload in the current training iteration, in seconds. This represents the time smoothing parameter, in seconds, with a value ranging from 0.3 to 0.5. This represents the strength of the alternative scheduling mitigation, which is dimensionless. This formula indicates that when the expected migration pause time exceeds the overload duration, and alternative mitigation capabilities still exist within the boundary, direct migration should be prohibited and alternative scheduling should be adopted.

[0070] During execution, if the determination strength of the prohibition on migration is higher than 0.70, the strongly resident subtask will be solidified as an object that is prohibited from migrating across boundaries; if it is lower than 0.70 but still contains optimizer state and gradient cache inheritance relationship, it is still not allowed to migrate directly, and can only enter the preheating copy acceptance process.

[0071] Sub-step 3.5 generates migration feasibility and alternative scheduling constraints results.

[0072] Specifically, the direct migration permission results for weakly residing subtasks, the restricted migration permission results for medium-residing subtasks, and the alternative scheduling constraint results for strongly residing subtasks are uniformly merged to form a constraint list for subsequent dynamic scheduling. For weakly residing subtasks, their migration candidate nodes, direct migration adaptation strength, allowed migration time window, and target link carrying conditions are recorded. For medium-residing subtasks, their restricted migration candidate nodes, model shard replica inheritance conditions, pipeline segment ownership conditions, activation cache inheritance conditions, and expected migration pause time are recorded. For strongly residing subtasks, their prohibited cross-boundary migration markers, in-boundary load reduction capacity, training batch local reduction capacity, capacity taken over by idle nodes in the same group, gradient synchronization rhythm delay conditions, parameter aggregation order adjustment conditions, and preheating replica takeover conditions are recorded. The final result includes migration feasibility and alternative scheduling constraints for training subtasks, including training subtask number, residing level, migration permission type, candidate takeover nodes, prohibited migration boundaries, alternative scheduling methods, and scheduling trigger conditions.

[0073] Step S140: Based on the results of the alternative scheduling constraints on the strongly resident subtasks, the boundary preservation correction is performed on the training iteration process of the AGI large model, and a preheating copy is established for the specified model slices to generate the boundary preservation scheduling strategy.

[0074] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes the following steps in step S140: Step S141: Obtain the graphics processor utilization, link congestion duration, and node energy consumption growth in the current training iteration, and perform different migration processes on weakly resident subtasks and mediumly resident subtasks according to the graphics processor utilization, link congestion duration, and node energy consumption growth.

[0075] Step S142: Based on the results of different transfer processing for weakly resident subtasks and mediumly resident subtasks, the training batch carrying capacity corresponding to the strongly resident subtask is corrected, and prediction replicas are established for the specified model shards according to the corrected training batch carrying capacity.

[0076] Among them, the boundary preservation scheduling strategy is obtained by uniformly solidifying the results of different migration processing for weakly resident subtasks and mediumly resident subtasks, as well as the corrected training batch carrying capacity and preheating replicas.

[0077] In a specific embodiment, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by this invention, step 4, generates a boundary-preserving dynamic scheduling scheme. Based on the migration feasibility and alternative scheduling constraints obtained in step 3, boundary-preserving corrections are performed on node overload, link congestion, and abnormal energy consumption growth during the training iteration process. When the overloaded subtask belongs to a weakly resident subtask, direct migration is performed; when the overloaded subtask belongs to a medium-resident subtask, restricted migration is performed under the same resident inheritance conditions; when the overloaded subtask belongs to a strongly resident subtask, cross-node forced migration is not triggered, but the training batch carrying ratio, forward computation start rhythm, backward gradient computation recycling rhythm, and parameter aggregation waiting order are adjusted within the memory resident boundary. At the same time, a preheating copy is established for model slices that can be slowly migrated, so that subsequent training iterations have smooth acceptance conditions, generating a boundary-preserving dynamic scheduling scheme, including the following sub-steps: Sub-step 4.1: Determine the dynamic correction trigger strength for training iterations.

[0078] Specifically, based on the migration feasibility and alternative scheduling constraints, the residency level, migration permission type, candidate receiving node, prohibited migration boundary, alternative scheduling method, and scheduling trigger condition of each training subtask to be scheduled are read. Then, combined with the GPU utilization, link congestion duration, and node energy consumption increase in the current training iteration, the dynamic correction trigger strength for the training iteration is calculated, expressed as: ; In the formula, Indicates the first The dynamic correction trigger strength of each training subtask to be scheduled, dimensionless; Indicates the first The utilization rate of the graphics processor where each training subtask to be scheduled is located, dimensionless, and takes a value of 0-1; This indicates the duration of link congestion corresponding to the training subtask, in milliseconds; This indicates the increase in energy consumption of the node corresponding to the training subtask, in watts; This represents a utilization smoothing parameter, which is dimensionless and ranges from 0.03 to 0.05. This parameter represents the smoothing parameter for the duration of link congestion, in milliseconds, and ranges from 80 to 120. This represents a parameter for smoothing energy consumption growth, in watts, with a value ranging from 100 to 150. , , All are dimensionless weight parameters, ranging from 0.30 to 0.40, and are normalized. This formula is used to uniformly convert node overload, link congestion, and abnormal energy consumption growth into scheduling trigger intensity. The higher the trigger intensity, the more likely the training subtask needs to enter the boundary-preserving correction first. The narrow range of the above parameters is to prevent a single node load or a single link congestion factor from excessively dominating the scheduling order.

[0079] Sub-step 4.2 performs direct migration of weakly resident subtasks and restricted migration of medium-resident subtasks.

[0080] Specifically, based on the dynamic correction of the triggering results during training iterations, different migration processes are performed for weakly resident subtasks and moderately resident subtasks. Migration only achieves execution stability when the migration demand is strong, the migration permission is strong, and the migration pause time is relatively short compared to the overload duration. For weakly resident subtasks, according to the direct migration permission results formed in step 3, nodes with lower state transition time, lower GPU utilization, higher link bandwidth margin, and higher energy consumption margin are selected from the candidate accepting nodes for direct migration. For moderately resident subtasks, restricted migration is performed only between candidate accepting nodes that have the same model shard replicas, belong to the same pipeline segment, or have inheritable activation caches.

[0081] Sub-step 4.3: Perform load reduction and rhythm rearrangement within the boundaries of the strongly resident subtask.

[0082] Specifically, based on the results of weak-terminal migration, we continue to process strongly-terminal subtasks that were not allowed to migrate across memory residency boundaries. For strongly-terminal subtasks, we do not trigger forced migration across nodes. Instead, we adjust the training batch load ratio, forward computation start rhythm, backward gradient computation retrieval rhythm, and parameter aggregation waiting order within their memory residency boundaries. First, we determine the number of training batches based on the alternative scheduling buffering strength and the dynamic correction triggering strength. When the dynamic correction triggering strength is high and the buffering capacity within the boundary is insufficient, the number of training batches should decrease; when the buffering capacity within the boundary is sufficient, the decrease in the number of training batches is smaller, thereby avoiding excessive reduction in training throughput.

[0083] Subsequently, the forward computation start interval, backward gradient computation retrieval interval, and parameter aggregation waiting order of strongly resident subtasks are rearranged to ensure that model fragments, activation caches, optimizer states, and gradient caches within the same memory residency boundary maintain their original binding relationships. For idle nodes within the same tensor parallel group and the same pipeline segment, they are only allowed to take on temporary computation fragments or gradient retrieval fragments, and are not allowed to take over complete strongly resident subtasks.

[0084] Sub-step 4.4: Establish the conditions for accepting the preheating copy of the model fragment.

[0085] Specifically, based on the correction results within the strong residency boundary, a pre-warming copy is created for model slices that can be slowly migrated. The capacity of the residency objects requiring pre-warming is divided by the available transmission bandwidth, and the synchronization overhead of different objects is added to obtain the time result of whether the conditions for acceptance can be formed before subsequent iterations. This pre-warming copy does not immediately replace the original residency position, but gradually loads the model slices into the target acceptance node and synchronizes the necessary optimizer state and gradient cache state, so that subsequent training iterations have smooth acceptance conditions.

[0086] Sub-step 4.5, solidify the boundary-preserving dynamic scheduling scheme.

[0087] Specifically, the execution results of weak-to-medium-terminal migration, the correction results within the strong-terminal boundary, and the pre-warming replica acceptance results are unified and solidified to generate a boundary-preserving dynamic scheduling scheme. This scheme includes the direct migration target node, direct migration time window, and post-migration link carrying conditions for weak-terminal subtasks; the restricted migration target node, model shard replica inheritance conditions, pipeline ownership conditions, and activation cache inheritance conditions for medium-terminal subtasks; and the prohibited cross-boundary migration marker, training batch carrying capacity, forward computation start rhythm, backward gradient computation recycling rhythm, parameter aggregation waiting order, temporary acceptance relationship of idle nodes in the same group, and model shard pre-warming replica acceptance status for strong-terminal subtasks. Ultimately, node overload, link congestion, and abnormal energy consumption growth during training iterations are dynamically corrected without violating the memory residency boundary.

[0088] Step S150: Perform distributed training of the AGI large model through the boundary-preserving scheduling strategy, and maintain the original resident inheritance relationship during the training iteration to output the dynamic computing power scheduling optimization results.

[0089] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention includes the following steps in step S150: Step S151: Perform distributed training of the AGI large model according to the boundary preservation scheduling strategy, and determine whether the model shards, activation cache, optimizer state and gradient cache maintain the original resident inheritance relationship after each training iteration to obtain the resident inheritance confirmation result.

[0090] Step S152: Extract the preheated copy of the model fragment corresponding to the strongly resident subtask from the resident inheritance confirmation result, in order to determine whether the preheated copy has the smooth acceptance condition, and output the preheated copy acceptance determination result.

[0091] The preheating copy acceptance determination result includes determining whether the preheating copy has smooth acceptance conditions and determining whether the preheating copy does not have smooth acceptance conditions.

[0092] In some embodiments, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention further includes the following steps in step S150: Step S153: When the preheating copy acceptance determination result is that the preheating copy has smooth acceptance conditions, the strong-resident subtask is changed from prohibited migration to restricted migration, and the model fragment with the preheating copy is inherited in the corresponding acceptance node.

[0093] In step S154, if the result of the preheating copy acceptance determination is that the preheating copy does not have the smooth acceptance condition, then the boundary preservation correction is continued for the training iteration process of the AGI large model.

[0094] In a specific embodiment, the dynamic computing power scheduling optimization method for distributed training of AGI large models provided by the present invention, in step 5, outputs the dynamic computing power scheduling optimization result with adaptive memory residency boundary. Based on the boundary-preserving dynamic scheduling scheme obtained in step 4, the distributed training of AGI large models continues, and the residency inheritance relationship between model shards, activation cache, optimizer state, and gradient cache is continuously maintained in subsequent training iterations. When the preheating copy reaches the acceptable state, the original strongly resided subtask is converted into a restricted migration subtask, and a smooth migration is completed without rebuilding the critical cache or resynchronizing the complete optimizer state; when the preheating copy does not reach the acceptable state, the in-boundary deload and local re-sharding processing are maintained. The final output includes the dynamic computing power scheduling optimization result containing the memory residency boundary, subtask migration permission relationship, in-boundary deload relationship, model shard preheating copy relationship, and continuous execution state of training iteration, including the following sub-steps: Sub-step 5.1 confirms the continuity of resident inheritance during training iterations.

[0095] Specifically, based on the boundary-preserving dynamic scheduling scheme, distributed training of the AGI large model continues. After each training iteration, it is confirmed whether the model shards, activation cache, optimizer state, and gradient cache still maintain their original residency inheritance relationships. For weakly residency subtasks, it is confirmed whether the state transition is completed after migration; for moderately residency subtasks, it is confirmed whether the inheritance conditions of model shard copies, pipeline ownership, and activation cache still hold; for strongly residency subtasks, it is confirmed whether they are still in the prohibited cross-boundary migration state, and whether the load reduction within the boundary and local resharding continue to take effect.

[0096] Sub-step 5.2: Determine whether the preheated copy has reached a ready state.

[0097] Specifically, in the continuity confirmation results of residency inheritance, the model sharding preheating replica relationship, estimated preheating completion time, remaining stable training window, available link bandwidth margin, and target node energy consumption margin corresponding to the strongly residency subtask are extracted. Only when residency inheritance is complete, the preheating completion time is less than the stable acceptance window, and both link and energy consumption have acceptance margins, can the preheated replica meet the conditions for smooth acceptance.

[0098] Sub-step 5.3: Perform state transition and smooth migration of the strongly resident subtask.

[0099] Specifically, when the preheating copy acceptance determination result shows that the preheating copy has reached the acceptable state, the original strongly resident subtask is converted from a prohibited cross-boundary migration object to a restricted migration subtask, and the preheated model fragment, optimizer state and gradient cache state are inherited in the target accepting node. Only a small number of incremental states generated by the current training iteration are supplemented, without reloading the complete model fragment, rebuilding the complete activation cache, or resynchronizing the complete optimizer state.

[0100] Sub-step 5.4 maintains the load reduction process within the boundary of unaccepted strongly persistent sub-tasks.

[0101] Specifically, for strongly persistent subtasks that have not yet reached a ready-to-accept state, the following measures are maintained as determined in step 4: load reduction within the boundary, local reduction of training batches, temporary acceptance by idle nodes in the same group, delayed gradient synchronization rhythm, and adjustment of parameter aggregation order. At this time, the model sharding location, activation cache location, optimizer state location, and gradient cache location remain unchanged. Only the local training batch carrying ratio is adjusted according to the current continuous execution state of the training iteration to ensure that the training process does not undergo forced migration across boundaries due to immature preheating copies.

[0102] Sub-step 5.5 outputs the dynamic computing power scheduling optimization results.

[0103] Specifically, the state transition results of strongly resident subtasks, the results of maintaining tasks within the boundary of unaccepted tasks, the direct migration results of weakly resident subtasks, and the restricted migration results of medium-resident subtasks are merged to form the final scheduling optimization result. This result includes memory residency boundaries, subtask migration permission relationships, prohibited cross-boundary migration relationships, in-boundary load reduction relationships, local resharding relationships of training batches, model shard preheating copy relationships, strongly resident subtask state transition relationships, and continuous execution status of training iterations. Ultimately, this enables AGI large-scale model distributed training to adaptively select direct migration, restricted migration, in-boundary load reduction, or preheating copy smooth acceptance based on memory residency boundaries when node overload, link congestion, or abnormal energy consumption occurs. This avoids model shard reloading, activation cache reconstruction, optimizer state resynchronization, and long-term training pauses caused by forced migration.

[0104] The following describes the dynamic computing power scheduling optimization device for distributed training of AGI large models provided by the present invention. The dynamic computing power scheduling optimization device for distributed training of AGI large models described below can be referred to in correspondence with the dynamic computing power scheduling optimization method for distributed training of AGI large models described above.

[0105] like Figure 2 As shown in one embodiment, a dynamic computing power scheduling optimization device for distributed training of AGI large models includes a state graph construction module, a dwell boundary partitioning module, a scheduling constraint module, a scheduling strategy generation module, and an optimization result output module.

[0106] The state graph construction module is used to obtain the computing power state parameters of each training subtask during the distributed training of the AGI large model, and bind the computing power state parameters with the corresponding computing power nodes to construct the state graph of the memory-resident computing network.

[0107] The memory residency boundary delineation module is used to classify the memory residency dependency of each training subtask by using the memory residency network state graph, so as to delineate the memory residency boundary.

[0108] The scheduling constraint module is used to determine the migration feasibility of training subtasks to be scheduled based on the memory resident boundary, in order to identify strongly resident subtasks and apply alternative scheduling constraints to the strongly resident subtasks.

[0109] The scheduling strategy generation module is used to perform boundary preservation correction on the training iteration process of the AGI large model based on the result of replacing the scheduling constraints on the strongly resident subtasks, and to create a preheating copy for the specified model slices in order to generate a boundary preservation scheduling strategy.

[0110] The optimization result output module is used to perform distributed training of AGI large models through the boundary-preserving scheduling strategy, and maintain the original resident inheritance relationship during the training iteration to output the dynamic computing power scheduling optimization results.

[0111] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0112] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0113] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0114] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0115] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0116] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0117] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A dynamic computing power scheduling optimization method for distributed training of large AGI models, characterized in that, The method includes: Obtain the computing power status parameters of each training subtask during distributed training of the AGI large model, and bind the computing power status parameters to the corresponding computing power nodes to construct a state map of the memory-resident computing network. The memory residency dependency of each training subtask is classified and processed by the memory residency network state map to delineate the memory residency boundary. Based on the memory residency boundary, the migration feasibility of the training subtasks to be scheduled is determined to identify the strongly residency subtasks, and alternative scheduling constraints are applied to the strongly residency subtasks. Based on the results of the alternative scheduling constraints on the strongly resident subtasks, the training iteration process of the AGI large model is corrected by boundary preservation, and a warm-up copy is established for the specified model slices to generate a boundary preservation scheduling strategy. The boundary-preserving scheduling strategy is used to perform distributed training of AGI large models and maintain the original resident inheritance relationship during the training iteration process to output dynamic computing power scheduling optimization results.

2. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 1, characterized in that, The step of obtaining the computing power state parameters of each training subtask during distributed training of the AGI large model and binding the computing power state parameters with the corresponding computing power nodes to construct a memory-resident computing network state graph includes: Obtain the execution order of training iterations in the distributed training of the AGI large model, extract multiple training subtasks according to the execution order of the training iterations, and record the tensor parallel group, pipeline parallel segment and parameter sharding group of each training subtask. Collect the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask, and determine the memory dwell intensity based on the memory occupied by each dwell position. The computing power status parameters include the tensor parallel group, pipeline parallel segment, parameter sharding group, model sharding residence position, activation cache residence position, optimizer state residence position, gradient cache residence position, and the memory occupied by each residence position.

3. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 2, characterized in that, The step of obtaining the computing power state parameters of each training subtask during distributed training of the AGI large model and binding the computing power state parameters with the corresponding computing power nodes to construct a memory-resident computing network state graph also includes: Calculate the node load pressure of the computing nodes corresponding to the training subtasks, and construct the memory-resident computing network load state set by combining the memory occupancy intensity. Based on the memory-resident computing network bearer state set, the tensor parallel group boundary, pipeline boundary and parameter partition boundary corresponding to each training subtask are recorded, and the number of cross-boundary associations corresponding to the tensor parallel group boundary, pipeline boundary and parameter partition boundary is counted. The memory-resident computing network state set is merged according to the sequential numbering of the training subtasks, and combined with the cross-boundary association quantity to generate the memory-resident computing network state map.

4. The dynamic computing power scheduling optimization method for distributed training of large AGI models according to claim 3, characterized in that, The step of classifying the memory residency dependency of each training subtask using the memory residency network state map to delineate memory residency boundaries includes: Read the model slice dwell position, activation cache dwell position, optimizer state dwell position and gradient cache dwell position corresponding to each training subtask in the memory-resident network state map, and determine whether the training subtask is attached to each dwell position to obtain the memory-resident object attachment result. Based on the memory-resident object dependency results, combined with the memory-resident occupancy intensity and the number of cross-boundary associations, the residency dependency intensity of each training subtask is determined, and the training subtask is graded according to the residency dependency intensity. Based on the results of the level labeling of the training subtasks, the strongly resident subtasks are determined, and the migration pause risk corresponding to the strongly resident subtasks is calculated, so as to delineate the memory residence boundary according to the migration pause risk.

5. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 1, characterized in that, The step of determining the migration feasibility of the training subtasks to be scheduled based on the memory residency boundary to identify strongly resident subtasks and applying alternative scheduling constraints to the strongly resident subtasks includes: The training subtasks to be scheduled are determined based on the memory residency boundary, and candidate receiving nodes corresponding to the training subtasks to be scheduled are established, so as to calculate the migration adaptation strength corresponding to the weak residency subtasks in the candidate receiving nodes. Based on the migration adaptation strength corresponding to the weakly resident subtask, the restricted migration inheritance strength of the resident subtask is calculated, and the alternative scheduling relief strength of the strongly resident subtask is determined according to the restricted migration inheritance strength. The alternative scheduling relief strength is used to determine whether the strongly resident subtask is prohibited from migration, and when the strongly resident subtask is determined to be prohibited from migration, an alternative scheduling constraint is applied to the strongly resident subtask.

6. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 5, characterized in that, Based on the results of the alternative scheduling constraints applied to the strongly resident subtasks, the training iteration process of the AGI large model is modified to maintain boundaries, and preheating copies are established for specified model slices to generate a boundary-maintaining scheduling strategy, including: Obtain the graphics processor utilization, link congestion duration, and node energy consumption increase in the current training iteration, and perform different migration processes on the weak-resident subtask and medium-resident subtask respectively based on the graphics processor utilization, link congestion duration, and node energy consumption increase. Based on the results of different transfer processing on the weakly resident subtasks and the mediumly resident subtasks, the training batch carrying capacity corresponding to the strongly resident subtasks is corrected, and prediction replicas are established for the specified model shards according to the corrected training batch carrying capacity. The boundary-preserving scheduling strategy is obtained by uniformly solidifying the results of different migration processes performed on the weakly resident subtasks and the mediumly resident subtasks, as well as the corrected training batch carrying capacity and the preheating replicas.

7. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 6, characterized in that, The process of performing distributed training of the AGI large model through the boundary-preserving scheduling strategy, and maintaining the original resident inheritance relationship during training iterations to output dynamic computing power scheduling optimization results, includes: AGI large model distributed training is performed according to the boundary preservation scheduling strategy, and after each training iteration, it is determined whether the model shards, activation cache, optimizer state and gradient cache maintain the original resident inheritance relationship to obtain the resident inheritance confirmation result. Extract the preheated copy of the model fragment corresponding to the strongly resident subtask from the resident inheritance confirmation result, determine whether the preheated copy has the smooth acceptance condition, and output the preheated copy acceptance determination result. The preheated copy acceptance determination result includes determining whether the preheated copy has smooth acceptance conditions and determining whether the preheated copy does not have smooth acceptance conditions.

8. The dynamic computing power scheduling optimization method for distributed training of AGI large models according to claim 7, characterized in that, The step of performing distributed training of the AGI large model through the boundary-preserving scheduling strategy, and maintaining the original resident inheritance relationship during the training iteration to output dynamic computing power scheduling optimization results, further includes: When the result of the preheating copy acceptance determination is that the preheating copy has smooth acceptance conditions, the strong-resident subtask is changed from prohibited migration to restricted migration, and the model fragment with the preheating copy is inherited in the corresponding acceptance node. If the result of the preheating copy acceptance determination is that the preheating copy does not have the smooth acceptance condition, then the boundary preservation correction will continue to be performed on the training iteration process of the AGI large model.

9. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-8.

Images