An AI agent training method based on machine learning

Abstract

The application relates to the technical field of artificial intelligence and machine learning training scheduling, in particular to an AI intelligent agent training method based on machine learning, which comprises the following steps: information initialization: receiving tasks, samples and node information, extracting features, and dividing a playback cache area according to sample versions; state evaluation: comprehensively evaluating multidimensional states according to the foregoing information to determine task types; combining time sequence smooth analysis to determine node state results; control and scheduling: deciding whether to perform preheating control according to switching delay risks, and scheduling resources based on task and node states; monitoring and intervention: monitoring sample expiration risks, performance trends and network congestion, and respectively performing expiration blocking, node degradation or state recovery when thresholds are triggered; feedback and updating: feeding back training, node and sample statistical results to update scheduling parameters; and the application avoids isolated processing of tasks, samples and nodes, and improves training stability and resource utilization effect.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and machine learning training scheduling technology, specifically to a machine learning-based AI agent training method. Background Technology

[0002] Existing intelligent agent training systems typically process training tasks, sample data, and node states independently, relying on unified queues, fixed priorities, or single hardware metrics for resource scheduling. This static and isolated approach has the following drawbacks:

[0003] The node status assessment has a single dimension; relying solely on instantaneous utilization or online status cannot accurately identify intermediate states such as mild node jitter, continuous degradation, or signs of impending blockage, which can easily lead to the mixing of stable and degraded nodes; sample consumption lacks a version alignment mechanism.

[0004] The first-in-first-out (FIFO) model cannot identify the degree of deviation between the generated sample version and the target training version, resulting in expired samples being mixed with current version samples in the same batch, which seriously affects training consistency; delayed resource preparation leads to high latency; when training switches from sample writing to model updating or gradient synchronization, resources such as communication cache and execution flow are mostly allocated in a post-event response manner, resulting in a significant increase in the initial synchronization waiting time and the initial batch startup latency; the scheduling strategy is fixed and lacks adaptive feedback; the use of a fixed unified queue and batch strategy makes it impossible to perform differentiated scheduling based on task type, node status, and sample relationships; and the system cannot use real-time training feedback to dynamically optimize scheduling parameters, causing the long-term control effect to gradually deviate from the actual operation requirements. Summary of the Invention

[0005] The purpose of this invention is to provide an AI agent training method based on machine learning, which avoids the distortion of scheduling criteria and the mismatch of sample usage caused by isolating training tasks, sample data and training nodes. It can also perform differentiated resource scheduling and preheating control based on collaborative evaluation of multi-dimensional states, so as to maintain consistency between training phase switching, sample version usage and node role.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] Receive training tasks, sample streams, and node status information, and determine task description information, sample version information, node underlying status information, network congestion duration, synchronization wait queue depth, sample write rate, and stage switching signal, respectively.

[0008] The replay cache is divided into current, nearby, and expired version sample areas based on sample version information;

[0009] Based on task description information, sample version information, and node underlying status information, determine stage information, latency information, node health information, version matching information, and load change information, and determine the task type;

[0010] The abnormal state is determined by time-series smoothing of the underlying state information of the node, and the node state result is determined by combining the number of node degradations and the duration of the abnormality.

[0011] The switching delay risk value is determined based on the synchronous waiting queue depth, sample write rate and stage switching signal; when the switching delay risk value is greater than the preset switching delay risk threshold, a warm-up control command is executed based on the task type and node status result.

[0012] Resource scheduling is performed based on task type, node status results, version matching information, and load change information.

[0013] Monitor sample expiration risk, performance change trends determined by node underlying state information, and network congestion duration; when sample expiration risk, node health information, or network congestion duration trigger their respective preset thresholds, generate flags to block expired samples from entering the batch, set the node to only carry asynchronous tasks, or switch to recovery state.

[0014] Feedback on training execution results, node status changes, sample consumption, and synchronization statistics is used to update scheduling parameters.

[0015] Preferably, the task description information includes task category identifier, process stage, and synchronization wait identifier; the sample version information includes sample generation strategy version number and target training version number; the node underlying status information includes graphics processor frequency deviation, video memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample reading latency.

[0016] Based on task description information, sample version information, and node underlying status information, stage information, latency information, node health information, version matching information, and load change information are determined, including: stage information is determined based on task category identifier, process stage, and synchronization wait identifier;

[0017] The latency information is determined based on the remaining execution time window corresponding to the training task, the depth of the synchronization waiting queue, and the preset timeout configuration.

[0018] Node health information is determined based on graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency.

[0019] Version matching information is determined based on the version difference between the sample generation strategy version number and the target training version number;

[0020] Load change information is determined based on the number of newly written samples per unit time, the growth of the playback queue, the number of nodes waiting for parameter synchronization, and the idle rate of the graphics processor and the backlog rate of the central processing unit.

[0021] Preferably, when the training task is in the model update stage or gradient synchronization stage, the latency risk level determined based on latency information is a high sensitivity level, the version matching information represents the current version sample and the node health information represents a stable node, the task type corresponding to the training task is determined to be a synchronization task.

[0022] When the training task is in the sample preprocessing stage or gradient merging stage, and the latency risk level determined based on latency information is high-sensitivity or medium-sensitivity, the task type corresponding to the training task will be determined as a latency-priority task.

[0023] If the training task belongs to replay pool sorting, historical sample recoding or ordinary training batch consumption, and the corresponding training node is judged to be in a jitter state or performance degradation state and is allowed to carry regular computing tasks, the task type corresponding to the training task will be determined as a regular computing task.

[0024] If the training task is an offline evaluation, log processing, historical sample compression, or background analysis task, the task type corresponding to the training task will be determined as a background processing task.

[0025] Preferably, abnormal states are determined by temporal smoothing of the underlying state information of nodes, and the node state result is determined by combining the number of node degradations and the duration of the abnormality, including:

[0026] The graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length and sample read latency in the node's underlying state information are smoothed to obtain the smoothed state information.

[0027] The abnormal state of a node is determined based on the smoothed state information, and the degradation count value of the corresponding training node is updated based on the abnormal state of the node.

[0028] Based on the node's abnormal state, the number of degradations, and the duration of the abnormality, the training node is divided into any of the following states: jitter state, performance degradation state, blocked state, or offline state.

[0029] The corresponding node status result is determined based on the status division result of the training node. The node status result is any of the following: allowed to carry synchronous training tasks, only carry asynchronous training tasks, pause receiving new version training tasks, or enter the node recovery state.

[0030] Preferably, the preheating control command is used to instruct the execution of at least one of the following preparation operations:

[0031] Communication preparation: pre-allocate the send buffer, perform memory page preprocessing on the parameter synchronization buffer, and maintain the heartbeat connection;

[0032] Computational preparation: Pre-start the data prefetching thread, preload the next batch of sample indexes, and submit placeholder tasks to the high-priority graphics processor execution stream;

[0033] Cache preloading: Expand the writable cache space, pre-allocate sample block information, and open a version-independent storage area.

[0034] Preferably, resource scheduling includes: performing at least one of the following on the synchronization task: allocating a dedicated execution queue, binding key threads to cores, high-priority execution flow, dedicated synchronization channel, and batching strategy corresponding to a preset batch range, and limiting it to only allowing the current version sample area to be read;

[0035] For latency-priority tasks, perform at least one of the following: allocate priority prefetching threads, bind preprocessing threads to cores, pre-set batching strategies corresponding to batch intervals, and gradient merging queues, and prioritize the allocation of stable network links;

[0036] Assign a first preset batch strategy, a first preset scheduling period, a shared synchronization channel, and a range for reading adjacent version sample areas to regular computing tasks, and allow regular computing tasks to be carried by jitter state nodes or performance degradation state nodes;

[0037] Idle time slices are allocated to background processing tasks, and suspension is performed when high-priority tasks preempt the task, while limiting input and output bandwidth usage.

[0038] Preferably, monitoring the risk of sample expiration includes: determining the risk of sample expiration based on the number of expired samples to be included in the current training batch, the total number of samples in the current training batch, the average version difference of samples in the current training batch, and the average lag steps between the sample generation version and the target training version;

[0039] If the risk of sample expiration is higher than the preset expiration risk threshold, perform at least one of the following: block expired version samples from entering the training batch, reduce the sampling ratio of adjacent version samples, suspend the writing of new exploration samples to the old version partition, and narrow the allowed version range.

[0040] Preferably, the training execution results, node state changes, sample consumption status, and synchronization statistics are fed back to update the scheduling parameters. Specifically, this includes: obtaining sample task information, sample version information, sample node status information, target training version information, training execution results, node state changes, sample consumption status, and synchronization statistics.

[0041] Based on sample task information, sample version information, and sample node status information, determine sample stage information, sample latency information, node health information, sample version difference information, and sample load change information;

[0042] Input the sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the initial scheduling parameter model to obtain sample task type information, node status result information, and sample preheating indication information. The initial scheduling parameter model includes sub-models for task type judgment, status judgment, and preheating judgment.

[0043] Resource preparation score, node stability score and task adaptation score are determined based on sample task type information, node status result information and sample preheating indication information, respectively. Version matching score is determined based on sample version difference information and sample consumption information during training.

[0044] The optimization target is determined based on resource preparation score, node stability score, task adaptation score and version matching score, and the model parameters of the initial scheduling parameter model are iteratively optimized accordingly to obtain the target scheduling parameter model.

[0045] Preferably, sample stage information, sample latency information, node health information, sample version difference information, and sample load change information are input into the sub-model used for task type determination to obtain sample task type information;

[0046] Node health information, node abnormal state information, and corresponding training node degradation count information are input into the sub-model used for state judgment to obtain node state result information; sample task type information, node state result information, synchronous waiting queue depth, sample write rate, and stage switching signal are input into the sub-model used for warm-up judgment to obtain sample warm-up indication information.

[0047] Preferably, the version difference information is determined based on the version difference between the sample generation strategy version number and the target training version number;

[0048] The risk of sample expiration is determined based on version difference information and sample consumption information during the training process.

[0049] A version matching score, which characterizes the consistency of sample versions, is obtained by weighting the version difference information and the sample expiration risk information.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention extracts stage information, latency information, node health information, version matching information, and load change information, and determines the task type corresponding to the training task accordingly. It can transform the training stage, synchronization pressure, node degradation trend, sample version deviation degree, and system load change into quantitative basis that can be directly involved in scheduling decisions, thereby avoiding the distortion of scheduling basis caused by relying solely on unified throughput indicators, fixed priority strategies, or single hardware signals.

[0052] 2. This invention performs time-series smoothing, anomaly identification, degradation counting, and state classification on the underlying state information of nodes to obtain node state results that allow them to carry synchronous training tasks, carry only asynchronous training tasks, pause receiving new version training tasks, or enter the node recovery state. It can effectively suppress erroneous actions triggered by single-cycle noise, and identify intermediate states such as mild jitter, continuous degradation, and blocking trends, thereby avoiding indiscriminately taking mildly degraded nodes offline or allowing degraded nodes to continue participating in critical synchronous tasks.

[0053] 3. This invention divides the replay cache into the current version sample area, the neighboring version sample area, and the expired version sample area according to the correspondence between the sample generation strategy version number and the target training version number. It also performs control actions such as blocking expired version samples from entering the training batch, reducing the sampling ratio of neighboring version samples, pausing the writing of new exploration samples into the old version partition, and narrowing the allowed version range in combination with the risk of sample expiration. This can effectively reduce the training offset risk caused by mixing old strategy samples with the current training target, thereby improving the consistency of sample usage and the stability of training convergence.

[0054] 4. By triggering warm-up control when the stage switching delay risk value exceeds the risk threshold, and performing warm-up actions such as pre-allocating communication cache, pre-starting data prefetching thread, pre-loading the next batch of sample index, submitting placeholder tasks to high-priority execution streams, and expanding cache writable space, the communication, computation, and cache path preparation can be completed in advance before the training process enters the synchronization load stage or high write stage, thereby reducing the initial synchronization waiting time and the initial batch startup latency. Attached Figure Description

[0055] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 The diagram shows a flowchart of an AI agent training method based on machine learning according to an embodiment of the present disclosure. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This implementation is applied to a training system consisting of a training controller, a training node cluster, a replay cache module, a parameter synchronization module, and a node proxy module. The training controller receives the training tasks to be executed through in-process object references, reads the sample header information of the sample stream through the shared memory mapping area, and receives the training node status information through the long connection reporting channel maintained by the node proxy module.

[0059] The node agent module encapsulates the collected graphics processor frequency deviation, video memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency into a structured status record according to a preset field order, and writes it into the memory queue corresponding to the scheduling process.

[0060] The training controller retrieves records sequentially from the memory queue and releases the corresponding cache slots after reading to ensure that the lifecycle of the state record only covers the current scheduling cycle;

[0061] In this embodiment, the training task, sample stream and training node status information are first uniformly organized to generate task description information, sample version information and node underlying status information. Then, based on the three types of information, such as task, sample and node, stage information, latency information, node health information, version matching information and load change information are extracted, and the task type corresponding to the training task is determined accordingly.

[0062] The underlying state information of the nodes is smoothed in time and classified in state to obtain the node state results of the training nodes; when the stage switching delay risk value exceeds the risk threshold, preheating control is triggered, and resource scheduling is performed in combination with task type, node state results, version matching information and load change information.

[0063] During training execution, the risk of sample expiration, the trend of node performance changes and the duration of network congestion are continuously monitored. When the corresponding conditions are met, blocking, degradation or recovery flags are generated, and the training execution results, node state changes, sample consumption and synchronization statistics are written back to the next round of scheduling parameter update process.

[0064] The above processing avoids treating training operations as a single task flow and avoids indiscriminately migrating tasks based solely on instantaneous hardware signals, thus ensuring consistency between training phase switching, sample version usage, and node carrying roles.

[0065] Please see Figure 1 A machine learning-based AI agent training method includes:

[0066] Receive training tasks, sample streams, and node status information, and determine task description information, sample version information, node underlying status information, network congestion duration, synchronization wait queue depth, sample write rate, and stage switching signal, respectively.

[0067] The replay cache is divided into current, nearby, and expired version sample areas based on sample version information;

[0068] Based on task description information, sample version information, and node underlying status information, determine stage information, latency information, node health information, version matching information, and load change information, and determine the task type;

[0069] The abnormal state is determined by time-series smoothing of the underlying state information of the node, and the node state result is determined by combining the number of node degradations and the duration of the abnormality.

[0070] The switching delay risk value is determined based on the synchronous waiting queue depth, sample write rate and stage switching signal; when the switching delay risk value is greater than the preset switching delay risk threshold, a warm-up control command is executed based on the task type and node status result.

[0071] Resource scheduling is performed based on task type, node status results, version matching information, and load change information.

[0072] Monitor sample expiration risk, performance change trends determined by node underlying state information, and network congestion duration; when sample expiration risk, node health information, or network congestion duration trigger their respective preset thresholds, generate flags to block expired samples from entering the batch, set the node to only carry asynchronous tasks, or switch to recovery state.

[0073] Feedback on training execution results, node status changes, sample consumption, and synchronization statistics is used to update scheduling parameters.

[0074] Obtain the training task to be executed, parse the training task to be executed according to the task field in the task scheduling entry, and generate task description information;

[0075] Among them, task description information refers to a set of fields that can characterize the business semantics of training tasks. For example, task category identifiers can be used to distinguish between model updates, sample preprocessing, replay pool organization and offline evaluation. Process stages can be used to characterize whether the current task is in the sample writing stage, batch consumption stage, gradient synchronization stage or parameter aggregation stage. Synchronization waiting identifiers can be used to characterize whether the current task is attached to a synchronization barrier.

[0076] The training controller preferably reads task objects directly from the task team list inside the training framework, without going through additional network forwarding, in order to reduce the latency caused by field copying;

[0077] Obtain the sample stream, and perform version sorting on the sample stream based on the version field in the sample header to determine the sample version information;

[0078] Among them, sample version information refers to the version relationship data composed of the sample generation strategy version number and the target training version number. For example, the version difference between the current version sample and the target training version is zero, the version difference between the neighboring version sample falls into the version tolerance window, and the version difference between the expired version sample exceeds the version tolerance window.

[0079] The sample header is preferably stored in a circular buffer maintained by the playback cache module. The training controller reads the header information field through shared memory. After reading, the sample body data is kept unchanged in the original buffer, and only a version index is established on the control plane.

[0080] Obtain training node status information, organize the training node status information according to the running status fields reported by the node agent, and determine the underlying status information of the node.

[0081] Among them, the node underlying state information refers to the set of underlying indicators that reflect the stability of the training node's operation. For example, the graphics processor frequency deviation reflects the computing power shift caused by thermal throttling or power supply fluctuations, the memory fragmentation rate reflects the ability of continuous memory allocation during batch switching, the bus retransmission count reflects the tendency of abnormal bus transmission within the node, the link latency reflects the network stability during parameter synchronization or remote data access, the central processing unit run queue length reflects the task queuing backlog, and the sample reading latency reflects whether the sample prefetching thread is blocked.

[0082] This implementation method coordinates task semantics, sample version and node running status within the same scheduling cycle, so that subsequent judgments are no longer limited to a single signal.

[0083] This implementation further preserves the sample version relationship, enabling the training controller to identify the degree of deviation between the old strategy samples and the current training target; this implementation introduces underlying metrics such as memory fragmentation rate, link latency, and sample read latency, which can capture online but degraded nodes.

[0084] Task description information includes task category identifier, process stage, and synchronization wait identifier; sample version information includes sample generation strategy version number and target training version number; node underlying status information includes graphics processor frequency deviation, video memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency.

[0085] Based on task description information, sample version information, and node underlying status information, stage information, latency information, node health information, version matching information, and load change information are determined, including: stage information is determined based on task category identifier, process stage, and synchronization wait identifier;

[0086] The latency information is determined based on the remaining execution time window corresponding to the training task, the depth of the synchronization waiting queue, and the preset timeout configuration.

[0087] Node health information is determined based on graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency.

[0088] Version matching information is determined based on the version difference between the sample generation strategy version number and the target training version number;

[0089] Load change information is determined based on the number of newly written samples per unit time, the growth of the playback queue, the number of nodes waiting for parameter synchronization, and the idle rate of the graphics processor and the backlog rate of the central processing unit.

[0090] Furthermore, task description information is obtained, and the task description information is tagged and mapped according to task category identifier, process stage, and synchronization wait identifier to determine stage information;

[0091] Among them, the stage information refers to the results used to characterize the business stage in which the training task is located, such as the environment interaction sampling stage, the playback cache writing stage, the batch sampling stage, the model update stage, the gradient synchronization stage, or the verification and evaluation stage; this stage information can be directly mapped through an explicit rule table to ensure that the scheduling judgment process is interpretable.

[0092] Obtain the remaining execution time window, synchronization wait queue depth, and timeout configuration parameters corresponding to the training task. Calculate the latency of the training task based on the latency urgency formula to determine the latency information.

[0093] Among them, latency information refers to the sensitivity of the training task to processing timeliness within the current scheduling cycle. For example, a high sensitivity level corresponds to tasks that are about to amplify the overall step latency due to barriers or waiting; a medium sensitivity level corresponds to auxiliary merging or preprocessing tasks; and a low sensitivity level corresponds to background tasks that are allowed to be delayed. To quantify this level, this implementation method first constructs a latency urgency index:

[0094]

[0095] Among them, the latency urgency index used to comprehensively measure how close the i-th training task is to the timeout boundary within the current system cycle is... The value is obtained by inverse linear mapping from the remaining execution time, which reflects the time pressure of the task. The normalized value of the synchronization waiting queue depth, reflecting the queuing pressure of similar tasks before the synchronization point. And the normalized value of the timeout configuration parameter, which reflects the strength of the training framework's allowable latency constraints for the task. It was calculated together;

[0096] To ensure strict consistency of the dimensions in the formula, this remaining execution time window Before being substituted into the calculation, the ratio is converted into a dimensionless ratio by dividing by the system's preset maximum tolerable timeout reference time;

[0097] During the calculation, a weighting coefficient is used where the sum of the three is 1. , , Dynamically balancing the combined impact of remaining time, synchronous queuing, and timeout constraints on latency risk; when A value above the first threshold can be classified as high sensitivity. A value less than or equal to the first threshold and greater than the second threshold can be classified as medium sensitivity. A value less than or equal to the second threshold can be classified as low sensitivity.

[0098] The system acquires graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency. Based on a health degradation calculation formula, it comprehensively evaluates the training nodes to determine node health information. Node health information refers to the stability of the training node, such as a stable node, a slightly degraded node, or a severely degraded node. To avoid insufficient sensitivity due to relying solely on a single utilization metric, this implementation constructs a health degradation metric:

[0099]

[0100] Among them, the health degradation index of the j-th training node represents the overall degree of the node's shift from a stable state to a degenerate state. Based on the normalized value of graphics processor frequency deviation, which reflects the loss of computing power caused by thermal throttling, power limitation, or load fluctuations in graphics processors. Memory fragmentation rate reflects the difficulty of allocating contiguous memory when applying for large quantities or new batches of memory. The normalized value of the bus retransmission count reflects the throughput loss caused by repeated retries in the internal transmission links of a node. The normalized link delay value reflects the transmission stability during parameter synchronization or remote data access. Normalized value of CPU run queue length reflecting thread scheduling waiting pressure. And the normalized value of sample read latency, reflecting the degree of blocking of the prefetch thread or read path. It was calculated together;

[0101] The calculation uses weighting coefficients whose sum is 1. to The combined effects of computing power, video memory, bus, network, central processing unit, and read link on node health are dynamically balanced.

[0102] The graphics processor core temperature is not output separately, but is used for correction. That is, when the temperature exceeds the thermal degradation initiation threshold, the frequency deviation is multiplied by a preset temperature correction factor to correct the frequency deviation, thereby enhancing the ability to identify thermal degradation.

[0103] Obtain the sample generation strategy version number and the target training version number. Based on the version difference, perform difference calculation on the sample version information to determine the version matching information.

[0104] Among them, version matching information refers to the degree of consistency between a sample and the target training version, such as the current version sample, the neighboring version sample, or the expired version sample;

[0105] To quantify the degree of version deviation, this implementation calculates the generation strategy version number of the k-th sample block. With the current target training version number The absolute difference between them is defined as the version difference of the k-th sample block, reflecting which generation of policy generated the sample and the number of steps it has deviated from the current training objective:

[0106]

[0107] Furthermore, based on this version difference and the version decay coefficient used to adjust the rate of decrease in sample availability as version differences increase. Construct a version consistency score for the k-th sample block that reflects the availability of samples during the current training. ;

[0108] when When it is judged as the current version sample; when When it is judged as a nearby version sample; The sample was judged to be an expired version. This is the version tolerance window, used to control the range of version biases allowed to enter the training process;

[0109] The system obtains the number of newly written samples per unit time, the growth of the replay queue, the number of parameter synchronization waiting nodes, and the idle rate of the graphics processor and the backlog rate of the central processing unit. Based on the burst intensity calculation formula, the system comprehensively evaluates the current load situation and determines the load change information.

[0110] Among them, load change information refers to the degree of pressure change between current sample writing, training consumption and synchronization waiting, such as regular flow, burst flow or congested flow;

[0111] Therefore, based on the normalized value of the number of newly written samples per unit time, which reflects the strength of the sampling side's injection into the playback buffer, The normalized value of the playback queue growth slope, reflecting the degree of deviation between write speed and consumption speed. The normalized value of the number of synchronization waiting nodes, a parameter reflecting the diffusion situation in front of the synchronization barrier. And the normalized value of the indicators that reflect the coexistence of graphics processor idle rate and central processing unit backlog rate, which reflect the uncoordinated state of idle computing power and congested scheduling. Specifically, this is obtained by multiplying the GPU idle rate and CPU backlog rate in the same cycle and then normalizing the result. This constructs a burst intensity index representing whether the training system experiences a sudden increase in sample writes, an expansion of playback backlog, and a combined pressure of synchronous waiting diffusion at the current system cycle t.

[0112]

[0113] The calculation utilizes weighting coefficients to The contribution of dynamic balancing to write surges, queue backlogs, synchronization waits, and resource mismatches to load changes;

[0114] In AI agent training, industry-standard evaluation benchmarks typically focus on average throughput, average step latency, and synchronization wait time. When only a unified throughput metric is used to judge scheduling, it is susceptible to short-term high throughput interference and cannot identify the spread of synchronization wait. It is also susceptible to single-link spike noise interference, which makes it impossible to expose sample version misalignment and node minor degradation in a timely manner.

[0115] This implementation method decomposes the coarse indicator of average throughput into a multi-dimensional basis that can directly affect training control by collaboratively extracting stage information, latency information, node health information, version matching information and load change information, making the risk sources of the synchronization stage, sample stage and node stage visible respectively.

[0116] Especially under extreme conditions such as a sudden increase in playback writes, an increase in the number of synchronous waiting nodes, and a still high graphics processor idle rate, this implementation method can still pass... Identify resource mismatches instead of misjudging the problem as simply insufficient computing power;

[0117] In this implementation, all normalized values, such as synchronization wait queue depth, timeout configuration parameters, graphics processor frequency deviation, and link latency, are uniformly converted using the extreme value normalization formula. The specific calculation logic is as follows:

[0118]

[0119] If the extreme values ​​collected in the current scheduling cycle are equal, the normalized value is assigned to 0.5 by default. In addition, the first threshold and the second threshold and other judgment conditions involved in this implementation are not set arbitrarily, but are obtained by technicians through benchmark performance stress testing.

[0120] For example, the calibration rule for the first threshold is: when the concurrent pressure of the injected tasks is continuously increased, and the proportion of tasks with high-sensitivity latency urgency indicators stabilizes between 15% and 20% of the total number of tasks, the latency urgency indicator at that moment is taken as the first threshold; the second threshold is calibrated based on the critical indicator when the corresponding medium-sensitivity tasks account for 30% to 40% of the total.

[0121] The various weighting coefficients in the above weighting formula, such as to , to , to It can be set according to the influence of the corresponding indicator in the business through the analytic hierarchy process, or dynamically allocated according to the variance ratio of the corresponding indicator in the system.

[0122] This embodiment can achieve quantitative characterization of training tasks, sample versions, and node states based on a multi-dimensional label extraction mechanism, avoiding isolated processing of tasks, samples, and nodes, and ultimately solving the core technical defect of existing technologies that cause distortion of training scheduling basis due to the lack of fine-grained feature extraction mechanisms.

[0123] This involves determining the task type corresponding to the training task based on stage information, latency information, node health information, version matching information, and load change information. This includes determining the task type corresponding to the training task as a synchronization task when the training task is in the model update stage or gradient synchronization stage, the latency risk level determined based on latency information is a high sensitivity level, the version matching information represents the current version sample, and the node health information represents a stable node.

[0124] When the training task is in the sample preprocessing stage or gradient merging stage, and the latency risk level determined based on latency information is high-sensitivity or medium-sensitivity, the task type corresponding to the training task will be determined as a latency-priority task.

[0125] If the training task belongs to replay pool sorting, historical sample recoding or ordinary training batch consumption, and the corresponding training node is judged to be in a jitter state or performance degradation state and is allowed to carry regular computing tasks, the task type corresponding to the training task will be determined as a regular computing task.

[0126] If the training task is an offline evaluation, log processing, historical sample compression, or background analysis task, the task type corresponding to the training task will be determined as a background processing task.

[0127] Furthermore, acquire stage information, latency information, node health information, version matching information, and load change information, and classify training tasks according to preset task type determination rules to determine task types;

[0128] Among them, task type refers to the business category used by the training controller when allocating resources, such as synchronous task, latency-priority task, regular computing task and background processing task;

[0129] The classification result is not a new training task field, but a scheduling label generated by the training controller for subsequent queue isolation, CPU core binding, GPU execution flow priority adjustment and sample reading range control.

[0130] In specific implementation, when the training task is in the model update stage or gradient synchronization stage, and the latency information represents the high sensitivity level, the version matching information represents the current version sample, and the node health information represents the stable node, the task type corresponding to the training task is determined as a synchronization task.

[0131] The model update phase and gradient synchronization phase in this rule correspond to business links with high parameter consistency requirements. If a unified queue is still used for execution, the overall synchronization wait time is easily prolonged due to the consumption of low-quality samples or the participation of slightly degraded nodes.

[0132] In practical implementation, when the training task is in the sample preprocessing stage or gradient merging stage, and the latency information represents a high or medium sensitivity level, the task type corresponding to the training task is determined as a latency-priority task. This task does not necessarily directly execute parameter submission, but its long waiting time will increase the latency of subsequent critical stages, so it needs to be given priority over regular throughput tasks in terms of resources.

[0133] In specific implementation, when the training task belongs to replay pool sorting, historical sample recoding or normal training batch consumption, and the corresponding training node is determined to be in a jitter state or performance degradation state and is allowed to carry regular computing tasks, the task type corresponding to the training task is determined to be a regular computing task.

[0134] This classification allows the system to continue using lightly degraded nodes to perform non-critical computations, avoiding the idle computing resources caused by the traditional approach of stripping all nodes as soon as their state becomes abnormal.

[0135] In practice, when the training task is an offline evaluation, log processing, historical sample compression, or background analysis task, the task type corresponding to the training task is determined as a background processing task. This task type is used to take over the workload that does not have an immediate impact on the main training path and is the category with the lowest preemption priority.

[0136] This implementation method uses task type determination rules to split tasks with high version consistency requirements, high latency requirements, and high throughput requirements into separate tasks.

[0137] This implementation incorporates latency information, node health information, and version matching information into the determination process, thereby avoiding the simple execution of the same type of task according to a fixed priority at different stages and under different node conditions.

[0138] Furthermore, based on the above embodiments, the replay cache partition, node status results, stage switching delay risk value, warm-up control, resource scheduling, and execution process monitoring are explained;

[0139] Obtain sample version information, and partition the replay cache according to the correspondence between the sample generation strategy version number and the target training version number to determine the current version sample area, the neighboring version sample area, and the expired version sample area. The replay cache preferably adopts a circular buffer partition structure, with the sample block header information stored in the version index table in a sequential index manner, and the sample body data retained in the buffer slot of the corresponding partition.

[0140] The current version sample area is used to store samples that are consistent with the target training version, the neighboring version sample area is used to store samples whose version difference is within the version tolerance window, and the expired version sample area is used to store samples whose version difference exceeds the tolerance window. After the partition index is written, read-write locks can be used to control concurrent access. After reading, the sample body position is not changed, only the read pointer and available flag are updated.

[0141] The system acquires the graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency. Based on timing smoothing rules, it smooths the underlying state information of the nodes and determines the smoothed state information.

[0142] Among these measures, smoothing is used to suppress false triggering of scheduling caused by single-cycle spikes; to this end, based on the first... The original state vector of each training node in the current system cycle and the smoothed state vector of the previous period By satisfying And the smoothing coefficient used to balance the contribution of the current sampled value and the smoothing result of the previous period. The first state indicator that reflects the continuous changing trend of multiple state indicators of the node is constructed. The smoothed state vector of each training node in the current system cycle:

[0143]

[0144] Construct the first The smoothed state vector of each training node in the current system cycle ,in This is the original state vector collected during the current system cycle; when When it is large, when When the value is greater than a preset smoothing threshold, it is used to quickly identify features of continuous degradation; when... When the value is less than or equal to the preset smoothing threshold, it is used to suppress occasional state spikes;

[0145] Obtain smoothed state information, identify anomalies in training nodes according to anomaly judgment rules, determine abnormal node states, and classify states based on abnormal node states, corresponding degeneration count values ​​of training nodes, and anomaly duration to determine node state results.

[0146] Among them, node abnormal state refers to the operational abnormality identified based on the smoothed state information, such as instantaneous jitter, continuous degradation, or blocking trend; degradation count value refers to the cumulative number of times a node is judged as abnormal in multiple consecutive system cycles, which is used to avoid triggering strong actions with a single spike; abnormal duration refers to the degree to which the same type of abnormality is maintained in consecutive cycles.

[0147] Node status results include: allowed to host synchronous training tasks, only host asynchronous training tasks, pause receiving new version training tasks, and enter node recovery state;

[0148] Preferably, when a node is experiencing slight jitter but has not yet continued to expand, it can continue to carry out regular computing tasks; when a node continues to degrade, it is restricted to carrying only asynchronous training tasks; when a node experiences significant blocking or prolonged congestion, it suspends receiving new version training tasks or enters a node recovery state.

[0149] Obtain the synchronous waiting queue depth, sample write rate, and stage switching signal. Based on the stage switching risk calculation formula, evaluate the training stage switching trend and determine the stage switching delay risk value.

[0150] The stage switching delay risk value is used to characterize the degree of resource insufficiency that may occur when the current training system switches from the first load stage to the second synchronous load stage. This stage switching delay risk value is calculated based on the normalized value of the synchronous waiting queue depth, which reflects the prediction period of the upcoming synchronous queuing pressure. Normalized sample write rate values ​​reflecting the prediction cycle during which the replay cache may withstand write impacts. And the stage switching signal quantization value of the prediction cycle, which reflects whether the process state machine will enter sensitive stages such as parameter synchronization, high write replay, or online evaluation. This implementation constructs a stage switching delay risk value for the current system cycle that reflects the potential risks of synchronous waiting amplification, sample writing crowding, and stage transition overlap in the next stage:

[0151]

[0152] Wherein, parameter p represents the prediction time step relative to the current system period t; the quantization value of the stage switching signal. The discrete mapping rule is used to obtain the following: when the prediction period is about to enter the parameter synchronization stage, it is mapped as follows: When entering the high write phase of playback, it is mapped to When entering the online evaluation phase, it is mapped to When maintaining the current non-sensitive phase, the mapping is as follows ;

[0153] Weighting coefficients are used in the calculation. , , The impact of synchronous queuing pressure, write shock, and stage transition on preheating triggering is balanced; for continuous values ​​such as synchronous waiting queue depth and sample write rate, the prediction period can be obtained by linear extrapolation using a sliding window.

[0154] Quantization value for phase switching signal Then, a state machine look-ahead query mechanism is adopted, which directly obtains the system cycle by reading the pre-arranged task pipeline instruction sequence in the training controller, without limiting the specific physical time point.

[0155] Obtain task type and node status results. When the risk value of stage switching delay exceeds the risk threshold, control the resource preparation process according to the preheating rules and determine the preheating control instructions.

[0156] Among them, the preheating control command refers to whether to execute preheating and information about the preheating project. The execution target can be the communication path, the computing path, and the cache path.

[0157] When the task type is a synchronous task or a delay-priority task, and the node status result allows the corresponding task to be carried, at least one of the following can be triggered: pre-allocate communication cache, pre-start data prefetching thread, preload the next batch of sample indexes, submit placeholder tasks to the high-priority graphics processor execution stream, and expand the cache writable space.

[0158] The training controller encapsulates the preheating control command into a structured scheduling message and sends it through a long connection with the node agent module, parameter synchronization module and playback buffer module. After receiving the message, the node agent module parses the corresponding fields in the local command buffer and writes the result status bit back to the shared status area after execution, so that the training controller can read it in the next system cycle.

[0159] Obtain task type, node status results, version matching information, and load change information; perform resource scheduling for training tasks according to resource orchestration rules; and generate scheduling execution instructions.

[0160] Resource scheduling includes at least queue isolation, CPU core binding, GPU execution flow priority adjustment, parameter synchronization link allocation, sample partitioning, low-priority task suspension, and shared processing channel merging.

[0161] Synchronous tasks are preferentially assigned independent execution queues, key threads are bound to cores, high-priority graphics processor execution flows, and dedicated synchronization channels, and are restricted to reading only from the current version's sample area; latency-priority tasks are preferentially assigned high-priority prefetch threads and stable links.

[0162] Regular computation tasks are allowed to be executed on lightly degraded nodes and can read neighboring version sample regions; background processing tasks are executed under low load conditions and are suspended when high-priority tasks preempt.

[0163] The system acquires information such as sample version usage, GPU frequency deviation, link latency, CPU run queue length, and network congestion duration during the training process. Based on blocking, degradation, and recovery rules, it continuously monitors the training process and generates control flags.

[0164] The risk of sample expiration can be determined by comprehensively considering the number of expired samples to be included in the current training batch, the total number of samples in the current training batch, the average version difference of samples in the current training batch, and the average lag steps between the sample generation version and the target training version.

[0165] When the risk of sample expiration is higher than the expiration risk threshold, a control flag is generated to block expired version samples from entering the training batch, and the control flag is written to the status bit of the corresponding partition of the replay cache module, so that the sampler skips expired version samples in the next batch construction.

[0166] When a node's health information falls below the health threshold, a status flag is generated to adjust the corresponding training node to only carry asynchronous training tasks, and the training controller modifies the node role table.

[0167] When the network congestion duration exceeds the congestion duration threshold, a status flag is generated to switch the corresponding training node to the node recovery state, and its acceptance of new version training tasks is restricted. The training execution results, node status changes, sample consumption status, and synchronization statistics are continued to be written back to the scheduling parameter storage area maintained by the training controller for the next round of parameter updates.

[0168] In distributed AI training, common evaluation criteria include the first synchronization wait time, the initial batch startup delay, the average sample write blocking time, and the repetition degradation rate after node recovery.

[0169] Typically, remedial measures are taken only after risks have already manifested by migrating tasks or restarting nodes. This independent approach can easily lead to delays in scheduling actions and can also cause repeated starting and stopping of threads and frequent adjustments to batches during phase switching, making the system itself a new source of instability.

[0170] This implementation method identifies resource preparation gaps in advance by using the stage switching delay risk value, and performs communication, calculation and cache preheating when the triggering conditions are met, so that the initial synchronization waiting time and the initial batch start delay have a buffer in advance.

[0171] Meanwhile, this implementation uses smoothed state information and degradation count values ​​to classify states, which can accurately limit lightly degraded nodes to asynchronous tasks, rather than simply taking them all offline, thereby achieving a balance between resource utilization and training stability.

[0172] The application example is as follows: The training controller is deployed in the main control process of the training system. The replay cache module, parameter synchronization module and node agent module are independent running units that maintain communication with the main control process. Newly arrived training tasks are written to the task queue through the internal task interface of the training framework. Newly generated sample header information is written to the shared memory buffer maintained by the replay cache module. Node status information is periodically written to the status queue by the node agent module.

[0173] The training controller reads the aforementioned data in each system cycle and sequentially completes the generation of task description information, generation of sample version information, organization of node underlying status information, tag extraction, task type determination, node status result determination, stage switching risk assessment, issuance of preheating control instructions, and issuance of resource scheduling instructions.

[0174] After receiving the scheduling execution instruction, the training node updates its local execution queue, thread binding strategy, graphics processor execution flow priority, and sample reading range; the replay cache module modifies the status bit of whether expired samples are readable based on the control flag; and the parameter synchronization module selects the corresponding synchronization channel based on the link allocation instruction.

[0175] After a system cycle is completed, the training execution results, node state changes, sample consumption status, and synchronization statistics are rewritten into the scheduling parameter storage area for direct use in the next round of scheduling, thus forming a continuously running training control process.

[0176] In a preferred embodiment of the present invention, the abnormal state is determined by temporal smoothing of the underlying state information of the node, and the node state result is determined by combining the number of node degradations and the duration of the abnormality, including:

[0177] The graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length and sample read latency in the node's underlying state information are smoothed to obtain the smoothed state information.

[0178] The abnormal state of a node is determined based on the smoothed state information, and the degradation count value of the corresponding training node is updated based on the abnormal state of the node.

[0179] Based on the node's abnormal state, the number of degradations, and the duration of the abnormality, the training node is divided into any of the following states: jitter state, performance degradation state, blocked state, or offline state.

[0180] The corresponding node status result is determined based on the status division result of the training node. The node status result is any of the following: allowed to carry synchronous training tasks, only carry asynchronous training tasks, pause receiving new version training tasks, or enter the node recovery state.

[0181] The frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency of the training nodes are obtained. Based on the smoothing rules, the underlying state information of the training nodes is smoothed to obtain the smoothed state information.

[0182] Among them, the smoothed state information refers to the state result after suppressing single-cycle fluctuations. For example, link spikes, short-term read sample blockages, or instantaneous frequency drops in a single sampling will not be directly used as the basis for scheduling actions.

[0183] Obtain the smoothed state information, judge the anomalies of the training nodes according to the anomaly judgment threshold, determine the abnormal state of the nodes, and update the degradation count value of the corresponding training nodes based on the abnormal state of the nodes.

[0184] Among them, abnormal node status refers to whether the training node exhibits an operating state that exceeds the stable range in the current system cycle, such as a continuous increase in frequency deviation, a continuous increase in link latency, or repeated exceedances of sample reading latency.

[0185] The degradation count value refers to the cumulative number of times a node is identified as abnormal across multiple consecutive system cycles, used to distinguish between single jitters and repeated degradations;

[0186] The degradation count is preferably stored in the node role table maintained by the training controller, and is read and updated once in each system cycle, overwriting the old value after the update is completed;

[0187] Obtain the abnormal state of the node, the count of degradation times and the duration of the abnormality. According to the state classification rules, classify the training node into states, and determine the jitter state, performance degradation state, blocking state or offline state.

[0188] Among them, jitter state refers to the state in which an anomaly exists but the duration is less than or equal to the preset period threshold, such as the link latency rising briefly within a few system cycles; performance degradation state refers to the state in which the anomaly has occurred continuously and affects the training throughput or synchronization stability, such as the graphics processor frequency deviation and the memory fragmentation rate increasing together.

[0189] Blocked state refers to a state in which link latency, CPU run queue length, and sample read latency all exceed their respective preset threshold values, thus hindering task progress; offline state refers to a state in which a node can no longer participate in the training task.

[0190] The duration of an anomaly is not expressed in terms of specific physical time, but rather in terms of the number of consecutive system cycles, in order to meet the requirements of unified judgment in different deployment environments. In the specific quantitative state transition judgment, taking the duration of an anomaly as an example, a first cycle threshold and a second cycle threshold are set.

[0191] When a certain underlying state information exceeds the smoothing threshold and triggers an anomaly, if the duration of the anomaly is less than or equal to the first cycle threshold, it is only judged as a jitter state.

[0192] When the duration of the anomaly exceeds the first cycle threshold and the count of degradations accumulates, the state transitions to a performance degradation state.

[0193] If the duration of the anomaly increases further and is greater than or equal to the second cycle threshold, it indicates that the node is trapped in a deep queuing loop, and the state is determined to be blocked. This determination logic clarifies the stage evolution boundary from a single indicator exceeding the limit to the final state of lockdown.

[0194] Obtain the state partitioning results of the training nodes, and perform logical judgments on the training nodes according to the state mapping rules to determine the node state results;

[0195] The node status results include: allowed to carry synchronous training tasks, only carry asynchronous training tasks, pause receiving new version training tasks, and enter the node recovery state.

[0196] Preferably, the jitter state can be mapped to allowing the carrying of some regular computing tasks or asynchronous training tasks, the performance degradation state can be mapped to carrying only asynchronous training tasks, the blocking state can be mapped to pausing the reception of new version training tasks, and the offline state or the recovery observation stage after long-term blocking can be mapped to entering the node recovery state.

[0197] This implementation avoids the scheduling logic being overwhelmed by noise by using smoothing processing, degradation counting, and coordinated changes in the duration of abnormalities.

[0198] This implementation classifies training nodes into jitter state, performance degradation state, blocking state, and offline state, so that lightly degraded nodes can still handle non-critical loads without having to take them all offline indiscriminately.

[0199] The warm-up control command is used to instruct the execution of at least one of the following preparation operations: Communication preparation: pre-allocate the transmit buffer, perform memory page locking preprocessing on the parameter synchronization buffer, and maintain the heartbeat connection;

[0200] Computational preparation: Pre-start the data prefetching thread, preload the next batch of sample indexes, and submit placeholder tasks to the high-priority graphics processor execution stream;

[0201] Cache preloading: Expand the writable cache space, pre-allocate sample block information, and open a version-independent storage area.

[0202] Acquire the synchronous waiting queue depth, sample write rate and stage switching signal, predict the training stage switching according to the risk assessment rules, determine the stage switching delay risk value, and trigger warm-up when the stage switching delay risk value is greater than the risk threshold.

[0203] Among them, the stage switching signal refers to the next stage indication information output by the process state machine, such as the parameter synchronization stage, the playback high write stage, or the online evaluation stage is about to arrive.

[0204] Obtain the preheating trigger result, pre-allocate the parameter synchronization link according to the communication preparation rules, and perform communication preparation; where communication preparation refers to pre-allocating the send buffer, performing memory page preprocessing on the parameter synchronization buffer, and maintaining the heartbeat connection;

[0205] Pre-allocated send buffers are used to reduce buffer allocation waits at the start of synchronization; memory page preprocessing is used to avoid page mapping delays when the parameter synchronization buffer is used for the first time; and maintaining heartbeat connections is used to prevent slow recovery after the link has been idle for a long time.

[0206] The training controller encapsulates this type of action as a communication warm-up field, writes it into the command buffer listened to by the parameter synchronization module, and executes it after being parsed by the parameter synchronization module.

[0207] Obtain the preheating trigger result, and prepare the node-side sampling and graphics processor execution flow in advance according to the calculation preparation rules, and perform calculation preparation; wherein, calculation preparation refers to pre-starting the data pre-fetching thread, preloading the next batch of sample indexes and submitting placeholder tasks to the high-priority graphics processor execution flow;

[0208] The pre-start data prefetching thread is used to avoid the initial startup of the thread after the phase switch, the preload of the next batch of sample indexes is used to reduce batch build waiting, and the placeholder task is used to keep the high-priority execution flow of the graphics processor available; the placeholder task is preferably a lightweight computing task, which is only used to wake up the execution flow without consuming significant computing power.

[0209] Obtain the preheating trigger result, expand the writable space and version-independent storage area of ​​the replay cache according to the cache preloading rules, and perform cache preloading;

[0210] Among them, cache preloading refers to expanding the writable cache space, pre-allocating sample block information, and opening a version-independent storage area; expanding the writable cache space is used to cope with the rapid increase in sample write rate, pre-allocating sample block information is used to reduce the metadata request overhead when writing sample headers, and opening a version-independent storage area is used to maintain independent records between the current version sample, adjacent version samples, and expired version samples.

[0211] This implementation method uses a phased switching delay risk value for quantitative assessment, and performs communication preparation, computation preparation and cache preloading in advance when the conditions are met, so that resource preparation actions and service switching actions are consistent in the system cycle.

[0212] This allows for the control of link recovery delay, thread initial startup delay, and cache expansion delay before the switchover, rather than passively bearing them after the switchover.

[0213] Resource scheduling includes: performing at least one of the following on synchronous tasks: allocating a dedicated execution queue, binding critical threads to cores, using high-priority graphics processor execution flow, a dedicated synchronization channel, and using a batch strategy corresponding to a preset batch range, and limiting the reading range to only the current version sample area;

[0214] For latency-priority tasks, perform at least one of the following: allocate priority prefetching threads, bind preprocessing threads to cores, pre-set batching strategies corresponding to batch intervals, and gradient merging queues, and prioritize the allocation of stable network links;

[0215] For regular computing tasks, a first preset batch strategy, a first preset scheduling period, a shared synchronization channel, and a range of reads allowed from adjacent version sample areas are assigned, and regular computing tasks are allowed to be carried by jitter state nodes or performance degradation state nodes; idle time slices are allocated to background processing tasks, and suspension processing is performed when high-priority tasks preempt, and input / output bandwidth usage is limited.

[0216] Furthermore, obtain the synchronization task, configure dedicated resources for the synchronization task according to its business requirements, and generate the synchronization task scheduling result;

[0217] Among them, synchronization tasks refer to tasks in the model update phase or gradient synchronization phase that meet the conditions of version consistency and node stability.

[0218] The results of the synchronous task scheduling include at least the allocation of a dedicated execution queue, binding of key threads to cores, execution flow of high-priority graphics processors, dedicated synchronization channels, and batch strategies corresponding to preset batch intervals, and are limited to the reading range of the current version sample area only.

[0219] Dedicated execution queues are used to prevent background tasks or throughput tasks from preempting critical synchronous tasks; critical thread binding to cores is used to reduce CPU scheduling jitter; high-priority GPU execution flow is used to ensure that critical operators are launched first; and dedicated synchronization channels are used to reduce the queuing impact of shared network links.

[0220] Obtain latency-priority tasks, allocate priority resources to latency-priority tasks based on their processing characteristics, and generate latency-priority task scheduling results.

[0221] Among them, latency-priority tasks refer to tasks with high processing time requirements in the sample preprocessing stage or gradient merging stage; the scheduling result of latency-priority tasks includes at least the allocation of priority prefetching threads, core binding of preprocessing threads, batching strategy corresponding to the preset batch interval, gradient merging queue, and priority allocation of stable network links; here, stable network links refer to synchronization paths with link latency volatility lower than the preset volatility threshold, rather than paths with bandwidth greater than the preset bandwidth threshold.

[0222] Obtain regular computing tasks, configure throughput-type resources for regular computing tasks based on node status results and version matching information, and generate regular computing task scheduling results;

[0223] Among them, regular computing tasks refer to replay pool organization, historical sample recoding, or ordinary training batch consumption tasks; the scheduling results of regular computing tasks include at least the allocation of large batch strategies, long scheduling cycles, shared synchronization channels, and allowance of reading range of adjacent version sample areas, and allow them to be carried by jitter state nodes or performance degradation state nodes.

[0224] The first preset batch strategy is used to improve throughput, the first preset scheduling cycle is used to reduce the frequency of scheduling switching, and the shared synchronization channel is used to reduce the occupation of high-priority links.

[0225] Obtain background processing tasks, configure low priority for background processing tasks based on available resources, and generate background processing task scheduling results;

[0226] Among them, background processing tasks refer to offline evaluation, log organization, historical sample compression, or background analysis tasks; the scheduling results of background processing tasks include at least allocating idle time slices, performing suspension processing when high-priority tasks preempt, and limiting input and output bandwidth usage; input and output bandwidth usage limits can be achieved by setting bandwidth usage thresholds to prevent background tasks from crowding out sample writing paths.

[0227] This implementation binds synchronization tasks, latency-priority tasks, regular computing tasks, and background processing tasks to different resource strategies. This allows tasks with high version consistency and high latency sensitivity to be accurately assigned to stable nodes and high-priority resources, while lightly degraded nodes are used for regular throughput tasks, reducing the situation where high-quality resources are occupied by low-priority tasks for a long time.

[0228] It also includes: monitoring sample expiration risk, including: determining sample expiration risk based on the number of expired samples to be included in the current training batch, the total number of samples in the current training batch, the average version difference of samples in the current training batch, and the average lag steps between the sample generation version and the target training version; if the sample expiration risk is higher than the preset expiration risk threshold, at least one of the following shall be executed: blocking expired version samples from entering the training batch, reducing the sampling ratio of adjacent version samples, suspending the writing of new exploration samples to the old version partition, and narrowing the allowed version range.

[0229] Furthermore, the number of expired samples to be included in the current training batch, the total number of samples in the current training batch, the average version difference of the samples in the current training batch, and the average lag steps between the sample generation version and the target training version are obtained. Based on the expiration risk assessment rules, the risk of batch samples is calculated to determine the sample expiration risk.

[0230] Among them, the risk of sample expiration refers to the degree to which the training objective deviates due to the inclusion of old version samples in the current training batch; to quantify this risk, this implementation method comprehensively reflects the number of expired samples that are intended to enter the current training batch, which is a percentage of samples in the batch that have significantly deviated from the target version. Compared with the total number of samples in the current training batch that provide the risk normalization base The ratio, reflecting the average version difference of the current training batch of samples, which indicates the degree of deviation between the entire batch of samples and the target training version. To ensure dimensional consistency when adding multidimensional indicators, the average version difference is pre-normalized by dividing by the maximum allowed version deviation steps set by the system, and the average lag steps between the sample-generated version and the target training version, reflecting the update steps between sample generation and the current training objective, are also included. Lag reference steps compared to versions used for normalization with different lag levels under different training conditions The ratio of these ratios is used to construct the sample expiration risk for the current system cycle.

[0231]

[0232] Weighting coefficients are introduced in the calculation. , , To balance the combined impact of the proportion of expired samples, average version difference, and average lag steps on expiration risk;

[0233] The risk of sample expiration is assessed, and based on the expiration risk threshold, the replay cache and batch sampling process are controlled, with at least one risk suppression action executed.

[0234] Risk mitigation measures include blocking expired version samples from entering the training batch, reducing the sampling ratio of adjacent version samples, pausing the writing of new exploration samples to the old version partition, and narrowing the allowed version range. Blocking expired version samples from entering the training batch can be achieved by setting the readable status of the expired version sample area to unreadable.

[0235] Reducing the sampling ratio of neighboring versions can be achieved by modifying the version weight parameter of the sampler; pausing the writing of new exploration samples to the old version partition can be achieved by updating the partition write permission status through the replay cache module; narrowing the allowed version range can be achieved by reducing the version tolerance window;

[0236] Weighting coefficients in the formula for calculating the phase handover delay risk value , , And the weighting coefficients in the formula for calculating the risk of sample expiration. , , All configurations are based on an adaptive method of obtaining variance from historical time windows;

[0237] That is, the greater the variance of a certain sub-indicator sequence, such as the synchronous waiting queue depth of the prediction period, within the historical window, the higher its contribution to the uncertainty of the current system, and the higher the weight the system assigns to it.

[0238] Furthermore, the aforementioned risk thresholds and preset expiration risk thresholds are set based on the inflection point distribution characteristics statistically analyzed during the system trial operation phase. The criteria for calibrating the risk thresholds are as follows:

[0239] The lower bound of the stage switching delay risk value within the previous system cycle is set as the risk threshold when the system records the first synchronization waiting time that significantly exceeds the baseline by 200% during historical operation. This ensures that the timing of the preheating mechanism is neither too early, leading to resource idleness, nor too late, causing congestion.

[0240] In reinforcement learning or online policy training, common evaluation benchmarks typically include the volatility of the target policy's evaluation return, the average number of lag steps per sample, and the proportion of version hybridization within a training batch.

[0241] The method of detecting sample contamination only occurs after the training effect has significantly declined. It is not sensitive enough to version misalignment and cannot distinguish between two different reasons: insufficient current version samples and too many expired samples.

[0242] This implementation method incorporates the proportion of expired samples, the average version difference, and the average number of lag steps into the quantitative evaluation through the risk of sample expiration, so that the sampler can perform version blocking or ratio adjustment before the training effect deteriorates, thereby reducing cross-version experience pollution.

[0243] The application example is as follows: During deployment, the node agent module writes the collected underlying state information into the shared state buffer. After reading it, the training controller completes smoothing, anomaly detection, state division, and node state result generation. The synchronous waiting queue depth, sample writing rate, and stage switching signal are written into a unified risk assessment buffer by the parameter synchronization module and the replay cache module. Based on this, the training controller generates a warm-up control command and sends it to the parameter synchronization module, node agent module, and replay cache module respectively through a long connection.

[0244] The resource scheduling module writes the queue type of different tasks, the CPU core binding strategy, the GPU execution flow priority, the synchronization channel, and the sample reading range into the scheduling execution queue, which is then executed by each training node in the order of the queue.

[0245] The replay cache module updates the readable status bits and sampling weights of the expired version sample area and the adjacent version sample area based on the sample expiration risk calculation results. In this way, newly arriving samples, newly generated node states, and newly emerging synchronization pressures can be reprocessed in subsequent system cycles and trigger the next round of node classification, warm-up, and scheduling actions.

[0246] In a preferred embodiment of the present invention, the feedback of training execution results, node state changes, sample consumption status, and synchronization statistics to update scheduling parameters specifically includes: acquiring sample task information, sample version information, sample node status information, target training version information, training execution results, node state changes, sample consumption status, and synchronization statistics; and determining sample stage information, sample latency information, node health information, sample version difference information, and sample load change information based on the sample task information, sample version information, and sample node status information.

[0247] Input the sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the initial scheduling parameter model to obtain sample task type information, node status result information, and sample preheating indication information. The initial scheduling parameter model includes sub-models for task type judgment, status judgment, and preheating judgment.

[0248] Resource preparation score, node stability score and task adaptation score are determined based on sample task type information, node status result information and sample preheating indication information, respectively. Version matching score is determined based on sample version difference information and sample consumption information during training.

[0249] The optimization target is determined based on resource preparation score, node stability score, task adaptation score and version matching score, and the model parameters of the initial scheduling parameter model are iteratively optimized accordingly to obtain the target scheduling parameter model.

[0250] Furthermore, obtain sample task information, sample version information, and sample node status information. Based on feature extraction rules, organize the input samples to determine sample stage information, sample latency information, node health information, sample version difference information, and sample load change information.

[0251] Among them, sample stage information, sample latency information, node health information, sample version difference information and sample load change information correspond to the five categories of criteria already used in the training control process: stage, latency, health, version and load. Their construction method can follow the field parsing and quantization rules in the aforementioned training control process to ensure that the model input is consistent with the scheduling site.

[0252] Sample version difference information refers to the version deviation between the sample generation strategy version number and the target training version number; sample load change information refers to the degree of load change formed by the combined effects of write intensity, replay backlog and synchronization wait per unit time.

[0253] Acquire sample stage information, sample latency information, node health information, sample version difference information, and sample load change information. Based on the initial scheduling parameter model, predict the sample scheduling behavior to obtain sample task type information, node status result information, and sample preheating indication information.

[0254] The initial scheduling parameter model includes sub-models for task type judgment, status judgment, and warm-up judgment. Its function is to map the extracted scheduling criteria into executable scheduling results.

[0255] This model does not change the training task itself, but is used to optimize the selection of scheduling parameters. In terms of specific algorithm structure decomposition, the task type judgment sub-model and the state judgment sub-model adopt a multilayer perceptron network. The input layer receives the concatenated feature vector of the above information, which is mapped by the nonlinear activation function of the hidden layer and then mapped by the output layer to the probability distribution of each category.

[0256] The process of constructing the concatenated feature vector is as follows: A feature mapping dictionary is established; the quantized sample stage information (using one-hot encoding), the sample latency information in scalar form, and the node health information in scalar form are concatenated; and the sample stage information is processed according to the following steps: environmental interaction sampling, playback buffer writing, batch sampling, model update, gradient synchronization, and validation evaluation. A defined business phase, mapped as A one-hot encoded vector of dimension;

[0257] This vector is concatenated with scalar-form sample latency information, node health information, sample version difference information, and sample load change information through channel concatenation, thereby forming a globally fixed vector at the model input layer. 3D input feature vector;

[0258] The multilayer perceptron network contains a first hidden layer and a second hidden layer, with the number of neurons being 32 and 16 respectively. Both layers use the linear rectified activation function, and the output layer uses the normalized exponential function to output the probability of each category.

[0259] To simultaneously address the prediction needs of both tasks and nodes, the output layer adopts a dual-branch architecture. The first branch has an output dimension of 4, which is mapped by a normalized exponential function to obtain the probability distribution of task type judgment. The second branch also has an output dimension of 4, which is mapped by a normalized exponential function to obtain the probability distribution of node state result judgment.

[0260] The preheating judgment sub-model processes time-related switching feature sequences, uses a long short-term memory network, and utilizes its gating structure to accumulate the evolution trend of queue depth and write rate over multiple cycles. The output hidden state is mapped through a fully connected layer to obtain the preheating indication.

[0261] The Long Short-Term Memory (LSTM) network is designed with a time step of five historical system cycles as the preferred time step. The input dimensions for each time step are three quantized values: the depth of the synchronous waiting queue, the sample writing rate, and the stage switching signal. The hidden state dimension of the LSM network is set to 32 to ensure that the computational flow when features are converted into warm-up indicators is completely transparent.

[0262] Sample task type information is used to characterize which task category the corresponding task of the sample should be classified into; node status result information is used to characterize which task role the node should undertake; sample preheating indication information is used to characterize whether communication preparation, computation preparation and cache preloading need to be performed before the phase switch.

[0263] Obtain sample task type information, node status result information, and sample preheating indication information. Based on the scoring rules, quantify the resource preparation status, node stability status, and task adaptation status, and determine the resource preparation score, node stability score, and task adaptation score respectively.

[0264] Among them, the resource preparation score is a numerical value that represents the degree of resource preparation before the training phase switch, calculated based on the warm-up execution status, resource allocation completion status, and waiting time before the phase switch; the resource preparation score is used to represent the degree to which the warm-up action reduces the initial start delay of resources.

[0265] The preheating execution status reflects whether the preheating actions were completed as instructed; the resource allocation completion status reflects whether the queues, threads, execution flow, and links are ready; and the waiting time before the stage switch reflects whether there is significant queuing in the training before the switch.

[0266] The node stability score is a numerical value that represents the stability of a training node, calculated based on the frequency of abnormal node states, the duration of abnormal states, and the number of degradation counts. It is used to objectively characterize the state stability of a node over multiple consecutive system cycles.

[0267] The task adaptation score is a numerical value calculated based on the consistency between the business allocation result and the node status result, which represents the degree of matching between the business allocation and the node status. It is used to represent the degree of resource matching of the nodes where tasks of different priorities are located.

[0268] Obtain sample version difference information and sample consumption information during training, quantify sample version usage according to version consistency evaluation rules, and determine version matching score;

[0269] The version matching score is a numerical value that represents the consistency of sample versions, calculated based on sample version difference information and sample consumption information. It is used to quantify the consistency distribution between the samples consumed during training and the target training version. Sample consumption information may include the consumption ratio of the current version sample, the consumption ratio of the adjacent version sample, the number of times the expired version sample attempted to enter the batch, etc.

[0270] Obtain resource preparation score, node stability score, task adaptation score and version matching score. Based on the model optimization target generation rules, construct the target for the initial scheduling parameter model, determine the model optimization target, and perform iterative optimization on the model parameters of the initial scheduling parameter model accordingly to obtain the target scheduling parameter model.

[0271] Among them, the model optimization objective refers to the comprehensive objective used to measure whether the output of the scheduling parameter model meets the training control requirements. This objective is not an abstract indicator detached from business, but is directly constructed around four aspects: whether the resource preparation is sufficient, whether the nodes are stable, whether the task allocation is appropriate, and whether the sample version is matched.

[0272] In the specific iterative mechanism, the system transforms the above four scores into a loss function, which is the total loss function:

[0273]

[0274] Among them, resource readiness scores are calculated separately. Node stability score Task Adaptability Score Version matching rating Achieve the corresponding preset target score , , and The squared deviation between them, combined with the weighted weights. to Summation yields the result;

[0275] To ensure that no dimensional misalignment or numerical imbalance occurs when the loss function is summed, the four types of scores and their corresponding target full scores are pre-scaled and mapped to a uniform scale before the difference is calculated. Within the dimensionless standard interval; during iterative forward propagation, the loss value generated by the current prediction result compared to the target high-score state is calculated;

[0276] During backpropagation, the gradient descent algorithm is used to calculate the partial derivative of the loss function with respect to the weight matrix in the sub-model, and the parameters of each layer are updated along the direction of gradient descent. The iterative optimization of model parameters can be completed by repeatedly inputting samples, recording historical records in the cache, and adjusting the sub-model parameters in this way, so that the subsequent output is closer to the direction of high score.

[0277] This implementation introduces a parameter optimization process based on training execution feedback, so that task type judgment, status judgment, and warm-up judgment no longer remain at the static configuration level;

[0278] This implementation method incorporates four types of scores—resource preparation, node stability, task adaptation, and version matching—into the optimization objectives simultaneously, ensuring that the model adjustment direction aligns with the actual needs of training scheduling.

[0279] Inputting sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the initial scheduling parameter model yields sample task type information, node status result information, and sample preheating indication information, including:

[0280] Input the sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the sub-model used for task type determination to obtain sample task type information;

[0281] Node health information, node abnormal state information, and corresponding training node degradation count information are input into the sub-model used for state judgment to obtain node state result information; sample task type information, node state result information, synchronous waiting queue depth, sample write rate, and stage switching signal are input into the sub-model used for warm-up judgment to obtain sample warm-up indication information.

[0282] Furthermore, the sample stage information, sample latency information, node health information, sample version difference information, and sample load change information are obtained. Based on the task type, the sub-model is judged, and the scheduling input features are classified to obtain the sample task type information.

[0283] Among them, the task type judgment sub-model is used to output the task type to which the sample should belong, such as synchronous task, time-priority task, regular computing task or background processing task.

[0284] The input to this sub-model is still built around the training phase, latency requirements, node health, version deviation, and load changes, thus maintaining consistency with the scheduling logic;

[0285] Obtain node health information, node abnormal state information, and corresponding training node degradation count information. Based on the state judgment sub-model, classify the node carrying capacity to obtain node state result information.

[0286] Among them, the state judgment sub-model is used to output whether the corresponding node is allowed to carry synchronous training tasks, only carry asynchronous training tasks, pause receiving new version training tasks, or enter the node recovery state.

[0287] Node abnormal status information reflects the abnormal behavior of the node in the current or recent system cycle, while degradation count information reflects whether the abnormality has been repeated. The two together determine the node status result, rather than relying solely on a single sample value.

[0288] Obtain sample task type information, node status result information, as well as synchronous waiting queue depth, sample write rate and stage switching signal. Based on the preheating judgment sub-model, judge the switching preparation requirements and obtain sample preheating indication information.

[0289] Among them, the preheating judgment sub-model is used to output whether preheating needs to be performed before the stage switch and the preheating items that need to be performed; the synchronization waiting queue depth is used to characterize the synchronization pressure, the sample write rate is used to characterize the cache write impact, and the stage switch signal is used to characterize the stage type that the training process is about to enter. The three together affect whether the preheating action is triggered in advance.

[0290] This implementation method uses the task type judgment sub-model, state judgment sub-model and preheating judgment sub-model within the initial scheduling parameter model for continuous processing, so that the judgment result of the previous stage can be used as the input basis for the next stage.

[0291] For example, the warm-up judgment sub-model will only give a more positive warm-up instruction when the task type information and node status result information jointly indicate that the task is worth preparing in advance, thereby reducing ineffective warm-up.

[0292] This embodiment can generate scheduling results based on the division of labor among sub-models and the input-output relationship between them, avoiding the separation of each judgment process and ultimately solving the core technical defect of the existing technology that causes inconsistent scheduling actions due to the lack of coordination between tasks, nodes and preheating judgments.

[0293] The version matching score is determined based on the sample version difference information and the sample consumption information during the training process, including: determining the version difference information based on the version difference between the sample generation strategy version number and the target training version number;

[0294] Based on version difference information and sample consumption information during training, sample expiration risk information is determined; a version matching score, which characterizes the consistency of sample versions, is obtained by weighting the version difference information and sample expiration risk information.

[0295] Furthermore, the sample generation strategy version number and the target training version number are obtained. Based on the version difference between the two, the difference in sample version information is calculated to determine the version difference information.

[0296] Among them, version difference information refers to the degree of deviation between the sample and the current target training version. For example, if the version difference is less than or equal to the preset tolerance threshold, it indicates that the sample is within the current training window, and if the version difference is greater than the preset tolerance threshold, it indicates that the sample belongs to the old strategy stage.

[0297] Obtain version difference information and sample consumption information during the training process, analyze the sample version usage according to the sample expiration risk assessment rules, and determine the sample expiration risk information.

[0298] Among them, the sample expiration risk information refers to the degree of risk formed by the accumulation of version deviation when consuming samples during the training process. The sample consumption information can reflect the consumption ratio of the current version sample, the consumption ratio of the adjacent version sample, and the attempt to enter the batch of expired version samples. This risk information is used to transform the static version difference into the actual version usage risk that occurs during the training process.

[0299] Version difference information and sample expiration risk information are obtained. Based on a weighted calculation rule, the consistency of sample versions is quantified to obtain a version matching score. The version matching score comprehensively reflects both the similarity between the sample version and the target version, as well as the expiration risk during training.

[0300]

[0301] A consistency score is derived from version difference information, reflecting the degree of deviation of the static version of the sample. And a risk score that reflects the impact of samples on training consistency during real consumption and is calculated from sample expiration risk information. The calculations were performed jointly; the specific conversion process is as follows: the version difference information is recorded as... The consistency score is calculated using an exponential decay mapping function, and the formula is as follows:

[0302]

[0303] in, The decay control constant is denoted as ; the sample expiration risk information is denoted as . The risk score is calculated using an inverse proportional mapping function, and the formula is as follows:

[0304]

[0305] in, This represents the risk penalty coefficient. Through the aforementioned nonlinear mapping, the absolute difference and risk probability are uniformly converted to [a value]. Within the standard numerical space;

[0306] Weighting coefficients are used in the calculation. and The contribution of static version difference and consumption risk to version matching scoring is dynamically balanced. This weighting method avoids relying solely on version difference while ignoring consumption distribution, and also avoids relying solely on consumption ratio while ignoring version deviation itself. In the version matching scoring formula:

[0307]

[0308] Weighting coefficient and These are not static constants, but rather piecewise coefficients dynamically determined during the model training phase. In the initial exploration phase of training, due to a relatively high tolerance for outdated samples, these coefficients can be set... The focus is on the deviation of the static version.

[0309] In the later convergence and consolidation phase of model training, the use of outdated sample data in iterations can easily lead to model divergence. Therefore, dynamic adjustments are made... The focus is on punishing risky consumer behavior.

[0310] Meanwhile, when constructing the weighted total loss function by using the squared deviations of resource preparation score, node stability score, task adaptation score and version matching score as penalty terms, the summation weights of each penalty term dimension are dynamically scaled using the inverse proportional method based on the gradient norm of the corresponding sub-item in the previous iteration, thereby effectively avoiding a serious deviation in the overall gradient update direction due to an accidental surge in a certain score term.

[0311] In AI training scheduling, using only whether the sample version number is equal to the target version number as the criterion is an insufficiently sensitive approach. This is because nearby version samples may still be reasonably used in certain system cycles, while expired version samples, even if few in number, may have a significant impact at critical stages.

[0312] Conversely, judging solely based on consumption ratios easily overlooks the cumulative effect of version differences themselves. This implementation method uses a combined weighting of version difference information and sample expiration risk information to ensure that version matching scores simultaneously cover both the deviation and usage dimensions, making it more suitable for guiding the optimization of scheduling parameter models.

[0313] This embodiment can calculate version matching scores based on version difference information and sample expiration risk information, avoiding version consistency evaluation from being limited to a single version number comparison, and ultimately solving the core technical defect of the existing technology that leads to inaccurate sample version control due to the lack of a dynamic version evaluation mechanism.

[0314] The application example is as follows: During the operation of the training system, the training controller continuously collects scheduling inputs and execution feedbacks and writes them into the sample record cache;

[0315] The initial scheduling parameter model reads sample records from the cache according to the system cycle. First, the task type judgment sub-model outputs the sample task type information, then the status judgment sub-model outputs the node status result information, and the preheating judgment sub-model outputs the sample preheating indication information.

[0316] The training controller calculates resource preparation score, node stability score, task adaptation score and version matching score based on model output and execution feedback, and updates the model optimization target accordingly.

[0317] The target scheduling parameter model obtained after iterative optimization is reloaded into the training controller and used as the basis for task type judgment, status judgment and warm-up judgment in subsequent system cycles. In this way, the newly arrived sample task information, sample version information and sample node status information can continuously enter the model update process and have a reverse effect on the subsequent scheduling process through the new model parameters.

[0318] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for training an AI agent based on machine learning, characterized in that, include: Receive training tasks, sample streams, and node status information, and determine task description information, sample version information, node underlying status information, network congestion duration, synchronization wait queue depth, sample write rate, and stage switching signal, respectively. The replay cache is divided into current, nearby, and expired version sample areas based on sample version information; Based on the task description information, the sample version information, and the node underlying state information, determine the stage information, latency information, node health information, version matching information, and load change information, and determine the task type; The abnormal state is determined by time-series smoothing of the underlying state information of the node, and the node state result is determined by combining the number of node degradations and the duration of the abnormality. The switching delay risk value is determined based on the synchronous waiting queue depth, sample write rate, and stage switching signal. When the switching delay risk value is greater than the preset switching delay risk threshold, a preheating control command is executed based on the task type and node status result. Resource scheduling is performed based on task type, node status results, version matching information, and load change information. Monitor sample expiration risk, performance change trends determined by node underlying state information, and network congestion duration; When the risk of sample expiration, the node health information, or the network congestion duration trigger their respective preset thresholds, a flag is generated to block expired samples from entering the batch, the node is set to only carry asynchronous tasks, or the node is switched to recovery state. Feedback on training execution results, node status changes, sample consumption, and synchronization statistics is used to update scheduling parameters.

2. The AI ​​agent training method based on machine learning according to claim 1, characterized in that, The task description information includes task category identifier, process stage, and synchronization wait identifier; the sample version information includes sample generation strategy version number and target training version number; the node underlying state information includes graphics processor frequency deviation, video memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency. Based on the task description information, the sample version information, and the node underlying state information, stage information, latency information, node health information, version matching information, and load change information are determined, including: Based on the task category identifier, process stage, and synchronization wait identifier, the stage information is determined. The latency information is determined based on the remaining execution time window corresponding to the training task, the depth of the synchronization waiting queue, and the preset timeout configuration. Node health information is determined based on the graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length, and sample read latency. Version matching information is determined based on the version difference between the sample generation strategy version number and the target training version number; Load change information is determined based on the number of newly written samples per unit time, the growth of the playback queue, the number of nodes waiting for parameter synchronization, and the idle rate of the graphics processor and the backlog rate of the central processing unit.

3. The AI ​​agent training method based on machine learning according to claim 2, characterized in that, The task type corresponding to the training task is determined based on stage information, latency information, node health information, version matching information, and load change information, including: When the training task is in the model update stage or gradient synchronization stage, the latency risk level determined based on latency information is a high sensitivity level, the version matching information represents the current version sample and the node health information represents a stable node, the task type corresponding to the training task is determined to be a synchronization task. When the training task is in the sample preprocessing stage or gradient merging stage, and the latency risk level determined based on latency information is high sensitivity or medium sensitivity, the task type corresponding to the training task is determined as a latency priority task. If the training task belongs to replay pool sorting, historical sample recoding or normal training batch consumption, and the corresponding training node is determined to be in a jitter state or performance degradation state and is allowed to carry regular computing tasks, the task type corresponding to the training task is determined to be a regular computing task. If the training task is an offline evaluation, log processing, historical sample compression, or background analysis task, the task type corresponding to the training task will be determined as a background processing task.

4. The AI ​​agent training method based on machine learning according to claim 1, characterized in that, The underlying state information of nodes is smoothed over time to determine abnormal states. The node state result is determined by combining the number of node degradations and the duration of the abnormality, including: The graphics processor frequency deviation, memory fragmentation rate, bus retransmission count, link latency, CPU run queue length and sample read latency in the node's underlying state information are smoothed to obtain smoothed state information. Based on the smoothed state information, the abnormal state of the node is determined, and the degradation count value of the corresponding training node is updated based on the abnormal state of the node. Based on the abnormal state of the node, the count of the number of degradations, and the duration of the abnormality, the training node is divided into any of the following states: jitter state, performance degradation state, blocked state, or offline state. The corresponding node status result is determined based on the status division result of the training node. The node status result is any of the following: allowed to carry synchronous training tasks, only carry asynchronous training tasks, pause receiving new version training tasks, or enter the node recovery state.

5. The AI ​​agent training method based on machine learning according to claim 1, characterized in that, The preheating control command is used to instruct the execution of at least one of the following preparation operations: Communication preparation: pre-allocate the send buffer, perform memory page preprocessing on the parameter synchronization buffer, and maintain the heartbeat connection; Computational preparation: Pre-start the data prefetching thread, preload the next batch of sample indexes, and submit placeholder tasks to the high-priority graphics processor execution stream; Cache preloading: Expand the writable cache space, pre-allocate sample block information, and open a version-independent storage area.

6. The AI ​​agent training method based on machine learning according to claim 1, characterized in that, The resource scheduling includes: Perform at least one of the following on the synchronization task: allocate a dedicated execution queue, bind key threads to cores, use high-priority graphics processor execution flow, use a dedicated synchronization channel, use a batch strategy corresponding to a preset batch range, and limit the reading range to only the current version sample area; For latency-priority tasks, perform at least one of the following: allocate priority prefetching threads, bind preprocessing threads to cores, pre-set batching strategies corresponding to batch intervals, and gradient merging queues, and prioritize the allocation of stable network links; A first preset batch strategy, a first preset scheduling period, a shared synchronization channel, and a range for reading adjacent version sample areas are assigned to regular computing tasks, and the regular computing tasks are allowed to be carried by jitter state nodes or performance degradation state nodes. Idle time slices are allocated to background processing tasks, and suspension is performed when high-priority tasks preempt the task, while limiting input and output bandwidth usage.

7. The AI ​​agent training method based on machine learning according to claim 1, characterized in that, Also includes: Monitoring sample expiration risk includes: determining sample expiration risk based on the number of expired samples to be included in the current training batch, the total number of samples in the current training batch, the average version difference of samples in the current training batch, and the average lag steps between the sample generation version and the target training version; If the risk of sample expiration is higher than the preset expiration risk threshold, perform at least one of the following: block expired version samples from entering the training batch, reduce the sampling ratio of adjacent version samples, suspend the writing of new exploration samples to the old version partition, and narrow the allowed version range.

8. The method for training an AI agent based on machine learning according to claim 7, characterized in that, The feedback of training execution results, node state changes, sample consumption status, and synchronization statistics to update scheduling parameters specifically includes: Obtain sample task information, sample version information, sample node status information, target training version information, training execution results, node status changes, sample consumption status, and synchronization statistics; Based on sample task information, sample version information, and sample node status information, determine sample stage information, sample latency information, node health information, sample version difference information, and sample load change information; The sample stage information, sample latency information, node health information, sample version difference information, and sample load change information are input into the initial scheduling parameter model to obtain sample task type information, node status result information, and sample preheating indication information. The initial scheduling parameter model includes sub-models for task type judgment, status judgment, and preheating judgment. Resource preparation score, node stability score and task adaptation score are determined based on sample task type information, node status result information and sample preheating indication information, respectively. Version matching score is determined based on sample version difference information and sample consumption information during training. Based on the resource preparation score, the node stability score, the task adaptation score, and the version matching score, the model optimization target is determined, and the model parameters of the initial scheduling parameter model are iteratively optimized accordingly to obtain the target scheduling parameter model.

9. The method for training an AI agent based on machine learning according to claim 8, characterized in that, Inputting sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the initial scheduling parameter model yields sample task type information, node status result information, and sample preheating indication information, including: Input the sample stage information, sample latency information, node health information, sample version difference information, and sample load change information into the sub-model used for task type determination to obtain sample task type information; The node health information, node abnormal state information, and corresponding training node degradation count information are input into the sub-model used for state judgment to obtain node state result information. The sample task type information, node status result information, as well as the synchronous waiting queue depth, sample write rate, and stage switching signal are input into the sub-model used for preheating judgment to obtain sample preheating indication information.

10. The method for training an AI agent based on machine learning according to claim 8, characterized in that, Version matching scores are determined based on sample version difference information and sample consumption information during training, including: Based on the version difference between the sample generation strategy version number and the target training version number, the version difference information is determined; Based on the version difference information and sample consumption information during the training process, the sample expiration risk information is determined; A version matching score, which characterizes the degree of consistency between sample versions, is obtained by weighting the version difference information and the sample expiration risk information.

Images