Computing power processing method and system for large model training

Abstract

This invention relates to the fields of artificial intelligence and computing power scheduling technology, specifically to a computing power processing method and system for large model training. The method includes: receiving training data and corresponding status information uploaded by multiple work devices to form event records, and merging them into event clusters based on attribute information; obtaining risk scores, data validity half-life, and data readiness to generate task priority parameters; determining the expected completion delay of event clusters at each candidate training site, and accordingly determining timeout loss values; obtaining available computing power quota, and combining the timeout loss values ​​and task priority parameters to determine the primary execution site, allocate computing power quota, and computing power validity period; generating scheduling instructions to control the execution of training tasks; triggering task preemption or computing power release when an event cluster with insufficient remaining available computing power quota arrives or a higher-priority event cluster arrives; recording task execution result indicators and updating computing power allocation strategy parameters. This invention improves the effectiveness of training data screening.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computing power scheduling technology, specifically to a computing power processing method and system for large model training. Background Technology

[0002] In the scheduling of computing power for training large-scale models of embodied work equipment, existing methods typically use weighted indicators with manually set weights to calculate the priority of training tasks; However, this fixed calculation method cannot dynamically and comprehensively reflect the scarcity of training data, the validity period of the data, the expected completion time, and the dynamic available computing power of candidate training sites. Due to the lack of dynamic response capabilities, existing methods can lead to high-priority training tasks not being processed in a timely manner when training data or site resource status changes, and a disconnect between computing power allocation and actual task requirements. This results in the problem of data failure still ineffectively occupying computing resources, reducing the execution efficiency of large model training and the overall utilization rate of computing resources. Summary of the Invention

[0003] The purpose of this invention is to provide a computing power processing method and system for large model training, which can simultaneously reflect the scarcity of training data, the validity period of data, the expected completion delay, and the dynamic available computing power of the training site. This avoids problems such as high-priority training tasks not being processed in a timely manner, mismatch between computing power allocation and task requirements, and the continued occupation of computing power resources after training data expires, thereby improving the execution efficiency of large model training tasks and the utilization effect of computing power resources.

[0004] The objective of this invention can be achieved through the following technical solutions: A computational processing method for training large models includes: receiving training data and corresponding status information uploaded by the operating device to form event records, and merging them into event clusters based on the attribute information of the event records; Obtain evaluation metrics for event clusters, including risk score, data validity half-life, and data readiness, and generate task priority parameters based on the evaluation metrics. Determine the expected completion latency of the event cluster at each candidate training site, and determine the timeout loss value based on the expected completion latency and the effective half-life of the data; Obtain the available computing power quota reported by the candidate training sites, combine the timeout loss value and task priority parameters to calculate the computing power, determine the main execution site, allocate computing power quota and computing power validity period; The scheduling instructions are generated based on the computing power calculation results and sent to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; When an event cluster with insufficient remaining available computing power arrives or a higher priority event cluster arrives, computing power scheduling is performed. Record task execution result metrics and update the corresponding computing power allocation strategy parameters based on the task execution result metrics.

[0005] Furthermore, the system receives training data and corresponding status information uploaded by the operating equipment to form event records, and then merges these event records into event clusters based on their attribute information, including: Receive training data uploaded by the operating equipment, as well as the corresponding physical state, environmental state and task results, and form an event record; The completeness score of each event record is determined based on the data completeness of the training data; The similarity of any two event records is determined based on spatial distance, time interval, differences in environmental attributes, and failure type. Event records with similarity scores reaching a preset merging threshold are grouped into the same event cluster, and event records with completeness scores higher than a preset completeness threshold are retained first.

[0006] Furthermore, after generating event clusters by merging the attribute information of event records, the process also includes: determining the total number of original samples and the number of differential samples in the event cluster, and determining the repetition ratio based on the ratio of the difference between the total number of original samples and the number of differential samples to the total number of original samples. The motion trajectory difference value is determined based on the path deviation between the historical trajectory sequences of the operating equipment, and samples that meet any of the following conditions are retained from the event cluster: the risk score is greater than the preset risk threshold, the integrity score is greater than the preset integrity threshold, or the motion trajectory difference value is greater than the preset difference threshold.

[0007] Furthermore, evaluation metrics for event clusters are obtained, and task priority parameters are generated based on these metrics, including: Risk scores are determined based on task failure rate, human intervention rate, obstacle distance, and peak acceleration. The scarcity of an event is determined by the number of times the event occurs within a preset historical statistical period. Data readiness is determined based on the data upload completeness rate and the proportion of trainable samples; The expected improvement range is determined based on the expected change obtained from historical training effect statistics. Combining risk score, situation scarcity, data validity half-life, expected improvement, repetition rate, and data readiness, a comprehensive evaluation score is generated based on the following formula and written into the task priority parameters:

[0008] in, To comprehensively evaluate the score, To effectively evaluate the total number of indicators, For the first Effective evaluation indicators For the first The weights corresponding to each effective evaluation indicator The repetition ratio, This is the preset penalty adjustment coefficient.

[0009] Furthermore, the expected completion latency of the event cluster at each candidate training site is determined, and the timeout loss value is determined based on the expected completion latency and the effective half-life of the data, including: For each candidate training site, obtain the transmission rate from the event cluster to the candidate training site; The expected completion delay is determined based on the expected queuing time, expected training time, and expected verification time of the event cluster at each candidate training site. The timeout loss value is determined according to the following formula: ;in, This is the timeout loss value. To account for the expected completion time, For the effective half-life of the data, This is the preset time window coefficient; The overall evaluation score is adjusted based on the timeout loss value, and the transmission rate, timeout loss value, and adjusted overall evaluation score are written into the task priority parameter.

[0010] Furthermore, by combining timeout loss values ​​and task priority parameters, computing power is calculated to determine the primary execution site, allocated computing power quota, and computing power validity period, including: The available computing power is determined based on the allocatable quota of candidate training sites, historical execution success rate, resource utilization rate, interruption probability, and site power limit. Based on the adjusted comprehensive evaluation score, historical execution success rate, and expected queuing time, the priority index for the allocation of event clusters to each candidate training site is determined; The sites are sorted from highest to lowest according to their allocation priority index, with the highest priority site being designated as the primary execution site and the second highest priority site as the backup site. The allocated computing power quota is determined based on the basic computing power quota, risk score, situation scarcity, timeout loss value, and duplication ratio. After reserving emergency computing power from the available computing power quota, the reserve computing power quota is determined, and the effective period of computing power is determined based on the effective half-life of the data.

[0011] Furthermore, scheduling instructions are generated based on the computing power calculation results, including: Compare the risk score with the adjusted comprehensive assessment score. If both are greater than the corresponding preset threshold, the admission level is determined to be mandatory admission. If only the risk score or the adjusted comprehensive assessment score is greater than the corresponding preset threshold, the admission level is determined to be normal admission. If neither the risk score nor the adjusted comprehensive assessment score is greater than the corresponding preset threshold, the admission level is determined to be rejection. Compare the data readiness with the preset readiness threshold. If the data readiness is greater than or equal to the preset readiness threshold, the training method is determined to be full training; otherwise, it is determined to be partial training. Compare the expected completion time with the effective half-life of the data. If the expected completion time is less than or equal to the effective half-life of the data, the execution phase is determined to be full training and validation; otherwise, it is determined to be validation with a reduced validation set. The access level, training method, execution phase, allocated computing power quota, reserve computing power quota, primary execution site, backup site, and computing power validity period are written into the scheduling instruction.

[0012] Furthermore, when an event cluster with insufficient remaining available computing power arrives, or when a higher-priority event cluster arrives, computing power scheduling is performed, including: When a new event cluster arrives, and the current time is within the effective period of computing power, the data readiness is greater than or equal to the preset computing power availability threshold, and the remaining available computing power quota of the main execution site is greater than or equal to the allocated computing power quota, computing power locking is performed; when the remaining available computing power quota of the main execution site is less than the allocated computing power quota, the reserved quota is released, and the corresponding task is set to the pending scheduling state to trigger the computing power release operation. When a new event cluster with higher priority arrives, and the comprehensive evaluation score of the new event cluster is higher than that of the current event cluster, and the timeout loss value of the current event cluster reaches the preset timeout loss threshold, the current training is paused and the remaining computing power is released to trigger task preemption. When the interruption probability of the primary execution site is higher than the preset interruption threshold, the training state is saved, and unexecuted phase tasks are sent to the candidate sites to perform site switching.

[0013] A computing power processing system for large model training is applied to a computing power scheduling server, which connects to multiple job devices and candidate training sites, including: The event merging module is used to receive training data and corresponding status information uploaded by the operating equipment to form event records, and to merge and generate event clusters based on the attribute information of the event records; The task evaluation module is used to obtain evaluation metrics for event clusters, including risk score, data validity half-life, and data readiness, and to generate task priority parameters based on the evaluation metrics. The latency assessment and calculation module is used to determine the expected completion latency of event clusters at each candidate training site, and to determine the timeout loss value based on the expected completion latency and the effective half-life of the data; to obtain the available computing power quota reported by the candidate training sites, and to perform computing power calculation in combination with the timeout loss value and task priority parameters, and to determine the main execution site, allocate computing power quota, and computing power validity period; The scheduling and execution module is used to generate scheduling instructions based on the computing power calculation results and send them to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; when event clusters with insufficient remaining available computing power quota arrive or higher priority event clusters arrive, computing power scheduling processing is performed. The task feedback module is used to record task execution result indicators and update the corresponding computing power allocation strategy parameters based on the task execution result indicators.

[0014] The beneficial effects of this invention are: 1. This invention, by performing event recording, buffer verification, sequential merging, and versioning processing on training data and corresponding status information uploaded by multiple operating devices, can organize scattered and continuously changing multi-source training data into calculable event clusters, avoiding the impact of abnormal uploads, missing data, or duplicate old version records on subsequent computing power scheduling; by calculating completeness scores, merged similarity, duplication ratio, and action trajectory difference values, it can reduce the occupation of training applications by duplicate samples while retaining high-risk, high-completeness, and trajectory difference samples, thereby improving the effectiveness of training data screening. Attached Figure Description

[0015] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating the computing power processing method for training large models provided in this application embodiment; Figure 2 This is a schematic diagram of the computing power processing system for large model training provided in the embodiments of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Multiple embodied operation devices continuously upload training data and corresponding status information in the same training service. The computing power scheduling server is connected to the operation devices and multiple candidate training sites respectively. The operation devices can be embodied robots, unmanned transportation equipment or inspection equipment. However, in this embodiment, each operation device is used to perform the same type of embodied operation task and collects multimodal training data, physical status, environmental status and task results according to the task execution process.

[0018] Please see Figure 1A computing power processing method for large model training is applied to a computing power scheduling server. The server is connected to multiple job devices and candidate training sites, including: receiving training data and corresponding status information uploaded by job devices to form event records, and merging them into event clusters based on the attribute information of the event records. Obtain evaluation metrics for event clusters, including risk score, data validity half-life, and data readiness, and generate task priority parameters based on the evaluation metrics. Determine the expected completion latency of the event cluster at each candidate training site, and determine the timeout loss value based on the expected completion latency and the effective half-life of the data; Obtain the available computing power quota reported by the candidate training sites, combine the timeout loss value and task priority parameters to calculate the computing power, determine the main execution site, allocate computing power quota and computing power validity period; The scheduling instructions are generated based on the computing power calculation results and sent to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; When an event cluster with insufficient remaining available computing power arrives or a higher priority event cluster arrives, computing power scheduling is performed. Record task execution result metrics and update the corresponding computing power allocation strategy parameters based on the task execution result metrics.

[0019] After receiving the data uploaded by the working equipment, the computing power scheduling server associates the data according to the equipment number, task number, status identifier, location coordinates and collection period to form an event record; The event log also stores physical status, environmental status, task results, and data integrity information. After the event log is generated, it first enters a limited-capacity receiving buffer set separately for each device and task. The receiving buffer forms a sliding window according to the acquisition time sequence. The data in the window enters the merging process after being corrected by timestamp and checked for integrity. Records with complete data and successful correction are marked as mergeable; records lacking necessary attributes are marked as pending completion; records that fail correction or have inconsistent check values ​​are marked as abnormal and enter the retransmission waiting process. The status changes of the event log are kept in order by the sequence number of the same task, and the server only advances the merge window for consecutive sequence numbers that have been confirmed to be received; When the receive buffer reaches the preset occupancy ratio, the server suspends data reception confirmation for low-priority devices and only receives high-risk event records and records that have entered the retransmission process; when the occupancy ratio drops below the recovery threshold, normal reception resumes. Each device's receiving window is set with a maximum number of unacknowledged records. When this number is exceeded, only the latest status information and training data with high-risk markers are retained within the window. Other unacknowledged data are discarded in chronological order and returned to the retransmission range. The merge process uses parallel processing partitions divided by task number and status identifier, and records of the same event cluster are processed sequentially by the same processing partition; After an event record is formed, it is written to the event record storage area, and the event record identifier is sent to the event merging processing queue; The queue operates using a producer-consumer and confirmation status mechanism: records enter the queue in a pending state, are changed to a processed state after successful merging and processing, and enter a delayed retry state according to the number of retries when an exception occurs. Records that fail to merge after reaching the maximum number of retries will retain the reason for the exception and exit the current merging window without blocking the recording processing of other devices. The server transmits the processing identifier and version number of the same event cluster between event clusters, task priority parameters and scheduling instructions. Subsequent processing only accepts records of version no lower than the current version. Repeatedly arriving old version records will not be re-executed for computing power calculation. After the event merging process is completed, the server calculates the risk score, data validity half-life, and data readiness for each event cluster. The risk score is used to reflect the risk level corresponding to task failure, manual takeover, obstacle distance, and acceleration peak. The effective half-life of data is used to reflect the time range within which the corresponding training data can effectively improve the current embodied task. The effective half-life of data is determined by extracting training effectiveness decay curves for similar tasks over historical statistical periods from the server. These curves are constructed using an exponential decay model. ; in, For the initial gain, for Gain at any moment For time variables, The decay constant, fitted based on historical records, has the dimension of the reciprocal of time. The server solves this function to determine the time span corresponding to when the gain drops to half of the initial gain as the effective half-life of the data for that event cluster. The specific calculation formula is as follows: ; Data readiness is used to reflect the completeness of training data, the proportion of trainable samples, and the readiness of data to be transmitted to candidate training sites; the server writes the above metrics into the event cluster record and generates task priority parameters according to weighting rules. For each candidate training site, the server determines the expected completion latency of the event cluster by combining the data transmission rate, expected queuing time, expected training time, and expected verification time. The server then compares the estimated completion time with the effective half-life of the data to generate a timeout penalty value; this timeout penalty value is written into the task priority parameter to adjust the overall evaluation score. The server receives available computing power from candidate training sites and, in conjunction with the adjusted comprehensive evaluation score, site execution capability, and timeout loss value, determines the primary execution site, allocates computing power, and sets the validity period of the computing power. The scheduling server writes the admission level, training method, execution stage, computing power quota, candidate sites and validity period into the scheduling instruction, and sends it to the main execution site through the scheduling message queue; the main execution site completes data loading, training and verification according to the scheduling instruction. The computing power consumption, task completion status, training result indicators and reasons for anomalies generated during training execution are written into the task execution record. Among them, the training result indicators are task execution result indicators, which are used to uniformly reflect the execution effect, resource consumption and business improvement results of the training task. If the remaining available computing power cannot meet the new allocation requirements, or if the newly arrived event cluster has a higher priority, the scheduling server will suspend the current task, release unused computing power, or switch to a backup site according to preset conditions, and record the corresponding preemption, release, or switching status in the task execution log. After the scheduling instruction is sent, the main execution site returns a receipt confirmation and a quota lock result within the specified confirmation time limit; if no confirmation is returned, the instruction is in the pending confirmation, retry, and invalid states in sequence. The server will not include the corresponding quota in the occupied quota before confirming the lock result. After the number of retries reaches the upper limit, the temporary reserved quota is released and recalculated. The server maintains the reception status according to the event records; if an event record is not received within the current period, the event record remains in the pending completion state and is not directly used as a complete training sample for computing power calculation. If a candidate training site fails to report its available computing power, the server retains the resource record of the site in the previous valid period and marks the site as a site to be confirmed. Before obtaining confirmation of the resources in the current period, the site will not be designated as the main execution site. If the scheduling instruction fails to be sent, the server retains the pending state and re-verifies the site availability based on the validity period of the computing power.

[0020] In one alternative approach, training data and corresponding status information uploaded by the operating equipment are received to form event records, and event clusters are generated based on the attribute information of the event records, including: Receive training data uploaded by the operating equipment, as well as the corresponding physical state, environmental state and task results, and form an event record; The completeness score of each event record is determined based on the data completeness of the training data; The similarity of any two event records is determined based on spatial distance, time interval, differences in environmental attributes, and failure type. Event records with similarity scores reaching a preset merging threshold are grouped into the same event cluster, and event records with completeness scores higher than a preset completeness threshold are retained first.

[0021] In this embodiment, the multi-source training data includes image frames, point cloud frames, control sequences, and data related to the task results; the server calculates the integrity score of the event records based on data integrity rate, label confidence, timestamp alignment error, and packet loss rate. Timestamp alignment error and packet loss rate are negatively correlated with completeness score; data completeness rate and label confidence are positively correlated with completeness score. Data integrity rate is the ratio of the number of received and verified data items to the number of data items that should be received; tag confidence is the average of the confidence of each valid tag; timestamp alignment error is the average of the absolute differences between the corrected timestamps of each data source and the reference timestamps; and packet loss rate is the ratio of the number of lost data packets to the number of data packets that should be received. The server converts all ratios into directionally consistent values ​​between 0 and 1. For timestamp alignment errors and packet loss rates, a reverse processing function is used for normalization. ; in, The score after normalization. This is the error value. For the penalty slope parameter, The dimensions are The reciprocal of the dimensions, to ensure It is a dimensionless quantity; and the completeness score is calculated by weighting according to the completeness weights saved in advance for the same task type; when any denominator is zero, the corresponding item is marked as invalid, and only the weights of the remaining valid items are reassigned, without replacing them with zero or full values; The server calculates the spatial distance and time interval for any two event records, and extracts the environmental attributes and failure types corresponding to the two event records. Environmental attributes may include illumination range, obstacle type, and road surface adhesion status; spatial distance is the straight-line distance between two recorded location coordinates in a unified coordinate system, and time interval is the absolute difference between the acquisition times of the two records after correction; Environmental attribute differences are compared item by item. If the attributes are the same, they are marked as consistent; if the attributes are different, they are marked as inconsistent. Missing attributes are not included in the comparison. Failure type consistency is determined by whether the failure types are the same or different. Missing failure types are marked as unknown. The server uses a Gaussian kernel function to map spatial distance and time interval to similarity terms ranging from 0 to 1: ; ; in, For spatial distance similarity, For items with similar time intervals, For spatial distance, For time intervals, and These are the feature scale parameters calibrated based on historical data, where... Dimensions and same, Dimensions and The environmental attribute differences and failure type consistency are converted to 0 or 1 using a discrete indicator function; then, according to the pre-saved similarity weights for the same task type, the merged similarity is calculated using a linear weighted summation formula: ; in, For merging similarity, Index for summation of features ( ), For the first The weight of each item The first [unclear] after conversion based on spatial distance, time interval, environmental attribute differences, and failure type consistency. 1 similar item; All weights are non-negative and sum to one. The weights are labeled by using records that have been confirmed to belong to the same situation in the same type of historical events as positive samples and records that have been confirmed to belong to different situations as negative samples. The corresponding weights are called when the task type changes and are not directly shared across task types. When the similarity reaches the preset merging threshold, the corresponding event records are grouped into the same event cluster. The preset merging threshold is determined according to the positive sample similarity distribution and negative sample similarity distribution of similar tasks, and is a boundary value that can make the positive sample and negative sample distinction reach the preset requirements. When the number of positive and negative samples is insufficient, the threshold of the previous task category is used and the statistical level is recorded. After the event cluster is formed, the server sorts them according to the completeness score and prioritizes the retention of event records with a completeness score higher than the preset completeness threshold while retaining the differential samples. The preset completeness threshold is determined based on the completeness distribution required for similar tasks to reach the minimum trainable condition, and is saved in association with the task type and training method. When an event record is missing some multi-source training data but still contains physical state, environmental state, and task results, the server retains the record and lowers its completeness score, instead of deleting it directly. If an event record lacks location coordinates or collection period information, the corresponding spatial distance or time interval cannot be calculated. The server retains it as a record to be merged, and it will participate in the merging process again after the missing attributes are completed. If the environmental attributes of two event records cannot be compared, the difference in environmental attributes will not be used as a positive similarity basis to avoid incorrect merging due to incomplete data. If the number of valid sub-items in spatial distance, time interval, environmental attribute differences, and failure type is lower than the preset minimum, the server will not generate a merge similarity score, but will only mark the two records as pending merge status; the formed event cluster will only receive complete records with version numbers not lower than the current event cluster version.

[0022] In one alternative approach, after generating event clusters by merging the attribute information of event records, the method further includes: determining the total number of original samples and the number of different samples in the event cluster, and determining the repetition ratio based on the ratio of the difference between the total number of original samples and the number of different samples to the total number of original samples. The motion trajectory difference value is determined based on the path deviation between the historical trajectory sequences of the operating equipment, and samples that meet any of the following conditions are retained from the event cluster: the risk score is greater than the preset risk threshold, the integrity score is greater than the preset integrity threshold, or the motion trajectory difference value is greater than the preset difference threshold.

[0023] After the event clusters are merged, the server counts the total number of original samples that are included in the event cluster, and determines the number of differential samples based on spatial attributes, temporal attributes, environmental attributes, failure type, and action trajectory. The server determines the duplication ratio by the ratio of the difference between the original total number of samples and the number of differential samples to the original total number of samples, and writes the duplication ratio into the event cluster record; when the original total number of samples is zero, the server does not perform the ratio calculation and marks the duplication ratio as pending evaluation. When the number of differential samples is greater than the total number of original samples, the number of differential samples is limited to the total number of original samples before recording, and no duplicate ratio greater than one is generated. The server extracts path change information from the historical trajectory sequences saved by the working equipment, compares the motion trajectories of different working equipment within the same event cluster, and forms motion trajectory difference values. Specifically, for each sample within an event cluster, the server aligns its motion trajectory with a unified starting position, time reference, and sampling interval, and compares it segment by segment with the reference trajectory formed based on valid samples within the same event cluster. The path deviations between corresponding trajectory points are uniformly converted to the same length unit and then summarized. The deviations are then scaled according to the effective path length of the reference trajectory to obtain the dimensionless motion trajectory difference value corresponding to the sample. The preset difference threshold uses the same scale. When a sample lacks a historical trajectory sequence sufficient to complete trajectory alignment, the motion trajectory difference value of the sample is marked as a state to be evaluated, and the sample is not screened out based on this indicator. Before performing the sample retention judgment, the server uses the event record as the granularity to form a sample-level risk score based on the task failure flag, manual takeover flag, obstacle distance and acceleration peak corresponding to the sample. The event cluster risk score is obtained by aggregating retained samples during the task evaluation phase. Both use the same standardized boundary but have different recording granularity. For samples with a risk score higher than the preset risk threshold, they are retained even if the similarity between the sample and other samples is greater than the preset merging threshold. Samples with a completeness score higher than the preset completeness threshold are retained; samples with a motion trajectory difference value higher than the preset difference threshold are also retained; other duplicate samples are not used as independent complete training applications, but their occurrence frequency and covered device information are retained. If the total number of original samples for an event cluster is empty, the server does not calculate the duplication ratio and marks the event cluster as a data incomplete state. If the historical trajectory sequence is missing, the server does not generate action trajectory difference values ​​and does not filter samples based on this condition. In this case, sample retention is still performed based on risk score and completeness score. After the duplication ratio is formed, it is passed to the task evaluation process as input for task priority parameters.

[0024] In one alternative approach, evaluation metrics for event clusters are obtained, and task priority parameters are generated based on these metrics, including: determining risk scores based on task failure rate, human intervention rate, obstacle distance, and peak acceleration; and determining situation scarcity based on the number of times the event records occur within a preset historical statistical period. The data readiness level is determined based on the data upload completeness rate and the proportion of trainable samples; the expected improvement is determined based on the expected change obtained from historical training effect statistics. Combining risk score, situation scarcity, data validity half-life, expected improvement, repetition rate, and data readiness, a comprehensive evaluation score is generated based on the following formula and written into the task priority parameters: ; in, To comprehensively evaluate the score, To effectively evaluate the total number of indicators, For the first Effective evaluation indicators For the first The weights corresponding to each effective evaluation indicator The repetition ratio, This is the preset penalty adjustment coefficient.

[0025] The server calculates the risk score for event clusters based on task failure rate, manual takeover rate, obstacle distance, and peak acceleration. The server first converts each indicator into a standardized score with the same direction according to the percentile range of the same type of task statistics. Among them, the task failure rate, manual takeover rate and acceleration peak are positively correlated with the corresponding risk score, while the obstacle distance is negatively correlated with the corresponding risk score. The server then generates a risk score according to the preset risk weight. Specifically, the server reads valid samples of the same task category, status identifier, and execution stage within a preset historical statistical period, and determines the lower and upper limits of each indicator; indicators below the lower limit are calculated according to the lower limit, and indicators above the upper limit are calculated according to the upper limit; task failure rate, manual takeover rate, and acceleration peak are treated as positive indicators; obstacle distance is treated as a negative indicator. ; ; The calculation results are limited to between 0 and 1; when the upper limit is equal to the lower limit, the indicator is marked as statistically indiscriminate and does not directly participate in this weighting; when the number of valid samples of the same type of task is lower than the minimum statistical quantity, the server adopts the statistical boundary of the task category of the next higher level and saves the adopted statistical level in the evaluation record. The scarcity of an event is determined based on the number of valid occurrences of the event record within a preset historical statistical period. The server first filters records of the same type according to the event identifier, task type, and execution stage, and then compares the number of valid occurrences with a preset quantile table. When the frequency of occurrence is in the lowest quantile interval, the scarcity of the situation is set to the high scarcity interval value; when it is in the middle quantile interval, it is set to the medium scarcity interval value; and when it is in the highest quantile interval, it is set to the low scarcity interval value. Each interval corresponds to a pre-saved fixed dimensionless score, and the value of the high-scarcity interval is greater than that of the medium-scarcity interval, and the value of the medium-scarcity interval is greater than that of the low-scarcity interval. When there are no records of the same situation in the statistical period or the number of valid occurrences is less than the minimum statistical quantity, the situation scarcity is marked as statistically insufficient and is not directly given the highest scarcity. Data readiness is determined by the data upload completeness rate and the proportion of trainable samples. The data upload completeness rate is determined by the number of data items received within the event cluster and the number of data items that should be received. The proportion of trainable samples is determined by the number of samples that meet the requirements of timestamp alignment, valid labels, and missing rate not exceeding a preset upper limit, and the total number of samples. The server first obtains the two ratios mentioned above, and then synthesizes them according to the preset ready weights. When one of them is missing, only the other valid value is used in the calculation according to its original weight, and the missing status corresponding to the remaining weight is written into the evaluation record, without replacing it with the default full value or zero value. When the number of data items or the total number of samples that should be received is zero, only the original count is retained and the data readiness is marked as pending evaluation; no default score is generated. The server calculates the expected improvement based on historical training results statistics; the server retrieves historical tasks that have been trained and have business results before and after training within a preset historical statistical period, according to the task category to which the current event cluster belongs, and by the situation identifier, task type and execution stage. When the number of matched records reaches the minimum statistical limit, the difference in task success rate before and after training and the decrease in manual intervention rate are calculated respectively, and the expected improvement range is formed according to the preset improvement weight; when the number of matched records is insufficient, the expected improvement range is marked as to be supplemented and the score of this item is not directly increased; the statistical results of the expected improvement range also save the number of matched records, the statistical period and the corresponding task category. The server first converts the effective half-life of the data into a timeliness score of 0 to 1 according to the historical effective time window of similar tasks, and then performs weighted averages on the risk score, situation scarcity, timeliness score, expected improvement range and data readiness. The server only applies weighting to metrics that pass the validity check and proportionally distributes the weights of missing metrics to the remaining valid metrics. When the number of valid metrics is lower than the preset minimum, no executable comprehensive evaluation score is generated, and the event cluster is marked as pending evaluation. The repetition rate is deducted once after the weighted result is formed, and the comprehensive evaluation score after deduction is limited to a preset score range. Specifically, the server uses a formula combining weighted summation and proportional penalty to calculate the comprehensive evaluation score, and applies a piecewise truncation function to ensure that the score is always constrained within a logically closed interval: ; in, To comprehensively evaluate the score, To effectively evaluate the total number of indicators, For the first The effective indicators include risk score, situation scarcity, timeliness score, expected improvement, and data readiness. For the first The weights corresponding to each effective evaluation indicator The repetition ratio, This is a preset penalty adjustment coefficient; Risk score, situation scarcity, timeliness score, expected improvement, data readiness, and duplication rate all retain their original values, validity markers, and processed values; when the duplication rate is high, the comprehensive evaluation score is reduced accordingly; when the data readiness does not meet the minimum training conditions, only the event cluster is allowed to enter the local training or data completion process. The preset risk weights and various weighted weights are calibrated based on the indicators and training improvement results of similar tasks in the historical statistical period, and are limited to a range of zero to one with a total of one. The preset risk threshold, minimum statistical quantity, preset ready threshold, and minimum training condition are determined based on the historical quantile intervals and minimum number of trainable samples for the same type of task, and are saved separately when the task category changes, without directly mixing the thresholds of different task categories. The task priority parameters that include the comprehensive evaluation score should at least include the event cluster identifier, risk score, situation scarcity, timeliness score, expected improvement, repetition rate, data readiness, validity markers for each indicator, weights after reallocation, and the final comprehensive evaluation score; the server writes the task priority parameter records according to fixed fields, so that subsequent site sorting can directly read the fixed fields. If there are no valid statistical records for the task failure rate or manual takeover rate, the server will not directly treat the missing item as a high-risk item. Instead, it will form a substitute value based on the available statistical records for similar tasks and record the statistical source in the task priority parameter. If the expected improvement lacks historical training results, the server will not increase the score for that item, but will update it after obtaining the corresponding training results; the weight parameters are adjusted by the task execution result indicators, and the adjustment process is limited to the preset adjustment range; after the adjustment is completed, the server performs a consistency check on the total weights. If the adjusted weights exceed the preset range, they are corrected according to the boundary values ​​and will participate in the update again in the next statistical period.

[0026] In one alternative approach, the expected completion latency of the event cluster at each candidate training site is determined, and a timeout penalty value is determined based on the expected completion latency and the effective half-life of the data, including: For each candidate training site, obtain the transmission rate from the event cluster to the candidate training site; The expected completion delay is determined based on the expected queuing time, expected training time, and expected verification time of the event cluster at each candidate training site. The timeout loss value is determined according to the following formula: ;in, This is the timeout loss value. To account for the expected completion time, For the effective half-life of the data, This is the preset time window coefficient; The overall evaluation score is adjusted based on the timeout loss value, and the transmission rate, timeout loss value, and adjusted overall evaluation score are written into the task priority parameter.

[0027] The server obtains the transmission rate of the event cluster to each candidate training site by the data transmission status reported by the candidate training sites; for each candidate training site, the server reads the amount of data to be transmitted, the effective transmission rate, the estimated queuing time, the estimated training time, and the estimated verification time of the event cluster in sequence. The data transmission duration is determined based on the amount of data to be transmitted in the event cluster and the effective transmission rate. Only when all five data items are valid and use the same time unit will the data transmission duration, estimated queuing duration, estimated training duration, and estimated verification duration be added together to form the estimated completion delay. If any item is invalid, the station will be marked as having a delay pending confirmation, and the estimated completion delay result for that station will not be output. The server receives the transmission status of candidate training sites according to a fixed resource status cycle. After obtaining at least two consecutive valid sample values ​​in each cycle, the server first removes the sample values ​​that are interrupted, have timed out, or have failed to be verified, and then forms a rate sample set according to the same task type and data size. When the number of samples reaches the minimum statistical quantity, the lower quantile value of the set is taken as the effective transmission rate; otherwise, the statistical rate of the previous task category is used and the statistical level is marked. When the effective rate of two consecutive cycles is lower than the preset proportion of the current estimate, the expected completion delay is recalculated. When only a single cycle is abnormal, the cycle is marked as pending confirmation and the site ranking result is not changed immediately. The estimated queuing time, estimated training time, and estimated verification time are determined based on the records of similar tasks at the candidate training sites within a preset historical statistical period. When the number of valid records reaches the minimum statistical count, the server takes the median of the corresponding task type and training stage as the initial estimate, and then corrects it based on the observations in the most recent resource state period. When the number of valid records is insufficient, use valid statistical records from the previous task category or the same training method, and save the statistical source; The expected training duration is determined by the training sample size range, training method, and historical throughput records of the site. The expected validation duration is determined by the validation sample size range and validation method. Local training only calls the historical duration records corresponding to the local training and does not mix the historical records of the complete training. The server performs summation only when all sub-items have a uniform time unit and have not exceeded the valid observation period. If any sub-item exceeds the valid observation period, the server will request the resource status of that sub-item again. The server subtracts the product of the data's effective half-life and the preset time window coefficient from the estimated completion time to obtain the latency difference. If the data's effective half-life is a valid positive number and passes the validity check, and the latency difference is greater than zero, then the timeout penalty is determined according to the ratio of the latency difference to the data's effective half-life. If the latency difference is less than or equal to zero, the timeout penalty is set to zero. The calculation relationship is as follows: ; The effective half-life of the data and the expected completion delay use the same time unit, and the preset time window coefficient is a dimensionless quantity; the server verifies that the effective half-life of the data is greater than zero before calculation. If the verification fails, no division is performed, and no comprehensive evaluation score is deducted based on the comparison result of the incomplete delay. The preset time window coefficient is determined based on the correspondence between the time point when the data validity in historical training results begins to decline significantly and the data validity half-life, and is limited to between the preset lower limit and the preset upper limit; when there is a lack of similar historical training results, the time window coefficient of the previous level task category is used and the statistical level is recorded. The server adjusts the overall evaluation score in a deterministic order: first, it reads the original overall evaluation score for the candidate training site; if the timeout penalty is zero, the original overall evaluation score remains unchanged. If the timeout loss value is greater than zero, the deduction ratio will be determined according to the range in which the timeout loss value is located, and the original comprehensive evaluation score will be deducted once, and the result will be limited to above the preset minimum value. The rule table includes at least a first threshold, a second threshold, and corresponding first deduction ratio, second deduction ratio, and third deduction ratio; When the timeout loss value is greater than zero and does not exceed the first threshold, the first deduction ratio is used; when it is greater than the first threshold and does not exceed the second threshold, the second deduction ratio is used; when it is greater than the second threshold, the third deduction ratio is used. The first deduction ratio is less than the second deduction ratio, and the second deduction ratio is less than or equal to the third deduction ratio. The server writes the transmission rate, expected completion latency, timeout loss value, original comprehensive evaluation score and adjusted comprehensive evaluation score into the task priority parameters for candidate training sites, and the same site will not be deducted repeatedly in the same calculation. If a candidate training site does not return a transmission rate, the server marks the site as having an unknown transmission status and does not directly sort the sites after determining the transmission rate to zero. If the estimated queuing time, estimated training time, or estimated verification time is missing, and the server cannot form a complete estimated completion delay, the candidate training site will not be designated as the primary execution site for the time being. If the transmission rate, queuing time, or training time changes during the calculation, the server records the corresponding sampling time. When the observations for two consecutive resource status cycles exceed the corresponding historical fluctuation range, the expected completion delay is reassessed, and a single abnormal observation is only marked as pending confirmation.

[0028] In one alternative approach, computing power is calculated by combining timeout loss values ​​and task priority parameters to determine the primary execution site, allocate computing power, and determine the validity period of the computing power, including: The available computing power is determined based on the allocatable quota of candidate training sites, historical execution success rate, resource utilization rate, interruption probability, and site power limit. The priority index for allocating event clusters to each candidate training site is determined based on the following formula: ; in, To allocate priority indices, The adjusted overall assessment score, Based on historical execution success rate, This is a waiting penalty value converted based on the estimated queuing time. , , These are the preset priority index weights; The sites are sorted from highest to lowest according to their allocation priority index, with the highest priority site being designated as the primary execution site and the second highest priority site as the backup site. The allocated computing power quota is determined based on the basic computing power quota, risk score, situation scarcity, timeout loss value, and duplication ratio. After reserving emergency computing power from the available computing power quota, the reserve computing power quota is determined, and the effective period of computing power is determined based on the effective half-life of the data.

[0029] The server receives the allocable quota, resource utilization rate, interruption probability and site power limit from the candidate training sites, and calculates the available computing power quota in combination with the historical execution success rate. The server first deducts the quota that has been locked by other tasks and has not been released from the allocable quota reported by the candidate training sites to obtain the quota to be constrained; then it verifies the resource utilization rate, interruption probability and the site power limit in turn to calculate the quota. Resource utilization rate, interruption probability, and historical execution success rate are all used as dimensionless reduction or admission factors in the range of 0 to 1, and are not directly added to the computing power quota; the site power limit is converted into computing power quota based on the computing power conversion relationship per unit power consumption of the site and the current computing power validity period before participating in the upper limit constraint; When all data is valid and all constraints are passed, the server first applies resource consumption reduction to the amount to be constrained according to a preset order, then applies interruption reduction according to the interruption probability, takes the minimum value among the reduction result, the site power limit conversion amount, and the original amount to be constrained, and verifies the redeemable amount according to the historical execution success rate to obtain the available computing power amount. If any constraint lacks valid data, the site will be marked as a resource pending confirmation, and its available computing power will not be presumed based on missing data; if the amount of pending constraints is less than zero, it will be treated as zero. Resource utilization, interruption probability, and historical execution success rate are determined based on valid task records within a preset historical statistical period and updated using data from the most recent resource status period; the site power limit is taken from the continuously available limit in the site's current power management configuration and must not exceed the hardware's rated limit. The preset resource usage limit, preset interruption threshold, and minimum historical execution success rate are determined based on the site's historical resource fluctuations and the percentile range of task completion records; sites lacking a minimum number of statistics remain in a resource pending confirmation state. For each candidate training site, the server outputs the available computing power quota in the following order: read the allocable quota, deduct the locked quota, check the resource utilization rate, check the interruption probability, check the power limit, and confirm the historical execution success rate. For each combination of event clusters and candidate training sites, the server calculates an allocation priority index based on the adjusted comprehensive evaluation score, historical execution success rate, and expected queuing time. The server first filters candidate training sites with pending resource confirmation, unallocable resources, and missing expected completion latency. Then, it uses the adjusted comprehensive evaluation score and historical execution success rate as positive inputs, and converts the expected queuing time into a waiting penalty value of 0 to 1 according to the quantile interval within the same statistical period as a negative input. Finally, it performs a linear synthesis of the three inputs based on a preset priority index weight table. The specific calculation formula is as follows: ; in, To allocate priority indices, This serves as a positive input for the adjusted overall assessment score. For the historical execution success rate, a positive input item. The input item is the reverse of the waiting penalty value. These are the non-negative weights corresponding to the preset priority index weight table; The preset priority index weight table includes the comprehensive evaluation score weight, the historical execution success rate weight, and the waiting penalty weight. All three weights are non-negative and their sum is one. When the synthesis result is lower than the preset admission threshold, the site retains the accounting record but is not included in the sorting list; when all three inputs are valid, the allocation priority index is generated; when any one of the inputs is invalid, the allocation priority index for the site is not generated. The servers are sorted from high to low according to the allocation priority index. The candidate training site with the highest ranking is determined as the main execution site, and the candidate training site with the second highest ranking is determined as the backup site. When multiple candidate training sites have the same priority index, the ranking result is determined in the following order: shorter expected completion time, higher historical execution success rate, and candidate training site identification order. After sorting, the server writes the candidate training site identifier, priority index, sorting number, and whether it is selected as the main execution site or a candidate site into the computing power calculation record; if there is no valid historical record, the assigned weight of the same type of site is used, and the sorting result of the site is not improved separately. The server determines the allocated computing power quota based on the basic computing power quota corresponding to the task type, combined with risk score, situation scarcity, timeout loss value and repetition ratio. The server first reads the basic computing power quota, then retrieves the corresponding increase tiers according to the risk score, situation scarcity, and timeout loss value, and retrieves the corresponding deduction tiers according to the repetition ratio. The basic computing power quota, the various increase tiers, and the deduction tiers are combined into the initial allocated computing power quota. Reserve emergency computing power in the available computing power of the main execution site, use the amount after deducting emergency computing power as the trainable amount, and compare it with the minimum executable amount, the trainable amount and the site power limit converted amount in turn, and output the result of full allocation, restricted allocation or non-executable. Each indicator is only looked up once in the same calculation to avoid the same indicator being repeatedly amplified; the allocated computing power quota shall not exceed the trainable quota of the main execution site after constraints and deduction of emergency computing power, nor shall it exceed the executable quota after the site power limit is converted; when the trainable quota of a candidate training site is lower than the basic computing power quota, the restricted allocation result shall be determined according to the trainable quota, and the training method or training stage shall be reduced. The server reserves emergency computing power in the available computing power quota of the main execution site according to the preset emergency ratio. The reserved emergency computing power and the unallocated balance together form the standby computing power quota. The portion already included in the reserve computing power quota will not be included in the initial locked training quota, and the sum of the allocated computing power quota, reserve computing power quota, and locked but not released quota shall not exceed the available computing power quota for the same period. Emergency computing power is only used for necessary stages of task status recovery, site switching, or already approved tasks, and is not included in the initial training quota locked by the main execution site; the preset emergency ratio is determined based on the proportion of recovery consumption of historical interrupted tasks at the site, and is limited to the preset ratio range. The basic computing power quota is determined based on the average computing power required for the task type to achieve the target training effect within a historical period, and is constrained by the upper and lower limits of the historical computing power consumption quantile of that task type. The effective period of computing power is generated according to the effective time window corresponding to the effective half-life of data; when the effective half-life of data can only cover a local training phase, the effective period of computing power is truncated to the end of that local training phase. If none of the candidate training sites meet the minimum allocatable conditions, the server will not generate a primary execution site, but will keep the task in the scheduling queue; if only one candidate training site meets the conditions, that site will be the primary execution site, but no alternative site will be generated, and the site switching related fields will be set to unavailable. The minimum allocable conditions are jointly determined by the minimum training computing power corresponding to the task type, the minimum training stage that can be completed within the effective half-life of the data, and the converted quota of the site power limit. When the limited allocation quota can only complete partial training or reduced verification, the server synchronously updates the training method and execution stage in the scheduling instruction to avoid inconsistency between the allocation quota and the execution content.

[0030] In one alternative approach, a scheduling instruction is generated based on the computing power calculation result, including: comparing the risk score with the adjusted comprehensive evaluation score; if both are greater than the corresponding preset threshold, the admission level is determined to be mandatory admission; if only the risk score or the adjusted comprehensive evaluation score is greater than the corresponding preset threshold, the admission level is determined to be normal admission; if neither the risk score nor the adjusted comprehensive evaluation score is greater than the corresponding preset threshold, the admission level is determined to be rejection. Compare the data readiness with the preset readiness threshold. If the data readiness is greater than or equal to the preset readiness threshold, the training method is determined to be full training; otherwise, it is determined to be partial training. Compare the expected completion time with the effective half-life of the data. If the expected completion time is less than or equal to the effective half-life of the data, the execution phase is determined to be full training and validation; otherwise, it is determined to be validation with a reduced validation set. The access level, training method, execution phase, allocated computing power quota, reserve computing power quota, primary execution site, backup site, and computing power validity period are written into the scheduling instruction.

[0031] The server compares the risk score and the adjusted comprehensive assessment score with the corresponding preset thresholds. When both are greater than the corresponding preset thresholds, the access level in the scheduling instruction is set to mandatory access. When only some indicators reach the threshold, the access level is set to normal access. When the risk score, the adjusted comprehensive assessment score, or the necessary resource conditions do not meet the minimum conditions, the access level is set to rejection. The server compares the data readiness with a preset readiness threshold; when the data readiness reaches the preset readiness threshold, the training method is determined to be full training; when it does not reach the threshold, the training method is determined to be partial training; the server also compares the expected completion latency with the effective half-life of the data. If the expected completion time does not exceed the effective half-life of the data, the execution phase is determined to be complete training and validation; if the expected completion time exceeds the effective half-life of the data, the execution phase is determined to be validation with a reduced validation set. After the scheduling instruction is generated, the server writes the admission level, training method, execution stage, allocated computing power quota, reserve computing power quota, main execution site, backup site, and computing power validity period into the instruction record, and sends the instruction record to the main execution site; The main execution site performs task access verification, quota locking, data loading, training and verification based on the fields; if the scheduling instruction lacks a main execution site or allocated computing power quota, the server sets the instruction status to unexecutable and does not start the task for the training site. If the data readiness level drops below the preset readiness threshold after the instruction is sent, the main execution station suspends data loading and returns the task status to the scheduling server.

[0032] In one alternative approach, when an event cluster with insufficient remaining available computing power arrives, or when a higher-priority event cluster arrives, computing power scheduling is performed, including: When a new event cluster arrives, and the current time is within the effective period of computing power, the data readiness is greater than or equal to the preset computing power availability threshold, and the remaining available computing power quota of the main execution site is greater than or equal to the allocated computing power quota, computing power locking is performed; when the remaining available computing power quota of the main execution site is less than the allocated computing power quota, the reserved quota is released, and the corresponding task is set to the pending scheduling state to trigger the computing power release operation. When a new event cluster with higher priority arrives, and the comprehensive evaluation score of the new event cluster is higher than that of the current event cluster, and the timeout loss value of the current event cluster reaches the preset timeout loss threshold, the current training is paused and the remaining computing power is released to trigger task preemption. When the interruption probability of the primary execution site is higher than the preset interruption threshold, the training state is saved, and unexecuted phase tasks are sent to the candidate sites to perform site switching.

[0033] After receiving the scheduling instruction, the main execution site checks whether the current period is within the effective period of computing power, then checks whether the data readiness has reached the preset computing power availability threshold, and whether the remaining available computing power quota has reached the allocated computing power quota. The remaining available computing power reaches the allocated computing power quota, which means that the remaining available computing power quota is greater than or equal to the allocated computing power quota that needs to be locked for this task; When all conditions are met simultaneously, the main execution site locks the corresponding computing power quota and returns the locking result to the computing power scheduling server; the locking result includes the task identifier, the locked quota, and the locking status; the locking process consists of three consecutive states: application, reservation, and confirmation. After the server issues a request, the main execution site returns a reservation result within the lockout time limit; after the server receives the reservation result, it returns a confirmation, and only then does the main execution site set the task to the execution state; if any confirmation times out or quota verification fails, the main execution site releases the reserved quota and sets the task to the pending scheduling state, and the server does not include unconfirmed quotas in the remaining available computing power; If the comprehensive evaluation score of the new event cluster is higher than that of the current event cluster, and the timeout loss value of the current event cluster reaches the preset timeout loss threshold, the scheduling server sends a pause command to the main execution site; the comprehensive evaluation score used for preemption comparison is the adjusted comprehensive evaluation score under the same statistical period, the same weight table, and the same site accounting caliber. If the two scores have different source periods or weight versions, the server will recalculate them before performing the comparison, and will not directly trigger preemption. The timeout loss value of the current event cluster has reached the preset timeout loss threshold, which means that continuing to occupy the full allocation quota can no longer complete the original execution phase within the remaining effective period; If the current task has entered the uninterrupted data write-back, model merging, or result solidification stage, the server only records the preemption status and will pause execution after the most recent saveable checkpoint is reached. The main execution site pauses the current training, saves the current training state, and releases any remaining computing power that has not yet been consumed; the execution is paused at the training phase boundary or a preset checkpoint as the smallest control granularity. Before the main execution site returns a successful status save, the scheduling server will not treat the task as a switchable task, nor will it repeatedly allocate locked quotas that have not yet been confirmed for release; after receiving the release result, the scheduling server updates the site's available computing power record, and then performs computing power calculation based on the allocation result of the new event cluster. The pause command uses a state propagation method with a monotonically increasing version; after receiving the command, the main execution station first returns a receipt confirmation, and after completing the current minimum control granularity, it returns a pause confirmation and a state saving verification result; the scheduling server only changes the original task from the execution state to the paused state after the verification result is consistent and the quota release confirmation is received. If the confirmation message does not arrive within the specified time limit, the server will keep the original task quota frozen and enter a waiting confirmation state to avoid duplicate release or allocation due to repeated instructions; after reaching the retry limit, the task will be set to a paused abnormal state and the site calculation will be re-executed. If the interruption probability of the main execution site is higher than the preset interruption threshold, the main execution site saves the training status and unexecuted stage information, and returns a site switching request to the scheduling server; after the scheduling server confirms that the candidate site has available computing power, it issues unexecuted stage tasks to the candidate site. The alternate site continues the unfinished phase based on the saved training state and writes the site switching result into the task execution record; the site switching is not completed simultaneously with the pause action; Before issuing tasks for the unexecuted phase, the scheduling server verifies that the model version, data shard check value, training framework version, and checkpoint format of the candidate site are consistent with those of the original main execution site, and confirms that the starting position of the unexecuted phase corresponds to the saved training state. If there is no alternative site, or if the alternative site fails the verification of available computing power and task status integrity, the scheduling server will not perform site switching and will keep the task in a paused and pending scheduling state. In order to eliminate the time axis deviation between different sites, the logical sequence number and sampling time of the task stage are recorded at the same time when the status is saved. After receiving the data, the backup site uses the logical sequence number as the recovery location and corrects the timeout of the remaining stage based on the time offset between the main execution site and the backup site for the most recent resource cycle. If the time offset exceeds the allowable range, the latency assessment is performed again. If the computing power locking conditions are not met, the main execution site will not lock the quota or start training; if the training state cannot be saved during the task pause, the scheduling server will not perform the candidate site switch and will mark the task as a state saving failure. If the backup site does not return a confirmation of receipt, the original primary execution site will remain in a suspended state, and the released quota will not be redistributed to the task until the backup site confirmation is completed or the computing power is recalculated. If the computing power validity period of the current event cluster has expired, the server releases the remaining quota and moves the task from the executable queue to the invalidation record.

[0034] Please see Figure 2 A computing power processing system for training large models, applied to a computing power scheduling server, which connects to multiple job devices and candidate training sites, including: The event merging module is used to receive training data and corresponding status information uploaded by the operating equipment to form event records, and to merge and generate event clusters based on the attribute information of the event records; The task evaluation module is used to obtain evaluation metrics for event clusters, including risk score, data validity half-life, and data readiness, and to generate task priority parameters based on the evaluation metrics. The latency assessment and calculation module is used to determine the expected completion latency of event clusters at each candidate training site, and to determine the timeout loss value based on the expected completion latency and the effective half-life of the data; to obtain the available computing power quota reported by the candidate training sites, and to perform computing power calculation in combination with the timeout loss value and task priority parameters, and to determine the main execution site, allocate computing power quota, and computing power validity period; The scheduling and execution module is used to generate scheduling instructions based on the computing power calculation results and send them to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; when event clusters with insufficient remaining available computing power quota arrive or higher priority event clusters arrive, computing power scheduling processing is performed. The task feedback module is used to record task execution result indicators and update the corresponding computing power allocation strategy parameters based on the task execution result indicators.

[0035] The computing power processing system in this embodiment is deployed on a computing power scheduling server; the event merging module receives multi-source training data and status information uploaded by multiple working devices, forms event records, and merges events according to spatial distance, time interval, environmental attribute differences, and failure type; the event merging module writes the event cluster number, duplication ratio, difference sample information, and completeness score into the event cluster storage record for the task evaluation module to call. The task evaluation module reads risk-related data, occurrence frequency, data completeness, proportion of trainable samples, effective half-life of data, and historical training results from the event cluster storage records, and generates risk scores, situation scarcity, data readiness, expected improvement range, and task priority parameters; the task evaluation module then transmits the comprehensive evaluation score and corresponding evaluation indicators to the delay evaluation and accounting module. The latency assessment and calculation module receives transmission rate, allocable quota, estimated queuing time, estimated training time, estimated verification time, historical execution success rate, resource utilization rate, interruption probability, and site power limit from candidate training sites. This module calculates the expected completion latency and timeout loss for each candidate training site, and determines the available computing power, allocation priority index, primary execution site, candidate site, allocated computing power, reserve computing power, and computing power validity period. The scheduling execution module generates scheduling instructions based on the results output by the delay assessment and calculation module, and sends the scheduling instructions to the main execution station; The primary execution site performs access verification, computing power locking, data loading, training, verification, pausing, state saving, computing power release, and candidate site switching according to scheduling instructions; The scheduling and execution module saves various status changes and performs computing power scheduling when an event cluster with insufficient remaining available computing power arrives or a higher priority event cluster arrives. The task feedback module receives task execution result indicators reported by the main execution site and the backup site. The task execution result indicators include computing power consumption, quota fulfillment status, task completion status, timeliness completion status, site interruption status, and business improvement results. The task feedback module updates the site’s historical execution success rate, task type basic computing power quota and task priority related weights based on the above indicators, and writes the updated computing power allocation strategy parameters back to the task evaluation module and the latency evaluation and accounting module. The modules communicate processing results through event logs, event cluster logs, task priority parameters, computing power calculation logs, scheduling instructions, and task execution logs. If any module fails to obtain a valid record from the previous module, it does not directly generate the execution result for the next stage, but instead marks the corresponding task as pending, incomplete, or unexecutable. After receiving data completion, site resource confirmation, or task status recovery information, the server re-executes the incomplete processing steps based on the corresponding records, thereby maintaining the data correspondence between event merging, computing power calculation, task execution, and result feedback.

Claims

1. A computational power processing method for training large models, applied to a computational power scheduling server, wherein the server is connected to multiple job devices and candidate training sites, characterized in that, include: The system receives training data and corresponding status information uploaded by the operating equipment to form event records, and then merges these event records to generate event clusters based on their attribute information. The evaluation metrics for the event clusters are obtained, and task priority parameters are generated based on these metrics. The evaluation metrics include risk score, data effective half-life, and data readiness. The data effective half-life characterizes the time range within which the training data for the event clusters can effectively improve the current task; specifically, the training gain follows an exponential decay model. The time span corresponding to the initial gain decaying to half of the initial gain, i.e. ,in For the initial gain, for Gain at any moment For time variables, The decay constant is fitted based on historical records; The step of generating task priority parameters based on the evaluation indicators includes: standardizing each evaluation indicator, then weighting and summing them according to preset weights to generate a comprehensive evaluation score. ,in To comprehensively evaluate the score, To effectively evaluate the total number of indicators, For the first Effective evaluation indicators For the first The weights corresponding to the effective evaluation indicators are determined, and the comprehensive evaluation score is written into the task priority parameter. The expected completion delay of the event cluster at each candidate training site is determined, and a timeout loss value is determined based on the expected completion delay and the effective half-life of the data; the timeout loss value is determined according to the formula. Confirmed, among which This is the timeout loss value. To account for the expected completion time, For the effective half-life of the data, This is the preset time window coefficient; Obtain the available computing power quota reported by the candidate training sites, and perform computing power calculation by combining the timeout loss value and the task priority parameter to determine the main execution site, allocate computing power quota, and computing power validity period; The scheduling instructions are generated based on the computing power calculation results and sent to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; When an event cluster with insufficient remaining available computing power arrives or a higher priority event cluster arrives, computing power scheduling is performed. Record the task execution result indicators, and update the corresponding computing power allocation strategy parameters based on the task execution result indicators.

2. The computational processing method for training large models according to claim 1, characterized in that, The system receives training data and corresponding status information uploaded by the operating device to form event records, and merges these event records into event clusters based on their attribute information, including: The system receives training data uploaded by the operating equipment, along with the corresponding physical state, environmental state, and task results, and forms the event record. The completeness score of each event record is determined based on the data completeness of the training data. The similarity of any two event records is determined based on spatial distance, time interval, differences in environmental attributes, and failure type. Event records whose similarity reaches a preset merging threshold are grouped into the same event cluster, and event records whose completeness score is higher than the preset completeness threshold are preferentially retained.

3. The computational processing method for large model training according to claim 2, characterized in that, After generating event clusters by merging the attribute information of the event records, the process also includes: Determine the total number of original samples and the number of differential samples for the event cluster, and determine the repetition ratio based on the ratio of the difference between the total number of original samples and the number of differential samples to the total number of original samples; The motion trajectory difference value is determined based on the path deviation between the historical trajectory sequences of the operating equipment, and samples that meet any of the following conditions are retained from the event cluster: the risk score is greater than a preset risk threshold, the integrity score is greater than the preset integrity threshold, or the motion trajectory difference value is greater than a preset difference threshold.

4. The computational processing method for training large models according to claim 3, characterized in that, Obtain the evaluation metrics for the event cluster, and generate task priority parameters based on the evaluation metrics, including: The risk score is determined based on the task failure rate, human intervention rate, obstacle distance, and peak acceleration. The scarcity of a situation is determined based on the number of times the event record occurs within a preset historical statistical period; wherein, the scarcity of a situation is used to characterize the rarity of the situation corresponding to the event record in historical statistics, the fewer the occurrences, the larger the value of the scarcity of a situation, and the more the occurrences, the smaller the value of the scarcity of a situation; The data readiness is determined based on the data upload completeness rate and the proportion of trainable samples; The expected improvement range is determined based on the expected change obtained from historical training effect statistics. Combining the risk score, situation scarcity, data validity half-life, expected improvement, repetition rate, and data readiness, a comprehensive evaluation score is generated based on the following formula and written into the task priority parameter: ； in, To comprehensively evaluate the score, To effectively evaluate the total number of indicators, For the first Effective evaluation indicators For the first The weights corresponding to each effective evaluation indicator The repetition ratio is... The preset penalty adjustment coefficient is used; the effective evaluation index is an evaluation index that has passed the effectiveness verification, including the risk score, the scarcity of the situation, the timeliness score corresponding to the effective half-life of the data, the expected improvement range, and the data readiness.

5. The computational processing method for training large models according to claim 4, characterized in that, Determine the expected completion delay of the event cluster at each candidate training site, and determine the timeout loss value based on the expected completion delay and the effective half-life of the data, including: For each candidate training site, obtain the transmission rate from the event cluster to the candidate training site; The expected completion delay is determined based on the expected queuing time, expected training time, and expected verification time of the event cluster at each candidate training site; The timeout loss value is determined according to the following formula: ； in, This is the timeout loss value. To account for the expected completion time, For the effective half-life of the data, This is the preset time window coefficient; The overall evaluation score is adjusted based on the timeout loss value, and the transmission rate, the timeout loss value, and the adjusted overall evaluation score are written into the task priority parameter.

6. The computational processing method for training large models according to claim 5, characterized in that, By combining the timeout loss value and the task priority parameter, computing power is calculated to determine the main execution site, the allocated computing power quota, and the validity period of the computing power, including: The available computing power quota is determined based on the allocatable quota of candidate training sites, historical execution success rate, resource utilization rate, interruption probability, and site power limit. Based on the adjusted comprehensive evaluation score, the historical execution success rate, and the expected queuing time, the allocation priority index of the event cluster in each candidate training site is determined; According to the allocation priority index, the sites are sorted from high to low, and the one with the highest priority index is determined as the main execution site, and the one with the second highest priority index is determined as the backup site. The allocated computing power quota is determined based on the basic computing power quota, the risk score, the scarcity of the situation, the timeout loss value, and the duplication ratio. After reserving emergency computing power from the available computing power quota, the reserve computing power quota is determined, and the effective period of the computing power is determined based on the effective half-life of the data.

7. The computational processing method for training large models according to claim 6, characterized in that, Based on the computing power calculation results, scheduling instructions are generated, including: The risk score is compared with the adjusted comprehensive evaluation score. If both are greater than the corresponding preset threshold, the admission level is determined to be mandatory admission. If only the risk score or the adjusted comprehensive evaluation score is greater than the corresponding preset threshold, the admission level is determined to be normal admission. If neither the risk score nor the adjusted comprehensive evaluation score is greater than the corresponding preset threshold, the admission level is determined to be rejection. The data readiness is compared with a preset readiness threshold. If the data readiness is greater than or equal to the preset readiness threshold, the training mode is determined to be full training; otherwise, it is determined to be partial training. Compare the expected completion time with the effective half-life of the data. If the expected completion time is less than or equal to the effective half-life of the data, the execution phase is determined to be complete training and validation; otherwise, it is determined to be reduced validation set validation. The admission level, the training method, the execution stage, the allocated computing power quota, the reserve computing power quota, the primary execution site, the backup site, and the computing power validity period are written into the scheduling instruction.

8. The computational processing method for training large models according to claim 6, characterized in that, When an event cluster with insufficient remaining available computing power arrives, or when a higher priority event cluster arrives, computing power scheduling is performed, including: When a new event cluster arrives, and the current time is within the validity period of the computing power, the data readiness is greater than or equal to the preset computing power availability threshold, and the remaining available computing power of the main execution site is greater than or equal to the allocated computing power, computing power locking is performed; when the remaining available computing power of the main execution site is less than the allocated computing power, the reserved quota is released, and the corresponding task is set to a pending scheduling state to trigger a computing power release operation. When a new event cluster with higher priority arrives, and the comprehensive evaluation score of the new event cluster is higher than the comprehensive evaluation score of the current event cluster, and the timeout loss value of the current event cluster reaches the preset timeout loss threshold, the current training is paused and the remaining computing power is released to trigger task preemption. When the interruption probability of the primary execution site is higher than the preset interruption threshold, the training state is saved, and unexecuted phase tasks are sent to the candidate sites to perform site switching.

9. A computing power processing system for large model training, applied to a computing power scheduling server, wherein the server is connected to multiple job devices and candidate training sites respectively, for implementing the computing power processing method for large model training as described in any one of claims 1 to 8, characterized in that, include: The event merging module is used to receive training data and corresponding status information uploaded by the operating equipment to form event records, and to merge and generate event clusters according to the attribute information of the event records; The task evaluation module is used to obtain evaluation indicators for the event cluster, including risk score, data effective half-life and data readiness, and generate task priority parameters based on the evaluation indicators. The delay assessment and calculation module is used to determine the expected completion delay of the event cluster at each candidate training site, and to determine the timeout loss value based on the expected completion delay and the effective half-life of the data. Obtain the available computing power quota reported by the candidate training sites, and perform computing power calculation by combining the timeout loss value and the task priority parameter to determine the main execution site, allocate computing power quota, and computing power validity period; The scheduling and execution module is used to generate scheduling instructions based on the computing power calculation results and send them to the main execution station to control the main execution station to execute large model training tasks based on the allocated computing power quota; When an event cluster with insufficient remaining available computing power arrives or a higher priority event cluster arrives, computing power scheduling is performed. The task feedback module is used to record task execution result indicators and update the corresponding computing power allocation strategy parameters based on the task execution result indicators.

Images