Model training detection method and electronic device

Abstract

The application discloses a model training detection method and an electronic device, relates to the technical field of model training detection, and comprises the following steps: obtaining training task execution data of a target model and resource state data of a target operation entity where the target model is located; determining a current training trend of the target model and comparing the current training trend with an expected training trend of the target model; determining a training resource support degree of the target operation entity to the target model and comparing the training resource support degree with a resource pressure intensity of the target model; in response to the fact that there is a deviation risk in training and there is no imbalance risk in resource supply, generating a training correction parameter to correct a training process of the target model and obtaining a training correction result; and in response to the fact that the training correction result represents training correction failure, determining that the target model training is abnormal. The application can realize efficient detection of abnormalities in the model training process, and at least solves the problems of scattered detection information, lagging behind in post-analysis positioning and misjudgment of recoverable risks as abnormalities in related technologies, thereby reducing resource waste.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training detection, in particular to a model training detection method and an electronic device. BACKGROUND

[0002] With the continuous expansion of the size of deep learning models and the significant increase in the amount of training data, model training gradually presents the characteristics of high intensity and long cycle, and the continuous occupation of resources such as calculation, storage and communication in the training process significantly increases, and the overall training overhead and complexity also rise. Under this background, once an exception or failure occurs in the training process, if it cannot be identified and disposed of in time, it will often lead to a long time of invalid training, continuous occupation of computing and storage resources, forced extension of the training period, and even the need for retraining, thereby causing serious resource waste and cost loss. Therefore, higher requirements are put forward for the detection of the model training process.

[0003] However, there are still significant deficiencies in the related art for model training detection: on the one hand, the related art usually separates the construction and use of functions such as training performance analysis, training result verification and exception troubleshooting, resulting in scattered detection information and fragmented diagnosis links; on the other hand, the related art relies on training logs or offline indicators for post-analysis, and there is a delay in detection and positioning, and a large amount of irreversible resource consumption may have been generated by the time the problem is discovered; in addition, a part of the risk problems in the training process can be overcome through recovery processing, but the related art often directly determines the risk problems as exceptions and disposes of them according to static rules, which lacks online identification and early warning mechanisms, and does not attempt to correct the problems after they occur, resulting in problems that could have been overcome through timely processing being directly determined as training exceptions and forced to retrain, thereby amplifying the computing and time costs in long-cycle training, and significantly increasing the risk of resource waste. SUMMARY

[0004] The present application provides a model training detection method that can determine the training trend during the training process and perform joint detection in combination with the resource state, identify potential risks and perform recovery correction when the conditions are met, thereby achieving efficient detection of exceptions in the model training process, to at least solve the problems of scattered detection information, delayed post-analysis positioning and misjudgment of recoverable risks as exceptions in the related art, resulting in resource waste.

[0005] The present application provides a model training detection method, comprising:

[0006] In response to the target model performing a training task on a target operation entity, training task execution data of the target model and resource state data of the target operation entity are obtained;

[0007] The current training trend of the target model is determined based on the training task execution data, and the current training trend is compared with the expected training trend of the target model to obtain the training trend comparison result.

[0008] Based on the historical training records and resource status data of the target model, the support of the target computing entity for the training resources of the target model is determined, and the support of the training resources is compared with the resource pressure intensity of the target model to obtain the resource comparison results.

[0009] If the training trend comparison results indicate that there is a risk of deviation in training and the resource comparison results indicate that there is no risk of imbalance in resource supply, then training correction parameters are generated based on the resource comparison results and the training trend comparison results, so as to correct the training process of the target model based on the training correction parameters and obtain the training correction results.

[0010] The training correction results indicate training correction failure and determine that the target model training is abnormal.

[0011] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model training and detection methods.

[0012] This application achieves a multi-faceted solution by acquiring training task execution data in real-time during the target model training process and determining the current training trend online based on this data, comparing it with the expected training trend. This allows for the simultaneous and continuous characterization of training performance changes, verification of whether the training results meet expectations, and immediate identification and indication of anomalies within the same online detection chain. This avoids separating "performance analysis—result verification—anomaly investigation" into independent construction and usage processes, thus solving the technical problems of fragmented detection information and disconnected diagnostic chains in existing technologies, which hinder timely and consistent judgments. Furthermore, by comparing the training trend... Using the results as the core criterion for anomaly detection and continuously outputting consistent trend assessment conclusions, this reduces the reliance on post-hoc analysis of offline logs and one-time judgments based on static rules, improving the timeliness and accuracy of anomaly detection. This solves the technical problem of existing technologies causing ineffective training due to delayed detection. Furthermore, since training correction parameters are generated when there is a risk of deviation in the training trend, and anomaly judgment is made based on the correction results, recoverable risks can be verified and converged first, rather than being directly judged as anomalies. This solves the technical problems of existing technologies lacking recoverable processing, easily triggering unnecessary interruptions and retraining, and amplifying the cost of long-term training. Attached Figure Description

[0013] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0014] Figure 1 A model training detection method application environment schematic diagram provided for the embodiments of the present application;

[0015] Figure 2 A model training detection method schematic diagram provided for the embodiments of the present application;

[0016] Figure 3 A model training detection step schematic diagram provided for the embodiments of the present application;

[0017] Figure 4 A model training detection device architecture schematic diagram provided for the embodiments of the present application;

[0018] Figure 5 An electronic device structure block diagram provided for the embodiments of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0020] It should be noted that in the description of the present application, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, not to describe a specific order or sequence.

[0021] In order to make those skilled in the art better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0022] The model training detection method provided by the present application can be applied to, for example Figure 1The application environment shown. Among them, the terminal 101 communicates with the server 102 through the network. Specifically, the terminal 101 is used to initiate the training task of the target model, configure the training parameters and present the training monitoring and detection results; the server 102 is used to bear the calculation execution of the target model training, and collect the training task execution data and the resource state data of the target operation entity during the training task execution, on this basis, determine the current training trend of the target model online and compare with the expected training trend, at the same time, determine the training resource support degree and compare with the resource pressure intensity based on the historical training record and the current resource state data, so as to obtain the training trend comparison result and the resource comparison result; the server 102 further judges the abnormal risk of the training process according to the above comparison results, generates the training correction parameters when the correction condition is met, verifies the training correction and outputs the training correction result, and outputs the model training abnormal conclusion when the correction fails or the abnormal judgment condition is met; the terminal 101 receives the detection conclusion / correction result returned by the server 102 and is used for alarm prompt, visual display or triggering subsequent processing process. Among them, the terminal 101 can be but not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 102 can be realized by independent server or server cluster composed of multiple servers.

[0023] In one embodiment, as Figure 2 shown, the present application provides a model training detection method, comprising:

[0024] Step 201, in response to the target model executing the training task on the target operation entity, obtaining the training task execution data of the target model and the resource state data of the target operation entity;

[0025] Step 202, determining the current training trend of the target model based on the training task execution data, and comparing the current training trend with the expected training trend of the target model to obtain the training trend comparison result;

[0026] Step 203, based on the historical training record of the target model and the resource state data, determining the training resource support degree of the target operation entity to the target model, and comparing the training resource support degree with the resource pressure intensity of the target model to obtain the resource comparison result;

[0027] Step 204, in response to the training trend comparison result representing that there is a deviation risk in the training and the resource comparison result representing that there is no imbalance risk in the resource supply, generating the training correction parameters according to the resource comparison result and the training trend comparison result, correcting the training process of the target model based on the training correction parameters and obtaining the training correction result;

[0028] Step 205, in response to the training correction result representing training correction failure, determining that the target model training is abnormal.

[0029] The model training detection method provided in this embodiment can continuously depict the training performance change, verify whether the training result is as expected, and immediately identify and indicate the abnormality in the same online detection link by acquiring the training task execution data in real time during the execution of the target model training task, and determining the current training trend based on the data and comparing the training trend with the expected training trend. Thus, the performance analysis, result verification, and abnormality troubleshooting are not separated into independent construction and use processes, and the technical problem of the existing technology that the scattered detection information and the split diagnosis link make it difficult to form a consistent judgment in time is solved. Meanwhile, the consistent trend evaluation conclusion is continuously output by taking the training trend comparison result as the core basis for abnormality detection, so that the dependence on offline log post-analysis and one-time determination of static rules is reduced, and the timeliness and accuracy of abnormality detection are improved, thereby solving the technical problem of the existing technology that the discovery lag causes invalid training to continue. In addition, the training correction parameter is generated when the training trend deviates from the expected trend, and the abnormality is determined again based on the correction result, so that the recoverable risk is verified and converged before being directly determined as abnormal, thereby solving the technical problem of the existing technology that lacks recovery processing, easily triggers unnecessary interruption and retraining, and amplifies the cost of long-period training.

[0030] In one embodiment, as shown in Figure 3 The method further includes:

[0031] Step 301, in response to the resource comparison result representing that the resource supply has an imbalance risk and the training trend comparison result representing that the training has no deviation risk, generating a resource supply optimization parameter, and performing resource supply optimization on the target operation entity based on the resource supply optimization parameter to obtain a resource supply optimization result.

[0032] Step 302, in response to the resource supply optimization result representing that the resource supply optimization fails, determining that the target model training is abnormal.

[0033] The target operation entity refers to an operation resource carrier that carries the target model training task, including but not limited to a single server, one training node in a cluster, a container instance, or a resource quota object allocated by a job scheduling system, and at least including CPU / accelerator card / memory / storage / network resources.

[0034] The resource comparison result refers to a determination result obtained by matching and comparing the resource support of the target operation entity for the target model training with the pressure intensity of the target model on the resource. The resource comparison result at least represents whether the resource supply has an imbalance risk, the resource dimension corresponding to the imbalance risk, and the imbalance direction.

[0035] wherein, the training trend comparison result refers to a result of comparing the current training trend with the expected training trend, and the training trend comparison result represents whether there is a deviation risk in the training, and no deviation risk in the training means that the current training trend is within the preset deviation tolerance threshold;

[0036] Preferably, the resource supply imbalance risk at least includes any of the following situations: insufficient computing resource risk, insufficient GPU / memory risk, insufficient storage bandwidth risk, and abnormal communication bandwidth or communication latency risk; and the imbalance direction includes but is not limited to insufficient supply, excessive supply fluctuation, and supply jitter leading to decreased stability.

[0037] Specifically, in the embodiment, in the case that the training trend does not deviate and the resource comparison result represents that the resource supply has an imbalance risk, the resource supply optimization parameter is generated and the resource supply optimization is performed on the target operation entity, which can further subdivide the determination and disposal of training abnormalities into two paths of "training side risk" and "resource side risk", thereby avoiding directly interrupting the training only because of the resource supply shortage or imbalance when the training process itself is normal; and by determining the target model training abnormality again after the resource supply optimization fails, the resource side problem has the opportunity of priority optimization and verification, which solves the problem in the prior art that the resource problem and the training problem are mixedly processed and the training is easily terminated too early, and helps to improve the continuity of the training process and reduce unnecessary abnormal determination.

[0038] In one preferred embodiment, the resource supply optimization parameter is generated to perform resource supply optimization on the target operation entity based on the resource supply optimization parameter and obtain a resource supply optimization result, which includes:

[0039] The resource comparison result is analyzed to determine at least one imbalance resource dimension corresponding to the resource supply imbalance risk and the imbalance direction, and to determine the target operation entity and its current training stage which need to perform resource supply optimization;

[0040] The resource state data of the target operation entity in the current training stage is obtained, and the resource allocation strategy, quota limit and scheduling strategy of the target operation entity are combined to determine the resource supply adjustable range and the resource supply constraint condition;

[0041] Based on the imbalance resource dimension and the resource supply constraint condition, a plurality of candidate resource supply optimization parameter sets are generated, wherein the candidate resource supply optimization parameter is used to perform supply side adjustment on the corresponding resource dimension of the target operation entity to improve the available resource level of the imbalance resource dimension or to improve the supply stability thereof;

[0042] The training configuration parameters and the training task execution data are based on the target model. The improvement effect of multiple candidate resource supply optimization parameters on the unbalanced resource dimension and the side effect on other resource dimensions are evaluated respectively. Under the premise of meeting the resource supply constraint condition, the target candidate resource supply optimization parameter is selected as the resource supply optimization parameter;

[0043] Based on the selected resource supply optimization parameter, resource supply optimization is performed on the target operation entity, and the optimized resource state data is collected within the preset resource evaluation period to obtain the resource state parameter set after resource supply optimization;

[0044] Based on the optimized resource state data, the training resource support of the target operation entity to the target model is re-determined, and compared with the resource pressure intensity of the target model to generate an optimized resource comparison result;

[0045] In response to the optimized resource comparison result representing that there is no imbalance risk in resource supply, a resource supply optimization result representing that the resource supply optimization is successful is output, otherwise a resource supply optimization result representing that the resource supply optimization fails is output.

[0046] Among them, the training stage refers to the stage division mark in the training process, which is used to reflect the difference of resource demand mode in different stages. The training stage can be divided according to the training step interval, epoch interval, preheating / stable / convergent stage, or according to the training configuration change point. Preferably, the training stage can be obtained by "training step count + preset stage division rule".

[0047] Among them, the resource state data refers to the collection of resource availability and occupancy indicators within the preset sampling period. Preferably, the resource state data can be collected through operating system monitoring interface, container / virtualization monitoring interface, job scheduling system interface and acceleration card management interface, including but not limited to: available computing power, acceleration card utilization rate, remaining video memory, available memory, I / O bandwidth, network bandwidth, communication delay, queue length, packet loss rate, etc.

[0048] Among them, the resource allocation strategy, quota limit and scheduling strategy refer to the constraint set of the adjustable range of resources in the environment where the target operation entity is located, including at least: whether to allow expansion / contraction, upper limit of expansion, preemption strategy, affinity / anti-affinity constraint, migration limit, minimum available instance number, GPU binding strategy, etc.

[0049] Preferably, the resource supply optimization parameter can be determined by the following methods, including:

[0050] The improvement amount of the unbalanced dimension is calculated based on the resource state data. Exemplarily, the average / peak / variance of available resources after optimization is compared with the average / peak / variance of available resources before optimization to obtain an improvement score.

[0051] calculating a training-side influence quantity based on the training task execution data, for example, comparing changes in unit step time consumption, throughput, communication waiting proportion, loss short window fluctuation, etc.

[0052] comprehensively scoring the improvement score and the influence quantity according to a preset weight, and selecting a candidate parameter with the highest score and meeting the resource supply constraint condition as the resource supply optimization parameter.

[0053] Specifically, in the embodiment, by analyzing the resource comparison result, the specific resource dimension and its imbalance direction that exist imbalance risk in the training process are explicitly identified, and combined with the training stage of the target operation entity, the resource allocation strategy and the scheduling constraint, the adjustable range of the resource supply is limited, and on this basis, multiple groups of candidate resource supply optimization parameters are generated and evaluated, so as to realize the targeted resource optimization facing the supply side. Compared with the way of passive interruption or restart of training only after resource shortage, controllable adjustment of the imbalanced resource can be made without violating the existing quota and scheduling constraints, and by evaluating the collateral effects of the candidate optimization parameters on other resource dimensions, new resource conflicts or system instability caused by single resource optimization can be avoided. Further, by recalculating the training resource support degree after resource supply optimization and comparing it with the resource pressure intensity, closed-loop verification of the optimization effect is realized, so that the resource optimization result has a quantifiable and determinable basis, thereby improving the accuracy and effectiveness of resource supply adjustment, reducing the risk of performance degradation, training interruption or repeated training caused by resource imbalance in the training process, and being conducive to improving the resource utilization efficiency and overall stability of long-period model training.

[0054] In one embodiment, the method further comprises:

[0055] In response to the resource comparison result representing that the resource supply exists imbalance risk and the training trend comparison result representing that the training exists deviation risk, determining whether there is an association between the resource supply imbalance risk and the training deviation risk according to the resource comparison result and the training trend comparison result;

[0056] If there is an association, generating a joint correction parameter based on the resource comparison result and the training trend comparison result, performing joint training correction on the target operation entity and the target model based on the joint correction parameter to obtain a joint training optimization result, and generating a model training anomaly judgment result of the target model according to the joint training correction result;

[0057] If there is no association, determining that the target model training is abnormal.

[0058] The association refers to that the resource-side imbalance risk and the training-side deviation risk have an interpretable corresponding relationship, at least including time synchronization or causality directionality, and preferably, the association can be determined by the following methods, including:

[0059] aligning the difference parameter of the imbalance dimension in the resource comparison result with the deviation term in the training trend feature in time;

[0060] calculating a correlation index, such as a correlation coefficient, mutual information, or lag correlation, of the two within a preset window, and determining whether it exceeds a preset correlation threshold;

[0061] confirming in combination with rules, for example, when “insufficient memory / communication congestion” and “throughput decline, loss shock intensification” appear at the same time and meet the sequence, it is determined that there is an association.

[0062] wherein the joint correction parameter refers to a combined parameter set containing both resource-side optimization parameters and training-side correction parameters, the resource-side optimization parameters are used to adjust the supply capacity or stability of the target operation entity, and the training-side correction parameters are used to adjust the target model training control parameters to reduce the instantaneous pressure on the limited resources or suppress the training divergence trend; preferably, the joint correction parameter can be obtained by “joint search / optimization with the minimum training deviation as the target under the resource supply constraint condition”.

[0063] wherein the joint training correction refers to the process of simultaneously (or in a preset order) performing resource adjustment on the target operation entity and writing training correction parameters to the target model to make them effective in the same correction period; preferably, the joint training correction result at least includes: the training trend comparison result after correction, the resource comparison result after correction, the throughput / time consumption change, and whether there is still an abnormal risk.

[0064] Specifically, in the present embodiment, when the resource comparison result indicates that there is an imbalance risk in resource supply and the training trend comparison result indicates that there is a deviation risk in training, it is further determined whether there is an association between the resource supply imbalance risk and the training deviation risk, and when there is an association, joint correction parameters are generated based on the resource comparison result and the training trend comparison result to perform joint training correction on the target operation entity and the target model, so as to integrate resource-side optimization and training-side correction into the same closed-loop processing process, avoid misjudgment and invalid intervention caused by blind adjustment according to a single risk signal, improve the pertinence and stability of training correction, and generate more accurate and interpretable model training abnormality judgment results according to the joint training correction result; at the same time, when there is no association, it is directly determined that the target model training is abnormal, which can reduce repeated trial and error and invalid parameter adjustment on irrelevant factors, reduce training cost and resource waste, and improve the timeliness and overall training efficiency of abnormal identification and disposal.

[0065] In one embodiment, after determining that the target model training is abnormal, the method further comprises:

[0066] determine an abnormal type corresponding to the model training abnormality based on a determination condition of the abnormality, wherein the determination condition refers to a rule source triggering the determination of the model training abnormality, and at least includes: resource supply optimization failure, training correction failure, training trend deviation exceeding a threshold, heartbeat timeout, and keyword hit serious error;

[0067] perform a corresponding abnormality handling operation based on the determined abnormal type.

[0068] The abnormal type is used to classify and identify the abnormal reason, and at least includes: resource deficiency type, resource fluctuation / jitter type, communication bottleneck type, training divergence type, training stagnation type, process abnormal exit type, and environment / driver type. Preferably, the abnormal type can be determined through a "determination condition-abnormal type mapping table", and when multiple determination conditions are triggered at the same time, a main abnormal type can be determined according to a preset priority or voting rule.

[0069] The abnormality handling operation includes but is not limited to: terminating training and saving the scene, reducing parallelism, switching training parameter groups, triggering resource expansion / contraction, migrating training tasks, restarting the communication backend, and alarm notification. Preferably, the handling operation can be configured as a script, an interface call, or a scheduling instruction.

[0070] Specifically, in the embodiment, after the model training abnormality is determined, the abnormal type is further determined based on the determination condition of the triggered abnormality, and the corresponding abnormality handling operation is performed accordingly. The embodiment can change the abnormality handling from the coarse-grained processing mode of "unified interruption or restart" to the differentiated response based on the abnormal cause, thereby avoiding the improper handling or low efficiency caused by the same handling strategy for different types of abnormality. Therefore, the pertinence and controllability of the abnormality handling are improved, unnecessary resource release, task migration or retraining operation are reduced, and the impact of the training abnormality on the overall training process is further reduced.

[0071] In one embodiment, the current training trend of the target model is determined based on training task execution data, including:

[0072] In a preset time window, loss values and model parameter snapshots of multiple training steps are obtained by analyzing the training task execution data;

[0073] The loss values of the multiple training steps are sequentially processed based on the execution order of the multiple training steps to generate a training loss sequence;

[0074] The loss values of adjacent training steps in the training loss sequence are sequentially processed to generate a loss change sequence;

[0075] performing statistical analysis on the loss change sequence to obtain a loss trend item at least characterizing an overall rising / falling direction of the loss and a loss change rate;

[0076] determining a loss fluctuation degree based on the loss change sequence, and generating a loss stability item at least characterizing a smoothness / oscillation degree of the loss according to the loss fluctuation degree;

[0077] determining a parameter change amount based on the model parameter snapshots of the adjacent two training steps, and mapping the parameter change amount to a parameter update amount to generate a model parameter update sequence based on the execution order of the plurality of training steps;

[0078] performing statistical analysis on the model parameter update sequence to obtain an update amplitude item at least characterizing an overall intensity of the model parameter update;

[0079] generating a training trend feature of the target model within a preset time window based on the loss trend item, the loss stability item, and the update amplitude item, and determining a current training trend.

[0080] wherein the training task execution data refers to a set of observable data generated by the training framework during runtime, and at least includes: training step number, loss value, learning rate, gradient statistics, throughput / time consumption, and parameter / gradient snapshot pointer; preferably, the training task execution data can be obtained through training framework callback, log analysis, or monitoring SDK reporting;

[0081] wherein the preset time window refers to a sliding window range for trend analysis, which can be defined by the number of training steps or the length of time; preferably, the preset time window can be set to the last N steps or the last T seconds / minutes, so as to ensure that the trend feature has statistical significance and can respond in a timely manner;

[0082] wherein the model parameter snapshot refers to the model parameter state or its abstract information recorded at the boundary of the training step; preferably, in order to reduce the overhead, the model parameter snapshot can be a hash digest of the parameter vector, a parameter norm, a key layer parameter subset, or a gradient norm.

[0083] Specifically, the loss trend item can be obtained by: determining the overall direction by averaging or mediating the loss change sequence, and calculating the average change amount per step to represent the change rate; the loss stability item can be obtained by the variance / standard deviation or peak-valley amplitude of the loss change sequence; and the update amplitude item can be obtained by the norm (e.g., L2 norm) of the difference between adjacent parameter snapshots or gradient norm statistics, to represent the parameter update intensity.

[0084] Specifically, in the present embodiment, by serializing, differentiating and statistically analyzing the training step loss value and model parameter snapshot within a preset time window, and jointly constructing the training trend features from multiple dimensions such as loss change trend, loss stability and parameter update intensity, the originally scattered and instantaneous training indicators can be converted into training trend representations with time continuity and statistical significance, thereby avoiding the contingency and instability caused by judging only according to single point loss value or single parameter change, improving the robustness and credibility of the current training trend description, and providing a stable and consistent basis for subsequent training trend comparison.

[0085] In one embodiment, the current training trend is compared with the expected training trend of the target model to obtain a training trend comparison result, including:

[0086] The expected training trend of the target model at the corresponding training stage in the preset time window is obtained, and the expected training trend is determined by one or more of the following: a preset training strategy, historical benchmark training data, and a preconfigured stage target.

[0087] The expected training trend is converted into a trend representation item consistent with the current training trend, and the trend representation item at least includes an expected loss direction, an expected change rate range, an expected fluctuation degree range, and an expected parameter update intensity range.

[0088] The loss trend item in the current training trend is compared with the expected loss direction and the expected change rate range to obtain a loss trend comparison item.

[0089] The loss stability item in the current training trend is compared with the expected fluctuation degree range to obtain a stability comparison item.

[0090] The update amplitude item in the current training trend is compared with the expected parameter update intensity range to obtain an update intensity comparison item.

[0091] In response to one or more of the loss trend comparison item, the stability comparison item, and the update intensity comparison item exceeding the corresponding deviation tolerance threshold, a training trend comparison result is generated, which represents that the training of the target model has a deviation risk.

[0092] Wherein, the expected training trend refers to the target interval that the loss change direction / rate / fluctuation and the parameter update intensity should fall into in a specific training stage under normal training conditions; preferably, the expected training trend can be obtained by at least one of the following ways: first, according to the preset training strategy, for example, the preheating stage allows larger fluctuation, and the stable stage expects the descending rate to be within a certain interval; second, according to the historical benchmark training data, for example, the historical normal operation statistical interval of the same data set, the same model structure and the same batch size; third, according to the preconfigured stage target, for example, the loss should be reduced to a certain range within a certain stage.

[0093] The deviation tolerance threshold refers to the allowed normal fluctuation boundary, and different thresholds can be set according to dimensions. Preferably, the deviation tolerance threshold can be set according to historical benchmark quantiles (for example, P90 / P95) or manual experience.

[0094] Specifically, the comparison adopts the "whether falling into the range + out-of-range amplitude" mode. For example, when the current value is not within the expected range and the out-of-range amplitude exceeds the threshold, it is determined that there is a deviation item in the dimension. When multiple deviation items appear simultaneously or persist for more than a preset number of steps, it is output that the training has a deviation risk.

[0095] Specifically, in this embodiment, by comparing the current training trend with the expected training trend of the corresponding training stage in multiple dimensions and quantifiable trend characterization, and identifying the deviation risk in terms of loss change direction, change rate, fluctuation degree and parameter update intensity, etc., the abnormal development trend in the training process can be identified in advance before the training result has been significantly deteriorated. Therefore, the training detection is changed from the post-judgment relying on the training end result or serious abnormal log to the online risk discovery and positioning based on the trend evolution, which provides sufficient response window for subsequent training correction and avoids the technical problems in the prior art that the training state is irreversible and the resource consumption is difficult to recover due to the discovery lag.

[0096] In one embodiment, based on the historical training records of the target model and the resource state data, the training resource support degree of the target operation entity to the target model is determined, including:

[0097] The historical training records of the target model are obtained and parsed to obtain resource monitoring records corresponding to multiple training steps in the historical training process of the target model. The historical training records refer to the training and resource monitoring data collected during the historical training of the target model, which can be obtained from a monitoring system, training logs, or a training platform database. Preferably, the historical training records can be limited to valid samples with the same model structure, the same data size, and the same parallel strategy to improve comparability.

[0098] Based on the preset resource dimensions, historical resource consumption data is extracted from the resource monitoring records. The resource dimensions include at least one or more of the following: computing resources, storage resources, and communication resources.

[0099] The historical resource consumption data is sequentially processed according to the training step order to generate a historical resource consumption sequence corresponding to the resource dimensions.

[0100] Based on the historical resource consumption sequence, statistical analysis is performed to generate a historical resource consumption feature set. The historical resource consumption feature set represents at least one or more of the following: the base consumption level, the peak consumption level, and the fluctuation of the corresponding resource dimensions.

[0101] The resource state data corresponding to the target operation entity in the current training stage is acquired, and the resource state data at least includes resource dimensions consistent with the historical resource consumption data;

[0102] The resource state data is parsed to determine the available resource state information of the target operation entity in the current training stage and generate an available resource state parameter set according to the resource dimensions. The available resource state parameter set at least includes one or more of available computing power, available storage capacity and available communication bandwidth.

[0103] According to the plurality of resource dimensions, the available resource state parameter set and the historical resource consumption feature set are respectively matched and evaluated to obtain a support evaluation result corresponding to the resource dimension. The support evaluation result at least represents the satisfaction degree and the redundancy degree of the current available resource to the historical ground state consumption and the peak consumption.

[0104] The support evaluation results of the plurality of resource dimensions are fused to generate a training resource support degree. The training resource support degree at least represents the overall support ability of the target operation entity to the target model training in the current training stage and the potential bottleneck resource dimension. Preferably, the fusion can adopt a weighted summation or a conservative fusion mode of taking the minimum value, so that the training resource support degree can reflect the shortest board resource dimension.

[0105] The ground state consumption level refers to the typical resource consumption level when the training is stably running, and is preferably represented by the mean / median of the historical resource consumption sequence. The peak consumption level refers to the resource consumption peak value occurring in the training process, and is preferably represented by the maximum value or high quantile (such as P95). The fluctuation condition refers to the stability degree of resource consumption, and is preferably represented by the standard deviation, coefficient of variation or peak-valley difference.

[0106] Specifically, the support evaluation result can be obtained by the following method:

[0107] For each resource dimension, the ratio of available resource to peak consumption and the ratio of available resource to ground state consumption are calculated, and a redundancy coefficient is set in combination with the fluctuation condition.

[0108] When the ratio is lower than the preset support threshold, it is determined that the dimension is a potential bottleneck.

[0109] Specifically, in the embodiment, by combining the historical training record of the target model with the current resource state data, the historical consumption feature and the current available resource are matched and evaluated from multiple resource dimensions, which can improve the judgment of resource support ability from static resource specification or instantaneous occupation to comprehensive evaluation based on the relationship between "historical training actual consumption-current resource supply ability". Therefore, not only whether the current resource meets the training demand can be described, but also the potential bottleneck resource dimension can be revealed, solving the problem that the prior art only relies on a single resource index and is difficult to fully reflect the training support ability.

[0110] In one embodiment, the training resource support degree is compared with the resource pressure intensity of the target model to obtain a resource comparison result, including:

[0111] Obtain training task execution data and training configuration parameters of the target model in the current training stage, wherein the training configuration parameters refer to a set of training parameters that significantly affect the resource usage form, including but not limited to batch size, sequence length, parallel strategy, gradient accumulation, mixed precision switch, communication backend, and parameter synchronization frequency.

[0112] Based on the training task execution data, actual resource consumption characteristics corresponding to multiple training steps are parsed, and resource usage pressure characteristics exerted by the target model on multiple resource dimensions in the current training stage are determined according to the actual resource consumption characteristics and the training configuration parameters, wherein the resource usage pressure characteristics refer to pressure representations of resource consumption evolution with training steps, at least including mean consumption, peak consumption, growth trend, fluctuation amplitude, and waiting proportion (such as communication waiting / data loading waiting).

[0113] According to a preset resource pressure modeling rule, the resource usage pressure characteristics are quantitatively modeled to generate resource pressure intensity parameters of corresponding resource dimensions, wherein the resource pressure modeling rule refers to a rule of mapping the resource usage pressure characteristics into comparable intensity scalars or vectors, preferably using linear weighting, normalization ratio, or piecewise function to map "peak / mean / growth rate / fluctuation" into pressure intensity.

[0114] The resource pressure intensity parameters of multiple resource dimensions are fused to generate the resource pressure intensity of the target model in the current training stage, which at least represents the overall resource pressure level and the pressure concentrated resource dimension exerted by the target model on the target operation entity during training.

[0115] The resource pressure intensity is matched and compared with the training resource support degree to obtain a difference parameter, which at least represents the deviation degree and direction of resource pressure relative to resource supply capacity.

[0116] In response to the resource pressure intensity being less than the training resource support degree and the difference parameter being less than or equal to a preset resource balance threshold, a resource comparison result representing that there is no imbalance risk in resource supply is generated, wherein the resource balance threshold is used to limit the normal acceptable supply-demand difference range, and preferably, the resource balance threshold can be configured according to historical normal training statistical interval or platform experience.

[0117] In response to the resource pressure intensity being greater than the training resource support degree or the difference parameter being greater than the resource balance threshold, a resource comparison result representing that there is an imbalance risk in resource supply is generated.

[0118] Specifically, the resource pressure intensity parameter can be obtained by combining the peak consumption and the ground state consumption by weight for any resource dimension, and introducing a growth rate penalty term and a fluctuation penalty term to obtain the pressure intensity of the dimension. When multi-dimensional fusion is used, weighted summation or maximum value can be used to reflect the central pressure dimension.

[0119] Specifically, in the embodiment, the resource pressure intensity is quantified by modeling the actual resource consumption behavior of the target model in the current training stage, and is matched and compared with the training resource support degree. The embodiment can describe the comprehensive pressure state of the model training on computing, storage and communication resources from the training load perspective, thereby avoiding passive judgment of resource problems by relying only on resource alarms or abnormal logs. At the same time, since the resource pressure intensity reflects the evolution trend of resource pressure rather than resource failure results, it can identify the resource supply imbalance risk before the resource overflow, congestion or explicit abnormality occurs, so as to change the resource side detection from post-alarm to forward identification and positioning, provide a disposal window for subsequent resource optimization or training correction, and reduce the risk of training interruption, retraining or long-term invalid occupation caused by the problem being discovered too late.

[0120] In one embodiment, the training correction parameter is generated according to the resource comparison result and the training trend comparison result, including:

[0121] The training trend comparison result is analyzed to determine the training trend deviation item of the target model in the current training process. The deviation trend item includes one or more of the loss change direction deviation item, the loss change rate deviation item, the loss fluctuation degree deviation item, and the parameter update intensity deviation item.

[0122] Based on the expected training trend of the target model in the current training process, the target adjustment direction and the allowed adjustment interval corresponding to the plurality of training trend deviation items are determined, and a training trend correction target set is generated.

[0123] Based on the training trend correction target set, a plurality of candidate training control parameters are determined. The candidate training control parameter set includes one or more of the learning rate parameter, the training batch parameter, the gradient accumulation parameter, the parameter synchronization frequency parameter, and the regularization related parameter. The training trend correction target set refers to the expected regression direction and amplitude constraint of the deviation item, such as "reducing loss fluctuation", "reducing update amplitude", "slowing down the descent rate to gain stability".

[0124] The resource comparison result is analyzed to determine the resource supply constraint condition corresponding to the target model in the current training stage, and the resource supply constraint condition at least includes a limited resource dimension identifier, an acceptable resource pressure increment threshold, and a resource redundancy interval parameter. Specifically, the resource supply constraint condition refers to the acceptable incremental pressure boundary on the resource side, and the limited resource dimension identifier is used to indicate the current most stressed resource dimension (such as video memory or communication bandwidth);

[0125] The resource pressure increment when the plurality of training control parameters are applied is predicted, and among the plurality of candidate training control parameters, the available training control parameter whose resource pressure increment satisfies the resource supply constraint condition is selected.

[0126] Based on the correlation between the plurality of available training control parameters, the plurality of available training control parameters are associated and optimized to generate a training correction parameter.

[0127] Specifically, the resource pressure increment is obtained by the following method:

[0128] According to the training configuration parameters and the experience mapping table, the influence direction of the control parameter change on the resource is estimated, for example, increasing the batch will increase the video memory occupation and throughput, increasing the gradient accumulation will increase the time but can reduce the instantaneous video memory; increasing the synchronization frequency will increase the communication pressure;

[0129] Short-period trial operation is performed within a preset small-step trial window, and the resource pressure intensity change of the trial operation is collected as a prediction calibration value;

[0130] The prediction result is compared with the acceptable resource pressure increment threshold to screen the available control parameter.

[0131] Preferably, the correlation optimization processing can adopt multi-objective optimization, for example:

[0132] Taking "maximum regression degree of deviation term" as the first target, taking "minimum resource pressure increment / satisfying constraint" as the second target, and taking "minimum training throughput reduction" as the third target, a training correction parameter is obtained.

[0133] Specifically, in this embodiment, by first determining the training trend item that needs to be corrected based on the training trend comparison result, and then combining the resource comparison result to constrain, screen and optimize the candidate training control parameter, the effectiveness of the correction can be ensured while avoiding the introduction of new resource imbalance risks by training parameter adjustment; thereby forming a parameter generation mechanism that takes training trend correction as the goal and takes resource supply capability as the boundary condition, avoiding the problem that training parameter adjustment is disconnected from resource carrying capacity and easily causes secondary abnormalities, and improving the executability and safety of training correction.

[0134] In one embodiment, the training process of the correction target model is corrected and a training correction result is obtained, including:

[0135] In the training task execution process of the target model, the parameter effective time at which the training parameter is allowed to change is determined, wherein the parameter effective time refers to the time point at which the training process allows safe modification of the training control parameter, and preferably, the parameter effective time can be a step boundary, a gradient accumulation boundary, a checkpoint saving point, or a learning rate scheduling point, so as to avoid inconsistent changes in the parameters during a reverse propagation;

[0136] At the parameter effective time, the training correction parameter is written into the training configuration parameter corresponding to the target model, so that the training correction parameter takes effect in the training process;

[0137] Based on the training configuration parameter after the training correction parameter is applied, the training task of at least one training correction evaluation period of the target model is driven, and the updated training task execution data and the resource state data of the target operation entity are collected within the training correction evaluation period, wherein the training correction evaluation period refers to the minimum running interval for verifying the correction effect, and is preferably continuous M training steps or a fixed time T, so as to ensure the statistical reliability of the trend and resource evaluation;

[0138] Based on the updated training task execution data, the current training trend of the target model within the training correction evaluation period is re-determined, and a corrected training trend comparison result is generated;

[0139] Based on the updated resource state data, the training resource support degree of the target operation entity to the target model is re-determined, and is compared with the resource pressure intensity of the target model, and a corrected resource comparison result is generated;

[0140] In response to the fact that the corrected training trend comparison result represents that there is no deviation risk in the training trend, and the corrected resource comparison result represents that there is no imbalance risk in the resource supply, a training correction result representing that the training correction is successful is output;

[0141] In response to the fact that the corrected training trend comparison result still represents that there is a deviation risk, or the corrected resource comparison result represents that there is an imbalance risk in the resource supply, a training correction result representing that the training correction fails is output.

[0142] Specifically, the determination of the success / failure of the training correction can adopt the joint condition of "simultaneously satisfying the training side and the resource side"; when either side still exceeds the threshold, the failure is output, so as to enter the further correction or abnormal handling process subsequently.

[0143] Specifically, in the present embodiment, by applying the training correction parameters at the effective moment of allowing the change of the parameters during the training process, and performing the training correction evaluation period after correction to reevaluate the training trend and the resource state, a closed-loop verification mechanism of "correction-evaluation-rejudgment" can be formed; thereby avoiding the assumption that the correction is successful by default only by parameter issuance, making the training correction effect be objectively verified, so as to provide reliable basis for whether to finally judge the training exception, improve the rationality of the exception judgment and reduce unnecessary interruption or retraining.

[0144] In one embodiment, the training task of the target model is a distributed training task, the target operation entity includes a plurality of operation nodes participating in the distributed training task, and the training resource support degree of the target operation entity to the target model is determined based on the historical training record of the target model and the resource state data, including:

[0145] Based on the resource state data corresponding to the plurality of operation nodes, the node resource support degree corresponding to each node is determined, and the comprehensive resource support degree is generated based on the plurality of node resource support degrees to represent the overall supply capability of the distributed training resources of the target operation entity;

[0146] According to the comprehensive resource support degree and the comprehensive resource seeking degree of the target model on the plurality of operation nodes, a comprehensive resource comparison result is determined, wherein the comprehensive resource seeking degree refers to a comprehensive demand representation of the target model for global resources in distributed training, which can be obtained by fusing the resource pressure intensity of each node;

[0147] The current training trend of the target model is determined, including:

[0148] Based on the training task execution data of the plurality of operation nodes in the distributed training task, the training trend consistency of the target model between the plurality of operation nodes is determined, wherein the training trend consistency refers to the consistency degree of the training trend characteristics of different nodes in direction, rate and fluctuation, preferably, the training trend consistency can be obtained by comparing the difference degree of the loss trend item / stable item / update amplitude item of each node, for example, calculating the mean and maximum of the difference value between nodes, and comparing with the consistency threshold;

[0149] Based on the training trend consistency and the expected training trend, a training comprehensive trend comparison result is determined;

[0150] In response to the training comprehensive trend comparison result representing that there is a deviation risk in the training and the comprehensive resource comparison result representing that the resource supply is normal, the method further includes:

[0151] Based on the training trend comparison result and the comprehensive resource support degree, a collaborative training correction parameter is generated, and the collaborative training correction parameter is applied to the plurality of operation nodes at a parameter synchronization effective point of the set distributed training task, wherein the parameter synchronization effective point refers to a synchronization time of parameter / parameter consistency effectiveness in the distributed training, and is preferably a global synchronization barrier point or before the next round of parameter broadcast, so as to ensure that the collaborative training correction parameter is applied by each node at the same time.

[0152] The operation node refers to a participating node instance in the distributed training, including a parameter server / work node or each node in multi-machine multi-card training, and each node can collect local resource state data and training execution data.

[0153] The node resource support degree is consistent with the training resource support degree, but is limited in the single node dimension; the comprehensive resource support degree is used to represent the global supply capacity, and preferably, the comprehensive resource support degree can be obtained by: weighting and fusing or taking the minimum value (reflecting the shortest board node) of the node support degrees, and recording the corresponding bottleneck node identifier.

[0154] Specifically, in the embodiment, the training resource support degree and the training trend analysis are expanded from a single operation entity to a comprehensive evaluation for multiple operation nodes, so that the model training detection method can adapt to the distributed training scene, uniformly depict and compare the overall resource supply capacity and the cross-node training trend consistency in the distributed training task, thereby avoiding the problem that the prior art only faces the single node or local information and is difficult to accurately reflect the global state of the distributed training. At the same time, in the case where the training deviates from the risk and the overall resource supply is normal, the collaborative training correction parameter is uniformly applied at the parameter synchronization effective point of the distributed training task, so that the plurality of operation nodes complete the synchronous correction under the consistent training rhythm, which is conducive to suppressing the amplification of the training deviation between the nodes. Therefore, the embodiment not only realizes the early identification and positioning of the training anomaly, provides an effective opportunity for subsequent correction, and reduces the waste of resources caused by post-discovery, but also naturally extends the training detection and correction capability to the distributed training scene, and improves the applicability and overall training stability of the scheme in large-scale model training.

[0155] In one embodiment, the model training detection method provided by the present application further comprises:

[0156] In the model training process, log information output by a training program is acquired, and the log information includes but is not limited to training framework logs, standard output logs, and developer-defined logs.

[0157] The log information is searched and matched based on a preset keyword rule, wherein the keyword rule is used to associate at least one keyword with a corresponding state type and a set of disposal actions;

[0158] When the target keyword is matched, it is determined that the training task is in a target state or an abnormal type corresponding to the keyword;

[0159] According to the target state or the abnormal type, a set of handling actions bound to the keyword is triggered, and the set of handling actions includes but is not limited to: collecting and saving kernel information, collecting and saving accelerator card state information, collecting and saving cache pool / video memory allocation information, terminating training, switching training configuration parameters, and sending notification information.

[0160] The keyword rule refers to a mapping relationship of "keyword / regular pattern-state type-handling action set"; preferably, the keyword rule can be loaded through a configuration file, an environment variable or a command line parameter, and supports runtime dynamic updating.

[0161] The kernel information refers to an operating system kernel log and an exception record; preferably, it can be obtained by calling a kernel log query command or a system log interface.

[0162] The accelerator card state information refers to utilization, video memory occupation, temperature, error code and process list of the accelerator card device; preferably, it can be obtained through an accelerator card management interface or a device query command.

[0163] The cache pool / video memory allocation information refers to video memory allocation and cache reservation of a training framework or a deep learning runtime; preferably, it can be obtained through a memory statistics interface provided by the training framework, a runtime monitoring API or a custom hook, and is used to represent "allocated but not released cache", "fragmentation", "available continuous video memory size" and the like.

[0164] In one example, when a video memory shortage related keyword appears in the log, accelerator card state information and cache pool / video memory allocation information are triggered to be collected, and the collection results are saved as a diagnosis file, so as to distinguish between resource occupation by other processes and allocation failure caused by too large video memory occupation or fragmentation of the training task itself.

[0165] In some embodiments, the keyword rule and the set of handling actions support configuration through a configuration file, an environment variable or a command line parameter, so as to realize flexible expansion and reuse.

[0166] In one embodiment, the model training detection method provided by the application further includes:

[0167] During the model training process, the survival state information of the training program is periodically obtained, which can be obtained by periodically reporting heartbeat information by the training process, or by periodically querying whether the training process exists by the detection end;

[0168] When it is detected that the target training process does not exist or the heartbeat timeout, it is determined that a process abnormal exit event occurs;

[0169] After it is determined that the process abnormal exit event occurs, log information is read or parsed to determine whether a normal end flag exists; if the normal end flag exists, it is determined that the training task is normally ended and no abnormal handling is triggered; if the normal end flag does not exist, it is determined that the training task abnormally exits.

[0170] When it is determined that the training task abnormally exits, a preset handling action is performed, and the preset handling action includes but is not limited to: terminating other processes related to the training task, collecting and saving exit scene information, recording an exit time point and generating an exit report file, and the scene information at least includes kernel information, accelerator card state information and system resource occupation information;

[0171] In some embodiments, the heartbeat period, the timeout threshold, the normal end flag and the abnormal handling action are all supported to be set through configuration information to adapt to a multi-machine multi-card, multi-process training scene.

[0172] The heartbeat information refers to survival proof data output by the training process at a period, which can be a timestamp, a step count, a process ID, a node ID and the like; preferably, the heartbeat information can be submitted through a local file, a shared KV, a message queue or an HTTP reporting mode.

[0173] The heartbeat timeout refers to that no heartbeat update of the target process is received within a plurality of sampling periods; preferably, the timeout threshold can be configured as K periods or a fixed time length.

[0174] The exit scene information preferably further includes system process occupation information such as top / ps, disk / network state summary and last several lines of key log fragments to improve the positioning property.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0176] As shown in FIG. 1, Figure 4 The embodiments of the present application also provide a model training detection device, the device includes an application layer, a service layer, an interface layer and an execution layer, wherein,

[0177] The application layer includes a training task management and detection interaction component for developing a training program or a training task management program for a developer, wherein the training task management and detection interaction component is configured to call a service interface provided by the service layer to enable or disable corresponding detection / optimization functions during the training process; meanwhile, the application layer supports setting device parameters through configuration information, which can be in the form of a configuration file, an environment variable, or a command line parameter, to achieve switch control, threshold setting, sampling period setting, and strategy selection of each service, thereby achieving flexible configuration and calling without modifying the main logic of the training code;

[0178] The service layer includes data collection and state monitoring services, training trend analysis and resource evaluation services, resource supply optimization and correction services, and abnormality detection and disposal services, which are configured to provide service capabilities for model training detection and intervention to the application layer, wherein the application layer uses different service functions by calling service interfaces exposed by the service layer; further, to improve universality and scalability, the services provided by the service layer can be divided into basic services, high-level services, and extended services according to functional attributes. Specifically, the basic services include data collection and state monitoring services, which are configured to collect training task execution data and resource state data, and to standardize, time-align, and encapsulate the data to form a unified input. The basic services have less dependence on the training environment and serve as essential capabilities of the device. The high-level services include training trend analysis and resource evaluation services, resource supply optimization and correction services, and abnormality detection and disposal services. The training trend analysis and resource evaluation services are configured to build training trend features and evaluate resource support and pressure intensity. The resource supply optimization and correction services are configured to generate resource supply optimization parameters and / or training correction parameters based on the evaluation results, and to review the optimization / correction effect. The abnormality detection and disposal services are configured to make abnormality judgments and generate disposal strategies or alarm information based on the trend analysis results, resource evaluation results, and review results. The above-mentioned high-level services can depend on certain environment capabilities or external interface support, but are encapsulated by the interface layer to reduce the use cost of the application layer. The extended services are configured to support developers to access customized strategies or functions in a preset format to achieve technology reuse and scenario expansion. The extended services can be accessed through the strategy registration mechanism of the interface layer and are implemented by the strategy extension execution module of the execution layer. In addition, the service layer can support one-key detection capability for the currently enabled service modules, i.e., each service provides a function detection entrance or traverses the readiness state of each service through a unified detection entrance. The service layer can also support module function introduction capability and configuration capability, enabling developers to obtain service function descriptions, usage methods, and configuration item meanings, and adjust parameters through configuration.

[0179] The interface layer includes a detection service uniform interface and a policy registration framework, is used for decoupling the service layer and the execution layer, and realizes service capability access and expansion in a plug-in / registration manner. Specifically, on one hand, the interface layer provides a uniform data reporting interface and a control issuing interface, which are used for receiving training data and resource data reported by the execution layer, and issuing optimization parameters, correction parameters and abnormality handling instructions to the execution layer; on the other hand, the interface layer provides a policy registration capability, which is used for binding and mapping a service name / policy name and an implementation entry, so as to support calling a corresponding implementation according to a service name. In some embodiments, the interface layer can also support dynamic library loading to realize functional expansion. When a corresponding dynamic library does not exist, it can be determined that the function is not supported, so as to avoid that a single function is missing and the whole system is unavailable. The interface layer can provide external users with basic interfaces and special interfaces. The basic interfaces at least include a function support interface, a state detection interface and a function introduction interface. The special interfaces are used for carrying specific business capabilities of each service. The definition and use method of the special interfaces can be described through the function introduction interface.

[0180] The execution layer includes a data collection agent module, a resource supply optimization execution module, a training correction execution module, an abnormality handling execution module and a policy expansion execution module, which are used for realizing specific function logic of each service module and responding to uniform calling and policy mapping of the interface layer. Specifically, the execution layer can support multiple language implementation forms, including but not limited to Python scripts, Shell scripts or C / C++ codes, and can realize cross-language calling through pybind, ctypes and the like, so as to integrate capabilities in different implementation forms into the same device. The data collection agent module is used for collecting training task execution data and resource state data and reporting. The resource supply optimization execution module is used for performing operations such as expansion and contraction, quota adjustment, migration or scheduling on a target operation entity according to issued resource supply optimization parameters and returning results. The training correction execution module is used for performing adjustment on adjustable parameters in a training process according to issued training correction parameters and returning correction results. The abnormality handling execution module is used for performing abnormality handling actions and returning handling results. The policy expansion execution module is used for carrying expansion policy logic accessed through the policy registration framework and outputting corresponding execution results. Further, the execution layer can also perceive a current environment according to configuration information such as GPU types and system types, select corresponding interface implementations for different environments, so as to improve universality and compatibility under multiple environments and multiple hardware conditions.

[0181] As shown in Figure 5 embodiments of the model training detection method. The processor is configured to run the computer program to execute the steps in the above-described any one model training detection method embodiment.

[0182] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the claimed application. Accordingly, modifications and / or additions, other than those explicitly described herein, can be obvious to those skilled in the art in the light of this disclosure. The claimed application is intended to embrace all such modifications and / or additions.

[0183] The model training detection method and the electronic device provided by the application are described in detail above. The principles and implementation manners of the application are described by applying specific examples in this paper. The above description of the embodiments is only applicable to help understand the method of the application and its core idea. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A model training detection method, characterized in that, include: In response to the target model executing a training task on the target computing entity, the training task execution data of the target model and the resource status data of the target computing entity are obtained. The current training trend of the target model is determined based on the training task execution data, and the current training trend is compared with the expected training trend of the target model to obtain the training trend comparison result. Based on the historical training records of the target model and the resource status data, the support degree of the target computing entity for the training resources of the target model is determined, and the support degree of the training resources is compared with the resource pressure intensity of the target model to obtain the resource comparison result. In response to the training trend comparison result indicating a risk of training deviation and the resource comparison result indicating no risk of resource supply imbalance, training correction parameters are generated based on the resource comparison result and the training trend comparison result to correct the training process of the target model and obtain the training correction result. In response to the training correction result indicating training correction failure, the target model is determined to be abnormally trained.

2. The model training detection method of claim 1, wherein, The method further includes: In response to the resource comparison result indicating a risk of resource supply imbalance and the training trend comparison result indicating no risk of training deviation, resource supply optimization parameters are generated to perform resource supply optimization on the target computing entity based on the resource supply optimization parameters and obtain resource supply optimization results. If the resource supply optimization result indicates that the resource supply optimization has failed, then the target model training is determined to be abnormal.

3. The model training detection method of claim 1 or 2, wherein, After determining that the target model training is abnormal, the method further includes: Based on the anomaly determination conditions that trigger the model training anomaly, the anomaly type corresponding to the model training anomaly is determined; Based on the determined exception type, perform the corresponding exception handling operation.

4. The model training detection method of claim 1, wherein, Determining the current training trend of the target model based on the training task execution data includes: Within a preset time window, the training task execution data is parsed to obtain loss values ​​and model parameter snapshots for multiple training steps; Based on the execution order of the multiple training steps, the loss values ​​of the multiple training steps are serialized to generate a training loss sequence. The loss values ​​of adjacent training steps in the training loss sequence are sequentially differentially processed to generate a loss change sequence; Perform statistical analysis on the loss change sequence to obtain a loss trend term that at least characterizes the overall upward / downward direction of loss and the rate of loss change; The degree of loss fluctuation is determined based on the loss change sequence, and a loss stability term, representing at least the degree of loss stability / oscillation, is generated based on the degree of loss fluctuation. The parameter change is determined based on the model parameter snapshots of two adjacent training steps, and the parameter change is mapped to the parameter update amount. A model parameter update sequence is generated based on the execution order of multiple training steps. Perform statistical analysis on the model parameter update sequence to obtain at least an update magnitude term that characterizes the intensity of the model parameter update. generate a training trend feature of the target model within the preset time window based on the loss trend item, the loss stability item, and the update amplitude item, and determine the current training trend as the current training trend.

5. The model training detection method of claim 4, wherein, The comparison of the current training trend with the expected training trend of the target model obtains a training trend comparison result, which includes: obtaining an expected training trend of the target model in a training stage corresponding to the preset time window, the expected training trend being determined by one or more of the following: a preset training strategy, historical benchmark training data, and a preconfigured stage target; convert the expected training trend into a trend representation item consistent with the current training trend, the trend representation item including at least an expected loss direction, an expected change rate range, an expected fluctuation range, and an expected parameter update intensity range; comparing the loss trend item in the current training trend with the expected loss direction and the expected change rate range to obtain a loss trend comparison item; comparing the loss stability item in the current training trend with the expected fluctuation range to obtain a stability comparison item; comparing the update amplitude item in the current training trend with the expected parameter update intensity range to obtain an update intensity comparison item; in response to one or more of the loss trend comparison item, the stability comparison item, and the update intensity comparison item exceeding a corresponding deviation tolerance threshold, a training trend comparison result is generated to represent a risk of deviation in the training of the target model.

6. The model training detection method of claim 1, wherein, The determination of the training resource support degree of the target operation entity for the target model based on the historical training record of the target model and the resource state data includes: obtaining and analyzing the historical training record of the target model to obtain resource monitoring records corresponding to multiple training steps in the historical training process of the target model; based on a preset resource dimension, extract historical resource consumption data from the resource monitoring records, the resource dimension including at least one or more of the following: computing resources, storage resources, communication resources; serializing the historical resource consumption data according to the training step order to generate a historical resource consumption sequence corresponding to the resource dimension; based on the historical resource consumption sequence, perform statistical analysis to generate a historical resource consumption feature set, the historical resource consumption feature set representing at least one or more of the following: ground state consumption level, peak consumption level, and fluctuation of the corresponding resource dimension; obtaining resource state data corresponding to the current training stage of the target operation entity, the resource state data including at least the resource dimension consistent with the historical resource consumption data; analyze the resource state data to determine the available resource state information of the target operation entity in the current training stage and generate an available resource state parameter set according to the resource dimension, the available resource state parameter set including at least one or more of the following: available computing power, available storage capacity, and available communication bandwidth; According to a plurality of resource dimensions, the available resource state parameter set and the historical resource consumption feature set are respectively matched and evaluated to obtain a support evaluation result corresponding to the resource dimension, and the support evaluation result at least represents a satisfaction degree and a redundancy degree of the current available resource to the historical ground state consumption and peak consumption; The support evaluation results of a plurality of resource dimensions are fused to generate the training resource support degree, and the training resource support degree at least represents the overall support ability and potential bottleneck resource dimension of the target model training in the current training stage.

7. The model training detection method of claim 6, wherein, The training resource support degree and the resource pressure intensity of the target model are compared to obtain a resource comparison result, which includes: Obtaining training task execution data and training configuration parameters of the target model in the current training stage; Based on the training task execution data, actual resource consumption features corresponding to a plurality of training steps are parsed, and resource use pressure features applied by the target model to a plurality of resource dimensions in the current training stage are determined according to the actual resource consumption features and the training configuration parameters; According to a preset resource pressure modeling rule, the resource use pressure features are quantitatively modeled to generate resource pressure intensity parameters corresponding to the resource dimensions; The resource pressure intensity parameters of a plurality of resource dimensions are fused to generate the resource pressure intensity of the target model in the current training stage, and the resource pressure intensity at least represents the overall resource pressure level and the pressure concentrated resource dimension applied by the target model to the target operation entity in the training process; The resource pressure intensity and the training resource support degree are matched and compared to obtain a difference parameter, and the difference parameter at least represents a deviation degree and a deviation direction of the resource pressure relative to the resource supply ability; In response to the resource pressure intensity being less than the training resource support degree, and the difference parameter being less than or equal to a preset resource balance threshold, a resource comparison result representing that there is no imbalance risk in resource supply is generated; In response to the resource pressure intensity being greater than the training resource support degree, or the difference parameter being greater than the resource balance threshold, a resource comparison result representing that there is an imbalance risk in resource supply is generated. 8.The model training detection method of claim 1, wherein, The training correction parameter is generated according to the resource comparison result and the training trend comparison result, which includes: The training trend comparison result is parsed to determine a training trend deviation item of the target model in the current training process; Based on the expected training trend of the target model in the current training process, a target adjustment direction and an allowed adjustment interval corresponding to a plurality of training trend deviation items are determined, and a training trend deviation correction target set is generated; Based on the training trend deviation correction target set, a plurality of candidate training control parameters are determined; The resource comparison result is parsed to determine a resource supply constraint condition corresponding to the target model in the current training stage; The resource pressure increment when a plurality of training control parameters are applied is predicted, and the available training control parameters in a plurality of candidate training control parameters are selected, which satisfy the resource supply constraint condition. Based on the correlation between the plurality of available training control parameters, the plurality of available training control parameters are associated and optimized to generate the training correction parameter. 9.The model training detection method of claim 1, wherein, The training process of the target model is corrected and a training correction result is obtained, including: During the execution of the training task of the target model, a parameter effective time when the training parameter is allowed to change is determined; At the parameter effective time, the training correction parameter is written into the training configuration parameter corresponding to the target model, so that the training correction parameter takes effect in the training process; Based on the training configuration parameter after applying the training correction parameter, the training task of at least one training correction evaluation period of the target model is driven, and the updated training task execution data and the resource state data of the target operation entity are collected within the training correction evaluation period; Based on the updated training task execution data, the current training trend of the target model within the training correction evaluation period is re-determined, and a corrected training trend comparison result is generated; Based on the updated resource state data, the training resource support degree of the target operation entity to the target model is re-determined, and compared with the resource pressure intensity of the target model to generate a corrected resource comparison result; In response to the corrected training trend comparison result representing that there is no deviation risk in the training trend, and the corrected resource comparison result representing that there is no imbalance risk in the resource supply, a training correction result representing training correction success is output; In response to the corrected training trend comparison result still representing that there is a deviation risk, or the corrected resource comparison result representing that there is an imbalance risk in the resource supply, a training correction result representing training correction failure is output.

10. An electronic device, comprising: Including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the model training detection method according to any one of claims 1 to 9.

Images