Model training control method and electronic equipment

Abstract

The invention discloses a model training control method and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining at least one intermediate calculation result including gradient data and a model learning loss value, and operation state parameters of target calculation equipment when the target calculation equipment executes a model training task, and performing anomaly analysis on the execution state of the target computing device for executing the model training task to obtain an anomaly analysis result, thereby realizing online monitoring of the model training process from two aspects of an intermediate calculation result generated by model training and the operation state parameter of the target computing device. According to the exception analysis result, the corresponding self-healing control parameter for the model training task is called, and the configuration of executing the model training task by the target computing device is adjusted according to the self-healing control parameter, so that the configuration of executing the model training task by the target computing device is optimized online, the model optimization efficiency is improved, and the user experience is improved. And equipment resources are saved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training control method and electronic device. Background Technology

[0002] With the development of artificial intelligence, the training scale of AI models has gradually expanded, consuming a significant amount of computing power and storage resources, and the training time has also been increasing. To ensure that the trained model can meet user needs, it is necessary to evaluate the quality of model training. However, traditional solutions mostly require waiting for the model to complete training and then validating it on a validation dataset to evaluate its quality. If the model quality does not meet the requirements, it needs to be retrained, resulting in low training efficiency.

[0003] How to monitor and optimize the state during model training is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a model training control method and electronic device to at least solve the problem of the lack of state monitoring and optimization schemes in the model training process in related technologies.

[0005] This invention provides a model training control method, comprising: When the target computing device performs a model training task, intermediate calculation results and the operating status parameters of the target computing device are obtained. Based on the intermediate calculation results and the running status parameters, an anomaly analysis is performed on the execution status of the target computing device performing the model training task, and an anomaly analysis result is obtained. Based on the anomaly analysis results, the corresponding self-healing control parameters for the model training task are invoked; The configuration for the target computing device to perform the model training task is adjusted according to the self-healing control parameters; The intermediate calculation results include at least one of gradient data and model learning loss values.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model training control methods.

[0007] This invention achieves online monitoring of the model training process from two aspects: intermediate computation results and model learning loss values. By acquiring intermediate computation results and operating status parameters of the target computing device during model training, anomaly analysis is performed on the execution status of the target computing device based on these results and operating status parameters. Furthermore, by invoking corresponding self-healing control parameters for the model training task based on the anomaly analysis results, the configuration of the target computing device for model training is adjusted. This enables online optimization of the configuration during model training, accelerating model optimization compared to traditional methods that wait for the model training task to complete before evaluation, thus saving device resources. Attached Figure Description

[0008] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart of a model training control method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a model training system provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a dynamic diagnostic module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the directional enhancement module provided in an embodiment of the present invention; Among them, 201 refers to monitoring equipment. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0011] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0012] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] The embodiments of the present invention provide a model training control method. The method is described in detail below in conjunction with the execution flow of the model training control method.

[0014] In related technologies, the model training process lacks real-time monitoring and self-healing capabilities. Currently, in deep learning training, monitoring the model's training status mainly relies on manual observation of trends in metrics such as loss function curves and accuracy. This monitoring method has significant lag and limitations; it cannot capture fine-grained anomalies during model training. For example, gradient explosion may cause the model to collapse within a few batches, but manual monitoring is unlikely to detect it in a timely manner.

[0015] Figure 1 A flowchart of a model training control method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a model training system provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a dynamic diagnostic module provided in an embodiment of the present invention.

[0016] like Figure 1 As shown, the model training control method provided in this embodiment of the invention may include: S101: when the target computing device performs a model training task, obtaining intermediate calculation results and the operating status parameters of the target computing device.

[0017] S102: Based on the intermediate calculation results and running status parameters, perform anomaly analysis on the execution status of the target computing device performing the model training task, and obtain the anomaly analysis results.

[0018] S103: Based on the anomaly analysis results, call the corresponding self-healing control parameters for the model training task.

[0019] S104: Adjust the configuration of the target computing device to perform model training tasks based on the self-healing control parameters.

[0020] The intermediate calculation results include at least one of gradient data and model learning loss values.

[0021] The model training control method provided in this invention can be applied to the target computing device, or to monitoring devices other than the target computing device. For example... Figure 2 As shown, the monitoring device 201 is connected to the target computing device to obtain intermediate calculation results and operating status parameters from the target computing device, and to perform fault diagnosis and self-healing strategy selection. To this end, a dynamic diagnostic module can be deployed in the monitoring device 201, which may include a real-time monitoring module, a fault diagnosis module, and a self-healing strategy selection module.

[0022] The target computing device in this embodiment of the invention is a computing device used to perform model training tasks. There can be one or more target computing devices. In large-scale distributed training, multiple target computing devices often jointly execute the same model training task; therefore, the target computing devices in this embodiment of the invention can be the target computing devices in a distributed training system.

[0023] The execution steps of the model training task targeted in this embodiment of the invention may include: obtaining initial model network parameters and a training dataset. The model training task may include N epochs of iterative training. One epoch may include multiple training steps, the specific number of which depends on the size of the training dataset and the batch size. For example, if the training dataset has 50,000 samples and the batch size is set to 250, then one epoch includes 50,000 / 250 = 200 training steps. In each training step, the target computing device obtains a batch of data from the training dataset and performs forward and backward propagation calculations using the model network parameters to update the model network parameters. After completing one training iteration on all samples in the training dataset, one epoch of iterative training is completed. Thus, after completing all N epochs of iterative training, the model training task is completed.

[0024] For S101, such as Figure 2 As shown, the real-time monitoring module in the dynamic diagnostic module can collect key indicators in all aspects at different frequencies during the process of the target computing device performing model training tasks.

[0025] In this embodiment of the invention, the state of the target computing device executing the model training task is monitored from two dimensions: the intermediate calculation results obtained by the target computing device in executing the model training task and the running state parameters of the target computing device when executing the model training task.

[0026] In this embodiment of the invention, intermediate calculation results may include at least one of gradient data and model learning loss values.

[0027] The gradient data refers to the set of derivatives (or partial derivatives) of the loss function with respect to each parameter of the model, which are calculated during backpropagation. In this embodiment of the invention, gradient data obtained when the target computing device performs the model training task can be collected at a first frequency. The first frequency can be the full collection of gradient data obtained at each training step when the target computing device performs the model training task, or it can be the collection of gradient data once at intervals of a first preset number of training steps. The magnitude of the first frequency can be dynamically adjusted according to the device load of the monitoring device 201 and the target computing device.

[0028] The model learning loss quantifies the difference between the model's predicted output and the true target on given data. It can be a single numerical value representing the sum of all prediction errors. Depending on the design of the model training task, the model learning loss can be calculated using a loss function designed according to the needs of the training task. For example, if the model training task is a regression task, the model learning loss can be the mean squared error between the model's predicted results and the true values, i.e., the average of the squares of the differences between the predicted and true values. If the model training task is a classification task, the model learning loss can be the cross-entropy loss between the model's predicted results and the true values, which measures the difference between the probability distribution predicted by the model and the true class distribution.

[0029] In this embodiment of the invention, the model learning loss value can include at least one of the following: the loss value calculated after one training step (batch loss value), the loss value calculated after one epoch of training (epoch-average training loss), and the loss value calculated after completing all epochs of training (final training loss). These three different types of model learning loss values ​​can measure the state of the target computing device performing the model training task from different granularities. The granularity of the model learning loss value can be selected according to the device load of the working scenario. In this embodiment of the invention, a second frequency can be used to collect the model learning loss value and the type of model learning loss value when the target computing device performs the model training task. The magnitude of the second frequency can be dynamically adjusted according to the device load of the monitoring device 201 and the target computing device.

[0030] In this embodiment of the invention, the operating status parameters of the target computing device when executing model tasks are collected. The types of operating status parameters may include resource utilization assessment parameters of the target computing device's hardware resources and / or software resources, such as memory utilization rate. In this embodiment of the invention, a third frequency can be used to collect the operating status parameters of the target computing device when executing model training tasks. The magnitude of the third frequency can be dynamically adjusted according to the device load of the monitoring device 201 and the target computing device.

[0031] For S102, such as Figure 2 As shown, the fault diagnosis module in the dynamic diagnosis module can perform anomaly analysis based on the intermediate calculation results and operating status parameters collected by the real-time monitoring module to obtain the anomaly analysis results.

[0032] In this embodiment of the invention, a pre-built exception event rule base can be used to record the exception data range corresponding to each parameter. The exception data range can be the exception value range of intermediate calculation results and / or operating status parameters, or the exception value range of evaluation parameters of intermediate calculation results and / or operating status parameters.

[0033] In step S102, based on the intermediate calculation results and running status parameters, an anomaly analysis is performed on the execution status of the target computing device performing the model training task to obtain the anomaly analysis results. These results may include: obtaining the first abnormal data range corresponding to the intermediate calculation results and the second abnormal data range corresponding to the running status parameters from the anomaly event rule base; calculating the quality evaluation parameters of the intermediate calculation results; and if there exists at least one of the quality evaluation parameters satisfying the first abnormal data range and the running status parameters satisfying the second abnormal data range, then it is determined that an abnormal event has occurred in the process of the target computing device performing the model training task.

[0034] In other words, for intermediate calculation results, anomaly detection is performed by checking whether their quality evaluation parameters fall within the first abnormal data range. For operational status parameters, anomaly detection is performed by checking whether the parameters themselves fall within the second abnormal data range.

[0035] In this embodiment of the invention, anomaly analysis can be performed at a fourth frequency. Anomaly analysis of intermediate calculation results and anomaly analysis of operating status parameters can be performed at different frequencies and are not synchronously. The magnitude of the fourth frequency can be dynamically adjusted according to the load of the monitoring device 201 and the target computing device.

[0036] Based on this, for gradient data, in some optional embodiments of the present invention, obtaining intermediate calculation results in S101 may include: after each first number of training steps, obtaining the gradient data to be detected obtained by the target computing device performing the model training task. Calculating the quality evaluation parameters of the intermediate calculation results may include: calculating the gradient norm of the gradient data to be detected; the quality evaluation parameters satisfying the first abnormal data range may include: the gradient norm of the gradient data to be detected being greater than a first threshold.

[0037] For example, after every 10 training steps (Steps) completed during model training on the target computing device, the L2 norm of the gradient data in those 10 training steps (Steps) is calculated. This value reflects the overall magnitude of parameter updates. By establishing a historical baseline for the gradient norm, abnormal fluctuation patterns can be identified. The first threshold can be 1e5; that is, if the gradient norm of the gradient data to be detected is greater than 1e5, it is determined that there is a risk of gradient explosion. At this time, the monitoring device 201 marks the current optimization trajectory of the target computing device performing the model training task as abnormal.

[0038] Regarding the model learning loss value, in some optional embodiments of the present invention, obtaining intermediate calculation results in S101 may include: obtaining the model learning loss value calculated by the target computing device after the target computing device completes the training step on a batch of data; calculating the quality evaluation parameters of the intermediate calculation results, including: statistically analyzing the first proportion of non-numerical data in the model learning loss value; and ensuring that the quality evaluation parameters meet the first abnormal data range, including: the first proportion being greater than a second threshold.

[0039] For example, after the target computing device completes a training step (including one forward propagation calculation and one back propagation calculation) on a batch of data, the proportion of non-numerical (NaN) data that appears during the calculation of the model learning loss value can be obtained. This mechanism can identify anomalies such as data corruption and numerical overflow. The second threshold can be 0.1%, meaning that if the proportion of non-numerical data in the model learning loss value is greater than 0.1%, it is determined that the training data may be corrupted or have a format error.

[0040] Regarding the running status parameters, in this embodiment of the invention, obtaining the running status parameters of the target computing device in S101 may include: obtaining the memory usage parameters allocated to the model training task by the target computing device and the rate of change of the memory usage parameters; the running status parameters satisfying the second abnormal data range includes: if at least one of the memory usage parameters being greater than a third threshold and the rate of change of the memory usage parameters being greater than a fourth threshold is satisfied, then it is determined that the running status parameters satisfy the second abnormal data range.

[0041] For example, runtime status parameters can include the memory usage rate of the target computing device. This memory usage rate can be accessed every second through the target computing device's memory management interface, while simultaneously recording the historical trajectory of memory allocation. Early warnings can be issued when memory leak characteristics (such as continuous linear growth) are detected. The third threshold can be 90%, meaning that if the memory usage rate of the target computing device exceeds 90% when performing model training tasks, a memory overflow event is predicted to occur.

[0042] In addition, such as Figure 3 As shown, in addition to the abnormal event rule base introduced above, the fault diagnosis module can also be used to call the pre-trained fault prediction model to analyze abnormal event scenarios not covered by the abnormal event rule base. That is, in S102, based on the intermediate calculation results and operating status parameters, anomaly analysis is performed on the execution status of the target computing device performing the model training task to obtain the anomaly analysis results. This can include: inputting the intermediate calculation results and operating status parameters into the pre-trained fault prediction model, outputting the abnormal event prediction results and the confidence score of the abnormal event prediction results; using the abnormal event prediction results and confidence scores as the anomaly analysis results; and calling the corresponding self-healing control parameters for the model training task based on the anomaly analysis results, including: if the abnormal event prediction results indicate the existence of an abnormal event and the confidence score is greater than the fifth threshold, then calling the corresponding self-healing control parameters according to the type of abnormal event included in the abnormal event prediction results.

[0043] The process of inputting intermediate calculation results and operating status parameters into a pre-trained fault prediction model and outputting abnormal event prediction results and confidence scores of the abnormal event prediction results may include: if neither the intermediate calculation results nor the operating status parameters meet the abnormal data range recorded in the abnormal event rule base, then the intermediate calculation results and operating status parameters are input into the pre-trained fault prediction model, and the abnormal event prediction results and confidence scores are output.

[0044] In the fault diagnosis module, "confidence level" refers to the degree of certainty that the pre-trained fault prediction model is about the most significant root cause of the currently diagnosed fault (i.e., the root cause category with the highest probability). The fifth threshold can be 0.85. This threshold is mainly set based on the model's performance analysis on the independent validation set and the requirements for controlling the risk of misdiagnosis. 0.85 is a relatively high threshold, meaning that automatic self-healing is only triggered when the pre-trained fault prediction model is highly certain, significantly reducing the risk of erroneous self-healing due to misdiagnosis. When the confidence score output by the pre-trained fault prediction model is lower than the fifth threshold, the monitoring device 201 can output a manual review prompt to reduce the risk in low-confidence scenarios through manual intervention, thereby ensuring the overall reliability of the problem diagnosis and self-healing process.

[0045] In this embodiment of the invention, inputting intermediate calculation results and operating status parameters into a pre-trained fault prediction model and outputting abnormal event prediction results and confidence scores of the abnormal event prediction results may include: calculating feature vectors corresponding to intermediate calculation results and operating status parameters, wherein the type of feature vectors may include at least one of gradient distribution parameters, graph structure, and hardware indicator parameters; inputting the feature vectors into the pre-trained fault prediction model and outputting abnormal event prediction results and confidence scores of the abnormal event prediction results.

[0046] In addition, pre-trained fault prediction models can be learned online, for example, the judgment logic can be optimized based on uploaded correction records.

[0047] The embodiments of the present invention may further include: if the confidence score is greater than the seventh threshold and less than the fifth threshold, sending the abnormal event prediction result to the administrator address, receiving the uploaded manual abnormal event judgment result, and updating the abnormal event rule base using the manual abnormal event judgment result.

[0048] Therefore, the fault diagnosis module provided in this embodiment of the invention adopts a hybrid architecture that combines rule-driven and pre-trained fault prediction model analysis. The rule engine has three core judgment logics: when the gradient L2 norm exceeds the first threshold (e.g., 1e5), it is determined that there is a risk of gradient explosion, and the system will mark the current optimization trajectory as abnormal; when the proportion of non-numerical data is greater than the second threshold (e.g., 0.1%), it is determined that the training data may be corrupted or have a format error; when the memory usage rate exceeds the third threshold (90%), it is predicted that memory overflow may occur.

[0049] In this embodiment of the invention, S103, based on the anomaly analysis result, calls the corresponding self-healing control parameters for the model training task, which may include: if the root cause of the anomaly analysis result is abnormal training sample data, then controlling the model training task of the target computing device to roll back to the historical checkpoint when the most recent anomaly analysis result was normal, and moving the training sample data and corresponding intermediate calculation results used between the historical checkpoint and the current checkpoint to the target isolated storage area; if the root cause of the anomaly analysis result is abnormal hyperparameters, then adjusting the learning rate of the target computing device executing the model training task using an exponential decay method; if the root cause of the anomaly analysis result is hardware abnormality, then migrating the model training task to another computing device for execution.

[0050] Specifically, a self-healing strategy library is deployed in the self-healing strategy selection module. Based on this library, for abnormal events detected by the fault diagnosis module using an abnormal event rule base and a pre-trained fault prediction model, differentiated self-healing strategies are implemented according to the abnormal event analysis results, and the self-healing execution is controlled by the self-healing strategy selection module. Specifically, when the problem originates from data, it first rolls back to the nearest valid checkpoint, moves the contaminated batch data to the isolation area, and simultaneously triggers the data augmentation module to generate supplementary data. When the problem originates from hyperparameters, the learning rate is adjusted using an exponential decay method (initial adjustment increment of 30%). If three consecutive adjustments fail to improve the situation, the optimizer is automatically switched, and mixed-precision training is attempted to alleviate memory pressure. When the problem originates from hardware, the status of available computing devices is queried, and after completing the migration of computing tasks, the NCCL communication group is automatically reinitialized and the process restarts.

[0051] The model training control method provided in this invention obtains intermediate calculation results and operating status parameters of the target computing device when the target computing device performs a model training task. The intermediate calculation results include at least one of gradient data and model learning loss values. Based on the intermediate calculation results and operating status parameters, anomaly analysis is performed on the execution status of the target computing device performing the model training task to obtain anomaly analysis results. This enables online monitoring of the model training process from both the intermediate calculation results generated by model training and the operating status parameters of the target computing device. By calling the corresponding self-healing control parameters for the model training task based on the anomaly analysis results, the configuration of the target computing device performing the model training task is adjusted according to the self-healing control parameters. This achieves online optimization of the configuration of the target computing device performing the model training task during the model training process, thereby accelerating the model optimization progress. Compared with the traditional method of waiting for the model training task to be completed before evaluation, this significantly improves the model optimization efficiency and saves equipment resources.

[0052] Figure 4 This is a schematic diagram of the structure of the directional enhancement module provided in an embodiment of the present invention.

[0053] Traditional data augmentation methods lack specificity. Commonly used data augmentation techniques (such as random image cropping and rotation, and synonym replacement in text) employ fixed augmentation strategies, failing to consider the model's current learning state and specific weaknesses. Furthermore, data augmentation in specialized fields (such as healthcare and finance) requires maintaining semantic consistency and domain specificity. Existing methods struggle to effectively protect key semantic information during the augmentation process, leading to a decline in augmented data quality. Most augmentation methods lack real-time evaluation mechanisms for augmentation effects, making it difficult to promptly identify and correct negative impacts (such as introducing bias or disrupting data distribution).

[0054] In this regard, the model training control method provided in this embodiment of the invention may further include: performing data augmentation on the training dataset according to the type of the model training task, so that the target computing device uses the augmented training dataset to perform the model training task. That is to say, unlike related technologies, this embodiment of the invention performs data augmentation on the training dataset at least according to the type of the model training task, rather than using a general method to augment the training dataset.

[0055] like Figure 2 As shown, the model training control method provided in this embodiment of the invention can be implemented based on a targeted augmentation module. The targeted augmentation module may include a weakness analysis module, an augmentation strategy execution module, an effect verification module, and a closed-loop optimization module.

[0056] In this embodiment of the invention, by analyzing the weaknesses in the model's performance during the current training phase, targeted augmentation data is generated to compensate for these weaknesses. Unlike traditional random augmentation methods, a closed-loop workflow of "monitoring-analysis-generation-validation" is established to ensure that each augmentation operation directly serves to improve the model's performance on a specific task.

[0057] In this embodiment of the invention, data augmentation of the training dataset according to the type of the model training task, so that the target computing device can use the augmented training dataset to execute the model training task, may include: when the anomaly analysis result is that the intermediate calculation result is abnormal, data augmentation of the training dataset according to the type of the model training task, so that the target computing device can use the augmented training dataset to execute the next subtask in the model training task.

[0058] Specifically, such as Figure 4 As shown, the weakness analysis module can be used to evaluate model performance from multiple dimensions. First, for classification tasks, quality assessment parameters can be calculated for each category. These parameters can include at least one of precision, recall, and F1 score. After calculating the quality assessment parameters for each category, categories with quality assessment parameters below the average level are labeled as weakness categories. For generation tasks, generation quality assessment parameters are calculated. These parameters can include generation quality difference parameters between texts of different lengths, and / or generation quality difference parameters between texts of different sentence structures, thereby identifying the generation task types where the generation model has weaknesses.

[0059] Then, the weakness analysis module traces the source of errors in the target computing device's performance of the model training task. Specifically, this can be done by collecting high-confidence samples of model prediction errors and using semantic clustering techniques to deeply analyze the common features of these samples in text content (such as high-frequency entities, semantic relationships, sentiment expression, and domain themes) and language structure (such as complex sentence structures and specific grammar). For example, error samples may be mostly long and difficult sentences, multiple negations, or error samples may be concentrated in a specific domain, such as pediatrics in the medical field.

[0060] The analysis results will generate a structured weakness report. This report not only describes the weakness patterns and their impact, but more importantly, it precisely identifies the data types that need enhancement and the specific directions for enhancement. For example, if the error samples are mostly long and complex sentences with multiple negations, the enhancement goal is to generate sentence skeletons that meet the requirements for nesting levels and negation structures, and to constrain the required sentence complexity. If the error samples are concentrated in a specific field, such as pediatrics in medicine, the enhancement goal is to generate text focusing on common and frequently occurring pediatric diseases. These specific "enhancement goals" directly drive subsequent targeted data generation, ensuring that the enhancement operations accurately target the model's capability deficiencies in specific patterns.

[0061] In this embodiment of the invention, data augmentation of the training dataset according to the type of model training task may include: if the type of model training task is a classification task, then at least one of the following augmentation methods is used to augment the training dataset: synonym replacement, entity substitution, local rewriting, and adversarial example generation; if the type of model training task is a generation task, then at least one of the following augmentation methods is used to augment the training dataset: content expansion, sentence diversification, and explicit reference.

[0062] In this embodiment of the invention, an augmentation strategy library can be pre-built to record one or more of the aforementioned data augmentation strategies for the training dataset. For example, data augmentation strategies may include: synonym replacement (replacing non-critical entities with a domain dictionary), entity substitution (replacing core entities while maintaining sentence structure), partial rewriting (rewriting text fragments within an error set), and adversarial example generation (adding subtle perturbations). For generation tasks, the focus is on: content expansion (inserting relevant information at key positions), sentence diversification (changing sentence structure while maintaining semantics), and explicit reference (replacing vague references with concrete entities). Each strategy must have clearly defined applicable conditions and expected effects, and its priority weights must be set.

[0063] The enhancement strategy execution module searches the enhancement strategy library based on the vulnerability report provided by the vulnerability analysis module, enabling enhancement strategy selection and execution. This module can first optimize strategy combinations; complex vulnerabilities may require combining multiple simple strategies. Strict quality control is implemented during execution, using semantic similarity calculations to ensure the enhanced text retains its original meaning. All enhancement operations are logged in detail, including the strategies used, modifications made, and results generated, providing a basis for subsequent analysis.

[0064] Traditional training optimization processes lack closed-loop feedback mechanisms. Optimization techniques often address individual sub-problems in isolation (such as optimizing data loading or gradient calculation separately), lacking a global perspective for collaborative optimization. This fragmented optimization approach can lead to performance cancellation between different components. Most training systems lack continuous performance monitoring and optimization feedback loops. Key metrics during training (such as gradient distribution and parameter update magnitude) are not systematically collected and analyzed, resulting in missed optimization opportunities.

[0065] In this regard, the model training control method provided in this embodiment of the invention may further include: after the target computing device performs a model training task using the enhanced training dataset, obtaining a second task evaluation parameter for the target computing device performing the model training task; determining an optimized parameter for the second task evaluation parameter compared to the first task evaluation parameter when the target computing device performs the model training task using the training dataset; if the optimized parameter meets a sixth threshold range, recording the enhancement configuration parameters used in the enhanced training dataset and the scene state parameters that trigger the data enhancement task; if the optimized parameter does not meet the sixth threshold range, returning to perform data enhancement on the training dataset according to the type of the model training task, so that the target computing device performs the model training task using the enhanced training dataset.

[0066] Specifically, such as Figure 4 As shown, a multi-dimensional evaluation system is established using the effect verification module to verify the enhancement effect. At the data level, the quality indicators of the enhanced data, such as semantic consistency, language fluency, and domain relevance, are checked. At the model level, the performance changes of the model on the validation set after training with the enhanced data are monitored, with particular attention to the improvement of original weaknesses. An incremental evaluation mechanism is designed to add enhanced data to the training in batches and observe the gradient changes in model performance. A feedback adjustment mechanism is established to downweight or eliminate ineffective enhancement strategies and upweight effective strategies. The verification results are fed back to the weakness analysis module, forming a closed loop of continuous optimization.

[0067] like Figure 4As shown, a closed-loop optimization module is used to quantitatively evaluate the actual effect of enhancement strategies, establishing a virtuous cycle for strategy optimization. For example, if a weakness is identified, the weight of the enhancement strategy is increased; if the effect is insufficient, the enhancement strategy is replaced, and the original enhancement strategy is reduced in weight or eliminated. Specifically, when the diagnostic module detects a specific weakness and triggers the enhancement process, the system initiates multi-dimensional effect tracking: the core indicator for effect evaluation is the weakness mitigation rate, i.e., the rate at which the original weakness is mitigated after training with enhanced data. For example, when the mitigation rate is >60%, it is considered significantly effective, and the strategy-effect mapping database is recorded, recording the best strategy combination under different scenarios. When the system detects a similar problem again, the intelligent strategy selector will prioritize historically high-performing strategy combinations.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0069] Embodiments of the present invention also provide a model training control device, comprising: a real-time monitoring module, configured to acquire intermediate calculation results and operating status parameters of the target computing device when the target computing device performs a model training task; a fault diagnosis module, configured to perform anomaly analysis on the execution status of the target computing device performing the model training task based on the intermediate calculation results and operating status parameters, and obtain anomaly analysis results; and a self-healing strategy selection module, configured to call corresponding self-healing control parameters for the model training task based on the anomaly analysis results; and adjust the configuration of the target computing device performing the model training task based on the self-healing control parameters; wherein the intermediate calculation results include at least one of gradient data and model learning loss values.

[0070] In this embodiment of the invention, the fault diagnosis module performs anomaly analysis on the execution status of the target computing device performing the model training task based on intermediate calculation results and running status parameters, and obtains anomaly analysis results, which may include: obtaining a first abnormal data range corresponding to the intermediate calculation results and a second abnormal data range corresponding to the running status parameters from the abnormal event rule base; calculating the quality evaluation parameters of the intermediate calculation results; if there is at least one of the quality evaluation parameters satisfying the first abnormal data range and the running status parameters satisfying the second abnormal data range, then it is determined that an abnormal event has occurred in the process of the target computing device performing the model training task.

[0071] In this embodiment of the invention, the real-time monitoring module acquires intermediate calculation results, which may include: after each first number of training steps, acquiring the gradient data to be detected obtained by the target computing device performing the model training task; the fault diagnosis module calculates the quality evaluation parameters of the intermediate calculation results, which may include: calculating the gradient norm of the gradient data to be detected; the quality evaluation parameters satisfying the first abnormal data range may include: the gradient norm of the gradient data to be detected being greater than a first threshold.

[0072] In this embodiment of the invention, the real-time monitoring module obtains intermediate calculation results, which may include: after the target computing device completes the training step on a batch of data, obtaining the model learning loss value calculated by the target computing device; the fault diagnosis module calculates the quality evaluation parameters of the intermediate calculation results, which may include: the first proportion of non-numerical data in the model learning loss value; the quality evaluation parameters satisfying the first abnormal data range may include: the first proportion being greater than a second threshold.

[0073] In this embodiment of the invention, the real-time monitoring module acquires the operating status parameters of the target computing device, which may include: acquiring the memory usage parameters allocated to the model training task by the target computing device and the rate of change of the memory usage parameters; the operating status parameters satisfying the second abnormal data range may include: if at least one of the memory usage parameters being greater than a third threshold and the rate of change of the memory usage parameters being greater than a fourth threshold is satisfied, then it is determined that the operating status parameters satisfy the second abnormal data range.

[0074] In this embodiment of the invention, the fault diagnosis module performs anomaly analysis on the execution status of the target computing device performing the model training task based on intermediate calculation results and operating status parameters, and obtains anomaly analysis results. This may include: inputting intermediate calculation results and operating status parameters into a pre-trained fault prediction model, and outputting anomaly event prediction results and confidence scores of the anomaly event prediction results; using the anomaly event prediction results and confidence scores as the anomaly analysis results; the self-healing strategy selection module calls the corresponding self-healing control parameters for the model training task based on the anomaly analysis results, which may include: if the anomaly event prediction results indicate the existence of an anomaly event and the confidence score is greater than a fifth threshold, then calling the corresponding self-healing control parameters according to the type of anomaly event included in the anomaly event prediction results.

[0075] In this embodiment of the invention, the fault diagnosis module inputs intermediate calculation results and operating status parameters into the pre-trained fault prediction model and outputs abnormal event prediction results and confidence scores of the abnormal event prediction results. This may include: if neither the intermediate calculation results nor the operating status parameters meet the abnormal data range recorded in the abnormal event rule base, then the intermediate calculation results and operating status parameters are input into the pre-trained fault prediction model, and the abnormal event prediction results and confidence scores are output.

[0076] In this embodiment of the invention, the self-healing strategy selection module calls the corresponding self-healing control parameters for the model training task based on the anomaly analysis results. This may include: if the root cause of the anomaly analysis results is abnormal training sample data, then controlling the model training task of the target computing device to roll back to the historical checkpoint when the most recent anomaly analysis result was normal, and moving the training sample data and corresponding intermediate calculation results used between the historical checkpoint and the current checkpoint to the target isolated storage area; if the root cause of the anomaly analysis results is abnormal hyperparameters, then adjusting the learning rate of the target computing device executing the model training task using an exponential decay method; if the root cause of the anomaly analysis results is hardware abnormality, then migrating the model training task to another computing device for execution.

[0077] The model training control device provided in this embodiment of the invention may further include a targeted augmentation module, which is used to augment the training dataset according to the type of model training task, so that the target computing device can use the augmented training dataset to perform the model training task.

[0078] In this embodiment of the invention, the targeted augmentation module augments the training dataset according to the type of the model training task, so that the target computing device can use the augmented training dataset to execute the model training task. This may include: when the anomaly analysis result is that the intermediate calculation result is abnormal, augmenting the training dataset according to the type of the model training task, so that the target computing device can use the augmented training dataset to execute the next subtask in the model training task.

[0079] In this embodiment of the invention, the targeted augmentation module performs data augmentation on the training dataset according to the type of the model training task, which may include: if the type of the model training task is a classification task, then at least one of the augmentation methods of synonym replacement, entity substitution, local rewriting, and adversarial example generation is used to augment the training dataset; if the type of the model training task is a generation task, then at least one of the augmentation methods of content expansion, sentence diversification, and explicit reference is used to augment the training dataset.

[0080] In this embodiment of the invention, the targeted augmentation module can also be used to obtain the second task evaluation parameters of the target computing device executing the model training task after the target computing device executes the model training task using the augmented training dataset; determine the optimized parameters of the second task evaluation parameters compared with the first task evaluation parameters when the target computing device executes the model training task using the training dataset; if the optimized parameters meet the sixth threshold range, record the augmentation configuration parameters used by the augmented training dataset and the scene state parameters that trigger the data augmentation task; if the optimized parameters do not meet the sixth threshold range, return to perform data augmentation on the training dataset according to the type of the model training task, so that the target computing device executes the model training task using the augmented training dataset.

[0081] For a description of the features in the embodiment corresponding to the model training control device, please refer to the relevant description in the embodiment corresponding to the model training control method, which will not be repeated here.

[0082] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the model training control method.

[0083] Embodiments of the present invention also provide a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model training control method embodiments during runtime.

[0084] In one exemplary embodiment, the aforementioned non-volatile storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0085] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model training control method embodiments.

[0086] Embodiments of the present invention also provide another computer program product, including a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model training control method embodiments.

[0087] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0088] The above provides a detailed description of the model training control method and electronic device provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A model training control method, characterized in that, include: When the target computing device performs a model training task, intermediate calculation results and the operating status parameters of the target computing device are obtained. Based on the intermediate calculation results and the running status parameters, an anomaly analysis is performed on the execution status of the target computing device performing the model training task, and an anomaly analysis result is obtained. Based on the anomaly analysis results, the corresponding self-healing control parameters for the model training task are invoked; The configuration for the target computing device to perform the model training task is adjusted according to the self-healing control parameters; The intermediate calculation results include at least one of gradient data and model learning loss values.

2. The model training control method according to claim 1, characterized in that, Based on the intermediate calculation results and the running status parameters, anomaly analysis is performed on the execution status of the target computing device performing the model training task, and the anomaly analysis results are obtained, including: The first abnormal data range corresponding to the intermediate calculation result and the second abnormal data range corresponding to the running status parameter are obtained from the abnormal event rule base. Calculate the quality evaluation parameters of the intermediate calculation results; If at least one of the quality evaluation parameters satisfies the first abnormal data range and the operating status parameter satisfies the second abnormal data range, then it is determined that an abnormal event has occurred during the execution of the model training task by the target computing device.

3. The model training control method according to claim 2, characterized in that, Obtaining the intermediate calculation results includes: After each first number of training steps, the gradient data to be detected obtained by the target computing device in performing the model training task is acquired; The quality evaluation parameters for the intermediate calculation results are calculated, including: Calculate the gradient norm of the gradient data to be detected; The quality evaluation parameters satisfy the first abnormal data range, including: The gradient norm of the gradient data to be detected is greater than the first threshold.

4. The model training control method according to claim 2, characterized in that, Obtaining the intermediate calculation results includes: After the target computing device completes the training step on a batch of data, the model learning loss value calculated by the target computing device is obtained; The quality evaluation parameters for the intermediate calculation results are calculated, including: The proportion of non-numerical data in the model learning loss value is calculated as follows: The quality evaluation parameters satisfy the first abnormal data range, including: The first ratio is greater than the second threshold.

5. The model training control method according to claim 2, characterized in that, Obtaining the operating status parameters of the target computing device includes: Obtain the memory usage parameters allocated by the target computing device to the model training task and the rate of change of the memory usage parameters; The operating status parameters satisfy the second abnormal data range, including: If at least one of the following conditions is met: the memory usage parameter is greater than the third threshold, or the rate of change of the memory usage parameter is greater than the fourth threshold, then the running status parameter is determined to meet the second abnormal data range.

6. The model training control method according to claim 1, characterized in that, Based on the intermediate calculation results and the running status parameters, anomaly analysis is performed on the execution status of the target computing device performing the model training task, and the anomaly analysis results are obtained, including: The intermediate calculation results and the operating status parameters are input into the pre-trained fault prediction model, and the abnormal event prediction results and the confidence scores of the abnormal event prediction results are output. The anomaly analysis results are the anomaly prediction results and the confidence scores. Based on the anomaly analysis results, the corresponding self-healing control parameters for the model training task are invoked, including: If the abnormal event prediction result indicates the existence of an abnormal event and the confidence score is greater than the fifth threshold, then the corresponding self-healing control parameter is invoked according to the type of abnormal event included in the abnormal event prediction result.

7. The model training control method according to claim 6, characterized in that, The intermediate calculation results and the operating status parameters are input into the pre-trained fault prediction model, which outputs the abnormal event prediction results and the confidence scores of the abnormal event prediction results, including: If neither the intermediate calculation result nor the running status parameter meets the abnormal data range recorded in the abnormal event rule base, then the intermediate calculation result and the running status parameter are input into the pre-trained fault prediction model, and the abnormal event prediction result and the confidence score are output.

8. The model training control method according to claim 1, characterized in that, Based on the anomaly analysis results, the corresponding self-healing control parameters for the model training task are invoked, including: If the root cause of the failure in the anomaly analysis result is abnormal training sample data, then the model training task of the target computing device is rolled back to the historical checkpoint when the most recent anomaly analysis result was normal, and the training sample data used between the historical checkpoint and the current checkpoint and the corresponding intermediate calculation results are moved to the target isolated storage area. If the root cause of the fault in the anomaly analysis results is an abnormal hyperparameter, then the learning rate of the target computing device performing the model training task is adjusted using an exponential decay method. If the root cause of the fault in the anomaly analysis results is a hardware malfunction, then the model training task will be migrated to another computing device for execution.

9. The model training control method according to claim 1, characterized in that, Also includes: The training dataset is augmented according to the type of the model training task, so that the target computing device can use the augmented training dataset to perform the model training task.

10. The model training control method according to claim 9, characterized in that, Data augmentation is performed on the training dataset according to the type of the model training task, so that the target computing device can use the augmented training dataset to perform the model training task, including: When the anomaly analysis result indicates that the intermediate calculation result is abnormal, the training dataset is augmented according to the type of the model training task, so that the target computing device can use the augmented training dataset to execute the next subtask in the model training task.

11. The model training control method according to claim 9, characterized in that, Data augmentation of the training dataset is performed according to the type of model training task, including: If the training task of the model is a classification task, then at least one of the following augmentation methods is used to augment the training dataset: synonym replacement, entity substitution, local rewriting, and adversarial example generation. If the training task of the model is a generation task, then at least one of the following enhancement methods is used to augment the training dataset: content expansion, sentence diversification, and explicit reference.

12. The model training control method according to claim 9, characterized in that, Also includes: After the target computing device performs the model training task using the enhanced training dataset, the second task evaluation parameters for the target computing device performing the model training task are obtained; Determine optimized parameters for the second task evaluation parameters relative to the first task evaluation parameters when the target computing device performs the model training task using the training dataset; If the optimization parameters meet the sixth threshold range, then record the enhancement configuration parameters used in the enhanced training dataset and the scenario state parameters that trigger the data augmentation task. If the optimization parameters do not meet the sixth threshold range, then the training dataset is augmented according to the type of the model training task so that the target computing device can use the augmented training dataset to perform the model training task.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model training control method as described in any one of claims 1 to 12.

Images