Model training control method and electronic device

Abstract

The application discloses a model training control method and electronic equipment, and relates to the technical field of artificial intelligence, which comprises the following steps: when a target computing device executes a model training task, obtaining at least one intermediate calculation result in gradient data and model learning loss value and running state parameters of the target computing device, performing abnormality analysis on the execution state of the target computing device in executing the model training task, and obtaining an abnormality analysis result, thereby realizing online monitoring of the model training process from two aspects of the intermediate calculation result generated by the model training and the running state parameters of the target computing device; and calling corresponding self-recovery control parameters of the model training task according to the abnormality analysis result, adjusting the configuration of the target computing device in executing the model training task according to the self-recovery control parameters, realizing online optimization of the configuration of the target computing device in executing the model training task, thereby improving the model optimization efficiency and saving device resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training control method and an electronic device. BACKGROUND

[0002] With the development of artificial intelligence, the training scale of artificial intelligence models gradually expands, occupying a large amount of device computing power and storage resources, and the training time consumption is also increasing. In order to ensure that the trained model can meet the use requirements of users, the quality of model training needs to be evaluated. However, most of the traditional schemes need to wait for the model training to be completed, and then verify the model on the verification data set to evaluate the model quality. If the model quality does not meet the requirements, it also needs to be retrained, resulting in low model training efficiency.

[0003] How to realize the state monitoring and optimization in the model training process is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The present application provides a model training control method and an electronic device to at least solve the problem of lack of state monitoring and optimization scheme in the model training process in the related art.

[0005] The present application provides a model training control method, comprising:

[0006] When a target computing device executes a model training task, an intermediate calculation result and a running state parameter of the target computing device are obtained;

[0007] According to the intermediate calculation result and the running state parameter, an execution state of the target computing device executing the model training task is analyzed for abnormality, and an abnormality analysis result is obtained;

[0008] According to the abnormality analysis result, a corresponding self-recovery control parameter for the model training task is called;

[0009] According to the self-recovery control parameter, the configuration of the target computing device executing the model training task is adjusted;

[0010] Among them, the intermediate calculation result includes at least one of gradient data and model learning loss value.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of any one of the above model training control methods.

[0012] By the present application, since the intermediate calculation result and the running state parameter of the target computing device are acquired when the target computing device performs the model training task, the intermediate calculation result includes at least one of gradient data and a model learning loss value; the execution state of the target computing device performing the model training task is abnormally analyzed according to the intermediate calculation result and the running state parameter, and an abnormal analysis result is obtained, so that the online monitoring of the model training process is realized from two aspects of the intermediate calculation result generated by the model training and the running state parameter of the target computing device; by calling the corresponding self-recovery control parameter of the model training task according to the abnormal analysis result, the configuration of the target computing device performing the model training task is adjusted according to the self-recovery control parameter, so that the online optimization of the configuration of the target computing device performing the model training task in the model training process is realized, thereby accelerating the model optimization progress, significantly improving the model optimization efficiency compared with the traditional method of waiting for the model training task to be completed before evaluation, and saving the device resources. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A flow chart of a model training control method provided for the embodiments of the present application;

[0015] Figure 2 A structural schematic diagram of a model training system provided for the embodiments of the present application;

[0016] Figure 3 A structural schematic diagram of a dynamic diagnosis module provided for the embodiments of the present application;

[0017] Figure 4 A structural schematic diagram of a directional enhancement module provided for the embodiments of the present application;

[0018] Among them, 201 is a monitoring device. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0020] It should be noted that in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0021] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0022] The embodiment of the present application provides a model training control method, and the method will be described in detail below in combination with the execution flow of the model training control method.

[0023] In the related art, the model training process lacks real-time monitoring and self-healing capability. In the current deep learning training process, the training state monitoring of the model mainly depends on the manual observation of the change trend of the loss function curve, accuracy and other indicators. This monitoring method has obvious lag and limitation. This method cannot capture the fine-grained abnormalities in the model training process. For example, gradient explosion may cause the model to collapse within a few batches, but manual monitoring is difficult to discover in time.

[0024] Figure 1 A flowchart of a model training control method provided by an embodiment of the present application is provided. Figure 2 A structural schematic diagram of a model training system provided by an embodiment of the present application is provided. Figure 3 A structural schematic diagram of a dynamic diagnosis module provided by an embodiment of the present application is provided.

[0025] As shown in Figure 1 The model training control method provided by the embodiment of the present application can include: S101: when a target computing device executes a model training task, obtaining an intermediate calculation result and a running state parameter of the target computing device.

[0026] S102: according to the intermediate calculation result and the running state parameter, performing abnormality analysis on the execution state of the target computing device executing the model training task, to obtain an abnormality analysis result.

[0027] S103: according to the abnormality analysis result, calling a corresponding self-healing control parameter for the model training task.

[0028] S104: adjusting the configuration of the target computing device executing the model training task according to the self-healing control parameter.

[0029] The intermediate calculation result includes at least one of gradient data and a model learning loss value.

[0030] The model training control method provided by the embodiments of the present application can be applied to a target computing device or a monitoring device other than the target computing device. Figure 2 As shown in the figure, the monitoring device 201 is connected to the target computing device for monitoring, to obtain the intermediate calculation result and the running state parameter from the target computing device, and to perform fault diagnosis and self-healing strategy selection. For this purpose, a dynamic diagnosis module can be deployed in the monitoring device 201, which can include a real-time monitoring module, a fault diagnosis module and a self-healing strategy selection module.

[0031] The target computing device to which the embodiments of the present application are directed is a computing device for performing a model training task. The number of target computing devices can be one or more. In the process of large-scale distributed training, multiple target computing devices are often involved in jointly performing the same model training task, and the target computing device to which the embodiments of the present application are directed can be each target computing device in a distributed training system.

[0032] The execution steps of the model training task to which the embodiments of the present application are directed can include obtaining initial model network parameters and a training data set. The model training task can include N rounds of iterative training (Epoch), and one round of iterative training (Epoch) can include multiple training steps (Step), the specific number of which depends on the size of the training data set and the batch data size (BatchSize). For example, if the training data set has 50,000 samples and the batch data size (Batch Size) is set to 250, then one round of iteration (Epoch) includes 50,000 / 250=200 training steps (Step). In each training step (Step), the target computing device obtains a batch of data (Batch) from the training data set, performs forward propagation calculation and backward propagation calculation using the model network parameters, and updates the model network parameters. After completing one training of all samples in the training data set, one round of iterative training (Epoch) is completed. Thus, after completing all N rounds of iterative training, the model training task is completed.

[0033] For S101, as shown in the figure, the real-time monitoring module in the dynamic diagnosis module can comprehensively collect key indicators in the process of the target computing device performing the model training task at different frequencies. Figure 2

[0034] In the embodiments of the present application, the state of the target computing device performing the model training task is monitored from two dimensions of the intermediate calculation result obtained by the target computing device performing the model training task and the running state parameter of the target computing device when performing the model task. ​

[0035] In the embodiments of the present application, the intermediate calculation result can include at least one of gradient data and a model learning loss value.

[0036] The gradient data is a set of derivatives (or partial derivatives) of a loss function with respect to each parameter of the model, which is calculated in the back propagation process. In the embodiments of the present application, the gradient data obtained when the target computing device executes the model training task at a first frequency can be used. The first frequency can be the gradient data obtained every training step (Step) when the target computing device executes the model training task at full capacity, or the gradient data obtained once every first preset number of training steps (Step). The size of the first frequency can be dynamically adjusted according to the device load of the monitoring device 201 and the target computing device.

[0037] The model learning loss value is used to quantify the difference between the predicted output of the model on the given data and the real target, which can be a single value representing the summary of all prediction errors. According to the design of the model training task, the model learning loss value can be calculated according to the loss function designed according to the requirements of the model training task. For example, if the model training task is a regression task, the model learning loss value can be the mean square error between the predicted result output by the model and the sample true value, that is, the average of the square of the difference between the predicted value and the true value; if the model training task is a classification task, the model learning loss value can be the cross-entropy loss between the predicted result output by the model and the sample true value, which is used to measure the difference between the probability distribution predicted by the model and the real class distribution.

[0038] In the embodiments of the present application, the model learning loss value can include at least one of the loss value calculated after a training step (Step) (batch loss value), the loss value calculated after an iteration training (Epoch) (periodic average training loss), and the loss value calculated after completing all iteration training (Epoch) (final training loss). The three different types of model learning loss values can measure the state of the target computing device executing the model training task from different granularities, and the model learning loss value with which granularity can be selected according to the device load of the working scene. In the embodiments of the present application, the model learning loss value and the type of the model learning loss value obtained when the target computing device executes the model training task at a second frequency can be used. The size of the second frequency can be dynamically adjusted according to the device load of the monitoring device 201 and the target computing device.

[0039] In the embodiments of the present application, the running state parameters of the target computing device when performing the model task are collected, and the types of the running state parameters can include resource occupation evaluation parameters of hardware resources and / or software resources of the target computing device, such as memory occupation rate. In the embodiments of the present application, the running state parameters of the target computing device when performing the model training task can be collected at a third frequency, and the size of the third frequency can be dynamically adjusted according to the device load size of the monitoring device 201 and the target computing device.

[0040] For S102, as shown in Figure 2 The abnormal analysis result can be obtained by performing abnormal analysis on the intermediate calculation results and the running state parameters collected by the real-time monitoring module based on the fault diagnosis module in the dynamic diagnosis module.

[0041] In the embodiments of the present application, the abnormal event rule library pre-constructed can be used to record the abnormal data range corresponding to each parameter. The abnormal data range can be the abnormal numerical range of the intermediate calculation result and / or the running state parameter, or the abnormal numerical range of the evaluation parameter of the intermediate calculation result and / or the evaluation parameter of the running state parameter.

[0042] In S102, the execution state of the target computing device performing the model training task is analyzed for abnormality based on the intermediate calculation result and the running state parameter, and the abnormal analysis result is obtained, which can include: obtaining the first abnormal data range corresponding to the intermediate calculation result and the second abnormal data range corresponding to the running state parameter from the abnormal event rule library; calculating the quality evaluation parameter of the intermediate calculation result; if at least one of the quality evaluation parameter meets the first abnormal data range and the running state parameter meets the second abnormal data range, it is determined that an abnormal event occurs in the process of the target computing device performing the model training task.

[0043] That is to say, for the intermediate calculation result, whether its quality evaluation parameter falls into the first abnormal data range is used for abnormal detection. For the running state parameter, whether the parameter itself falls into the second abnormal data range is used for abnormal detection.

[0044] In the embodiments of the present application, the abnormal analysis can be performed at a fourth frequency, and the abnormal analysis of the intermediate calculation result and the abnormal analysis of the running state parameter can be performed at different frequencies and not synchronously. The size of the fourth frequency can be dynamically adjusted according to the device load size of the monitoring device 201 and the target computing device.

[0045] Based on this, for gradient data, in some optional embodiments of the embodiments of the present application, the intermediate calculation result obtained in S101 can include: obtaining the gradient data to be detected obtained by the target computing device performing the model training task every first number of training steps. The quality evaluation parameter of the intermediate calculation result can include: calculating the gradient norm of the gradient data to be detected; the quality evaluation parameter satisfying the first abnormal data range can include: the gradient norm of the gradient data to be detected being greater than the first threshold.

[0046] For example, the L2 norm of the gradient data in 10 training steps (Step) can be calculated once every 10 training steps (Step) completed by the target computing device performing the model training task, and this value reflects the overall magnitude of the parameter update. By establishing a historical baseline of the gradient norm, an abnormal fluctuation pattern is identified. The first threshold can be 1e5, that is, if the gradient norm of the gradient data to be detected is greater than 1e5, it is determined that there is a risk of gradient explosion, and the monitoring device 201 marks the current optimization trajectory of the target computing device performing the model training task as abnormal.

[0047] For model learning loss value, in some optional embodiments of the embodiments of the present application, the intermediate calculation result obtained in S101 can include: obtaining the model learning loss value calculated by the target computing device after completing a training step on a batch of data; the quality evaluation parameter of the intermediate calculation result includes: statistics of the first proportion of non-numerical data in the model learning loss value; the quality evaluation parameter satisfying the first abnormal data range includes: the first proportion being greater than the second threshold.

[0048] For example, the proportion of non-numerical (Not a Number, NaN) data in the model learning loss value calculation process of the target computing device can be obtained after the target computing device completes a training step (including once forward propagation calculation and once back propagation calculation) on a batch of data. This mechanism can identify abnormal situations such as data corruption and numerical overflow. The second threshold can be 0.1%, that is, if the first proportion of non-numerical data in the model learning loss value is greater than 0.1%, it is determined that the training data may be damaged or have format errors.

[0049] For the running state parameter, in the embodiments of the present application, the running state parameter of the target computing device obtained in S101 can include: obtaining the memory occupation parameter allocated by the target computing device to the model training task and the change rate of the memory occupation parameter; the running state parameter satisfying the second abnormal data range includes: if at least one of the memory occupation parameter being greater than the third threshold and the change rate of the memory occupation parameter being greater than the fourth threshold is satisfied, it is determined that the running state parameter satisfies the second abnormal data range.

[0050] For example, the running state parameter can include the memory occupancy rate of the target computing device, for which the memory occupancy rate can be accessed through the memory management interface of the target computing device every second, while recording the historical trajectory of memory allocation, and giving an early warning when a memory leakage feature (such as continuous linear growth) is detected. The third threshold can be 90%, that is, if the memory occupancy rate of the target computing device when performing the model training task exceeds 90%, it is predicted that a memory overflow event may occur.

[0051] In addition, as Figure 3 As shown, in addition to the abnormal event rule library introduced above, the fault diagnosis module can also be used to call a pre-trained fault prediction model to analyze abnormal event scenarios not covered by the abnormal event rule library. That is, in S102, the execution state of the target computing device performing the model training task is abnormally analyzed according to the intermediate calculation result and the running state parameter, and an abnormal analysis result is obtained, which can include: inputting the intermediate calculation result and the running state parameter into the pre-trained fault prediction model, outputting an abnormal event prediction result and a confidence score of the abnormal event prediction result; taking the abnormal event prediction result and the confidence score as the abnormal analysis result; according to the abnormal analysis result, calling the corresponding self-healing control parameter for the model training task, including: if the abnormal event prediction result indicates that there is an abnormal event and the confidence score is greater than a fifth threshold, calling the corresponding self-healing control parameter according to the type of the abnormal event included in the abnormal event prediction result.

[0052] Among them, inputting the intermediate calculation result and the running state parameter into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score of the abnormal event prediction result can include: if the intermediate calculation result and the running state parameter do not satisfy the abnormal data range recorded in the abnormal event rule library, inputting the intermediate calculation result and the running state parameter into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score.

[0053] In the fault diagnosis module, "confidence" refers to the certainty of the pre-trained fault prediction model to the most main fault root cause diagnosed at present (i.e. the root cause category with the highest probability). The fifth threshold can be 0.85. This threshold is mainly set based on the performance analysis of the model on the independent validation set and the control requirement of the misdiagnosis risk. 0.85 is a relatively high threshold, which means that only when the pre-trained fault prediction model is very sure, the automatic self-healing is triggered, which significantly reduces the risk of false self-healing caused by misdiagnosis. When the confidence score output by the pre-trained fault prediction model is lower than the fifth threshold, the monitoring device 201 can output a manual review prompt to reduce the risk in the low confidence scenario through manual intervention, thereby ensuring the overall reliability of the problem diagnosis and self-healing process.

[0054] In the embodiment of the present application, inputting the intermediate calculation result and the running state parameter into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score of the abnormal event prediction result can include: calculating a feature vector corresponding to the intermediate calculation result and the running state parameter, and the type of the feature vector can include at least one of gradient distribution parameters, graph structure, and hardware indicator parameters; inputting the feature vector into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score of the abnormal event prediction result.

[0055] In addition, the pre-trained fault prediction model can also be subjected to online learning, for example, the judgment logic can be optimized according to the uploaded correction record.

[0056] Therefore, the embodiment of the present application can also include: if the confidence score is greater than the seventh threshold value and less than the fifth threshold value, sending the abnormal event prediction result to an administrator address, receiving an uploaded artificial abnormal event determination result, and updating the abnormal event rule library using the artificial abnormal event determination result.

[0057] Therefore, the fault diagnosis module provided by the embodiment of the present application adopts a hybrid architecture combining rule driving and pre-trained fault prediction model analysis. The rule engine has three types of core determination logic built-in: when the gradient L2 norm exceeds the first threshold value (such as 1e5), it is determined that there is a risk of gradient explosion, at which time the system will mark the current optimization trajectory as abnormal; when the proportion of non-numerical data exceeds the second threshold value (such as 0.1%), it is determined that the training data may be damaged or have format errors; when the memory occupancy rate exceeds the third threshold value (90%), it is predicted that a memory overflow may occur.

[0058] In the embodiment of the present application, the self-healing control parameter for the model training task corresponding to the abnormal analysis result is called in S103, which can include: if the fault root cause of the abnormal analysis result is training sample data abnormality, the model training task of the target computing device is controlled to roll back to the historical checkpoint when the last abnormal analysis result is normal, and the training sample data and the corresponding intermediate calculation result between the historical checkpoint and the current checkpoint are moved to the target isolated storage area; if the fault root cause of the abnormal analysis result is hyperparameter abnormality, the learning rate of the target computing device executing the model training task is adjusted in an exponential decay manner; if the fault root cause of the abnormal analysis result is hardware abnormality, the model training task is migrated to another computing device for execution.

[0059] Specifically, the self-recovery strategy library is deployed in the self-recovery strategy selection module, and based on the self-recovery strategy library, after the abnormal event detected by the abnormal event rule library and the pre-trained fault prediction model in the fault diagnosis module, the differentiated self-recovery strategy is implemented according to the abnormal event analysis result based on the component self-recovery strategy library, and the self-recovery execution is controlled by the self-recovery strategy selection module. Specifically, when the diagnosis problem comes from data, first, roll back to the nearest valid checkpoint, move the contaminated batch data into the isolation area, and at the same time trigger the data enhancement module to generate supplementary data; when the diagnosis problem comes from hyperparameters, adjust the learning rate in an exponential decay manner (the initial adjustment amplitude is 30%), if the adjustment does not improve for 3 times in a row, automatically switch the optimizer, and at the same time try to enable mixed precision training to relieve memory pressure; when the diagnosis problem comes from hardware, query the state of the available computing device, and automatically reinitialize the NCCL communication group after completing the computing task migration, and re-run.

[0060] The model training control method provided by the embodiment of the application realizes online monitoring of the model training process from two aspects of intermediate calculation results generated by model training and running state parameters of the target computing device, and realizes online optimization of the configuration of the target computing device performing the model training task in the model training process by adjusting the configuration of the target computing device performing the model training task according to the self-recovery control parameter corresponding to the model training task according to the abnormal analysis result, thereby speeding up the model optimization progress, significantly improving the model optimization efficiency compared with the traditional method of evaluating after the model training task is completed, and saving device resources.

[0061] Figure 4 The structure diagram of the directional enhancement module provided by the embodiment of the application is shown.

[0062] The conventional data enhancement method lacks pertinence. The commonly used data enhancement technology (such as random cropping and rotation of images, synonym replacement of texts) adopts a fixed enhancement strategy without considering the current learning state and specific weaknesses of the model. Moreover, data enhancement in professional fields (such as medical care and finance) needs to maintain semantic consistency and field specificity. The existing method is difficult to effectively protect key semantic information in the enhancement process, resulting in a decline in the quality of enhanced data. Most enhancement methods lack real-time evaluation mechanisms for the enhancement effect, making it difficult to discover and correct the negative effects (such as introducing bias and destroying data distribution) caused by enhancement in a timely manner.

[0063] To this end, the model training control method provided by the embodiments of the present application can further include: performing data augmentation on the training data set according to the type of the model training task, so that the target computing device performs the model training task by using the augmented training data set. That is to say, unlike the related art, the embodiments of the present application perform data augmentation on the training data set at least according to the type of the model training task, rather than using a general method to perform data augmentation on the training data set.

[0064] As shown in Figure 2 The model training control method provided by the embodiments of the present application can be implemented based on a directional augmentation module. The directional augmentation module can include a weakness analysis module, an augmentation strategy execution module, an effect verification module, and a closed-loop optimization module.

[0065] In the embodiments of the present application, the performance weaknesses of the model in the current training stage are analyzed, and the augmented data that can make up for these weaknesses are generated accordingly. Unlike the traditional random augmentation method, a closed-loop workflow of "monitoring-analysis-generation-verification" is established to ensure that each augmentation operation directly serves to improve the performance of the model on a specific task.

[0066] In the embodiments of the present application, the data augmentation on the training data set according to the type of the model training task, so that the target computing device performs the model training task by using the augmented training data set, can include: when the abnormality analysis result is an intermediate calculation result abnormality, performing data augmentation on the training data set according to the type of the model training task, so that the target computing device performs the next subtask in the model training task by using the augmented training data set.

[0067] Specifically, as shown in Figure 4 The weakness analysis module can be used to evaluate the performance of the model from multiple dimensions. First, for a classification task, quality evaluation parameters of each category can be calculated, and the types of the quality evaluation parameters can include at least one of precision, recall, and F1 value. After calculating the quality evaluation parameters of each category, the categories with quality evaluation parameters lower than the average level are marked as weak categories. For a generation task, a generation quality evaluation parameter is calculated, and the types of the generation quality evaluation parameter can include a generation quality difference parameter between texts of different lengths, and / or a generation quality difference parameter between texts of different sentence patterns, so as to determine the generation task type in which the generation model has weaknesses.

[0068] Then, the weakness analysis module error traces the weaknesses of the target computing device performing the model training task, which can be specifically performed by collecting high-confidence samples of model prediction errors, and using semantic clustering techniques to in-depth analyze the common features of these samples in text content (such as high-frequency entities, semantic relationships, emotional expressions, domain topics), language structure (such as complex sentence patterns, specific grammar), and the like. For example, error samples are mostly long and difficult sentences, multiple negative sentences, or error samples are concentrated in a specific field such as pediatrics in medicine, and the like.

[0069] The analysis result will generate a structured weakness report, which not only describes the weakness pattern and its impact, but more importantly, accurately indicates the data type and specific enhancement direction that needs to be enhanced. For example: for error samples that are mostly long and difficult sentences, multiple negative sentences, the enhancement target is to generate sentence skeletons that meet the requirements of nested layer number and negative structure, and to constrain the complexity of sentences that meet the requirements; for error samples concentrated in a specific field such as pediatrics in medicine, the enhancement target is to generate text focusing on common diseases and frequently-occurring diseases in pediatrics. These specific “enhancement targets” directly drive subsequent targeted data generation, ensuring that the enhancement operation accurately targets the ability defects of the model in a specific mode.

[0070] In the embodiment of the application, the data augmentation of the training data set according to the type of the model training task can include: if the type of the model training task belongs to a classification task, at least one of the following augmentation methods is used to augment the training data set: synonym replacement, entity replacement, local rewriting, and adversarial sample generation; if the type of the model training task belongs to a generation task, at least one of the following augmentation methods is used to augment the training data set: content expansion, sentence diversity, and reference clarification.

[0071] In the embodiment of the application, the data augmentation strategy for the training data set can be recorded by pre-constructing an enhancement strategy library. For example, the data augmentation strategy can include: synonym replacement (using a domain dictionary to replace non-key entities), entity replacement (replacing core entities while maintaining sentence structure), local rewriting (rewriting text fragments in the error set), and adversarial sample generation (adding imperceptible perturbations). For the generation task, the focus is on: content expansion (inserting relevant information at key positions), sentence diversity (changing sentence structure while maintaining semantics), and reference clarification (replacing ambiguous references with specific entities). Each strategy needs to define clear application conditions and expected effects, and set priority weights.

[0072] The enhancement strategy execution module searches the enhancement strategy library based on the vulnerability report provided by the vulnerability analysis module, enabling enhancement strategy selection and execution. This module can first optimize strategy combinations; complex vulnerabilities may require combining multiple simple strategies. Strict quality control is implemented during execution, using semantic similarity calculations to ensure the enhanced text retains its original meaning. All enhancement operations are logged in detail, including the strategies used, modifications made, and results generated, providing a basis for subsequent analysis.

[0073] Traditional training optimization processes lack closed-loop feedback mechanisms. Optimization techniques often address individual sub-problems in isolation (such as optimizing data loading or gradient calculation separately), lacking a global perspective for collaborative optimization. This fragmented optimization approach can lead to performance cancellation between different components. Most training systems lack continuous performance monitoring and optimization feedback loops. Key metrics during training (such as gradient distribution and parameter update magnitude) are not systematically collected and analyzed, resulting in missed optimization opportunities.

[0074] In this regard, the model training control method provided in this embodiment of the invention may further include: after the target computing device performs a model training task using the enhanced training dataset, obtaining a second task evaluation parameter for the target computing device performing the model training task; determining an optimized parameter for the second task evaluation parameter compared to the first task evaluation parameter when the target computing device performs the model training task using the training dataset; if the optimized parameter meets a sixth threshold range, recording the enhancement configuration parameters used in the enhanced training dataset and the scene state parameters that trigger the data enhancement task; if the optimized parameter does not meet the sixth threshold range, returning to perform data enhancement on the training dataset according to the type of the model training task, so that the target computing device performs the model training task using the enhanced training dataset.

[0075] Specifically, such as Figure 4 As shown, a multi-dimensional evaluation system is established using the effect verification module to verify the enhancement effect. At the data level, the quality indicators of the enhanced data, such as semantic consistency, language fluency, and domain relevance, are checked. At the model level, the performance changes of the model on the validation set after training with the enhanced data are monitored, with particular attention to the improvement of original weaknesses. An incremental evaluation mechanism is designed to add enhanced data to the training in batches and observe the gradient changes in model performance. A feedback adjustment mechanism is established to downweight or eliminate ineffective enhancement strategies and upweight effective strategies. The verification results are fed back to the weakness analysis module, forming a closed loop of continuous optimization.

[0076] like Figure 4As shown, by using the closed-loop optimization module, the actual effect of the enhancement strategy is evaluated quantitatively to establish a virtuous cycle of strategy optimization, such as a weak link, the weight of the enhancement strategy is increased; if the effect is insufficient, the enhancement strategy is replaced, and the original enhancement strategy is reduced in weight or eliminated. Specifically, after the diagnosis module detects a specific weakness and triggers the enhancement process, the system starts multi-dimensional effect tracking: the core indicator of effect evaluation is the weakness mitigation rate, that is, the mitigation rate of the original weakness after increasing the enhancement data training. For example, when the mitigation rate is > 60%, it is determined to be significantly effective, and the strategy-effect mapping database is recorded, recording the best strategy combination in different scenarios. When the system detects similar problems again, the intelligent strategy selector will preferentially call the strategy combination with excellent historical performance.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0078] The embodiment of the application also provides a model training control device, comprising: a real-time monitoring module, configured to acquire intermediate calculation results and running state parameters of a target computing device when the target computing device executes a model training task; a fault diagnosis module, configured to perform abnormality analysis on an execution state of the target computing device executing the model training task according to the intermediate calculation results and the running state parameters, and obtain an abnormality analysis result; and a self-recovery strategy selection module, configured to call corresponding self-recovery control parameters of the model training task according to the abnormality analysis result, and adjust configuration of the target computing device executing the model training task according to the self-recovery control parameters; wherein the intermediate calculation results comprise at least one of gradient data and model learning loss values.

[0079] In the embodiment of the application, the fault diagnosis module performs abnormality analysis on the execution state of the target computing device executing the model training task according to the intermediate calculation results and the running state parameters, and obtains an abnormality analysis result, which can include: acquiring, from an abnormal event rule library, a first abnormal data range corresponding to the intermediate calculation results and a second abnormal data range corresponding to the running state parameters; calculating quality evaluation parameters of the intermediate calculation results; and if at least one of the quality evaluation parameters meets the first abnormal data range and the running state parameters meets the second abnormal data range, it is determined that an abnormal event occurs in the process of the target computing device executing the model training task.

[0080] In the embodiment of the present application, the real-time monitoring module obtains the intermediate calculation result, which can include: obtaining the to-be-detected gradient data obtained by the target computing device performing the model training task after every first number of training steps; the fault diagnosis module calculates the quality evaluation parameter of the intermediate calculation result, which can include: calculating the gradient norm of the to-be-detected gradient data; the quality evaluation parameter satisfying the first abnormal data range can include: the gradient norm of the to-be-detected gradient data being greater than a first threshold.

[0081] In the embodiment of the present application, the real-time monitoring module obtains the intermediate calculation result, which can include: obtaining the model learning loss value calculated by the target computing device after the target computing device completes a training step on one batch of data; the fault diagnosis module calculates the quality evaluation parameter of the intermediate calculation result, which can include: calculating a first proportion of non-numerical data in the model learning loss value; the quality evaluation parameter satisfying the first abnormal data range can include: the first proportion being greater than a second threshold.

[0082] In the embodiment of the present application, the real-time monitoring module obtains the running state parameter of the target computing device, which can include: obtaining the memory occupation parameter allocated by the target computing device to the model training task and the change rate of the memory occupation parameter; the running state parameter satisfying the second abnormal data range can include: if at least one of the memory occupation parameter being greater than a third threshold and the change rate of the memory occupation parameter being greater than a fourth threshold is satisfied, it is determined that the running state parameter satisfies the second abnormal data range.

[0083] In the embodiment of the present application, the fault diagnosis module performs abnormal analysis on the execution state of the target computing device performing the model training task according to the intermediate calculation result and the running state parameter, and obtains an abnormal analysis result, which can include: inputting the intermediate calculation result and the running state parameter into the pre-trained fault prediction model to output an abnormal event prediction result and a confidence score of the abnormal event prediction result; taking the abnormal event prediction result and the confidence score as the abnormal analysis result; the self-recovery strategy selection module calls the corresponding self-recovery control parameter for the model training task according to the abnormal analysis result, which can include: if the abnormal event prediction result indicates that there is an abnormal event and the confidence score is greater than a fifth threshold, the corresponding self-recovery control parameter is called according to the type of the abnormal event included in the abnormal event prediction result.

[0084] In the embodiment of the present application, the fault diagnosis module inputs the intermediate calculation result and the running state parameter into the pre-trained fault prediction model to output an abnormal event prediction result and a confidence score of the abnormal event prediction result, which can include: if the intermediate calculation result and the running state parameter do not satisfy the abnormal data range recorded in the abnormal event rule library, the intermediate calculation result and the running state parameter are input into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score.

[0085] In the embodiment of the present application, the self-recovery strategy selection module calls the corresponding self-recovery control parameters for the model training task according to the abnormality analysis result, which can include: if the fault root cause of the abnormality analysis result is training sample data abnormality, the model training task of the target computing device is controlled to roll back to the historical checkpoint when the last abnormality analysis result is normal, and the training sample data and the corresponding intermediate calculation results used between the historical checkpoint and the current checkpoint are moved to the target isolated storage area; if the fault root cause of the abnormality analysis result is hyperparameter abnormality, the learning rate of the target computing device executing the model training task is adjusted in an exponential decay manner; if the fault root cause of the abnormality analysis result is hardware abnormality, the model training task is migrated to another computing device for execution.

[0086] The model training control device provided by the embodiment of the present application can further include a directional enhancement module for performing data enhancement on the training data set according to the type of the model training task, so that the target computing device executes the model training task by using the enhanced training data set.

[0087] In the embodiment of the present application, the directional enhancement module performs data enhancement on the training data set according to the type of the model training task, so that the target computing device executes the next subtask in the model training task by using the enhanced training data set, which can include: when the abnormality analysis result is intermediate calculation result abnormality, the directional enhancement module performs data enhancement on the training data set according to the type of the model training task, so that the target computing device executes the next subtask in the model training task by using the enhanced training data set.

[0088] In the embodiment of the present application, the directional enhancement module performs data enhancement on the training data set according to the type of the model training task, which can include: if the type of the model training task belongs to a classification task, at least one of synonym replacement, entity replacement, local rewriting and adversarial sample generation is used to enhance the training data set; if the type of the model training task belongs to a generation task, at least one of content expansion, sentence diversity and reference clarification is used to enhance the training data set.

[0089] In the embodiment of the present application, the directional enhancement module can also be used to obtain a second task evaluation parameter of the target computing device performing the model training task after the target computing device performs the model training task using the enhanced training data set; determine an optimization parameter of the second task evaluation parameter compared with a first task evaluation parameter of the target computing device performing the model training task using the training data set; if the optimization parameter meets a sixth threshold range, record the enhanced configuration parameter used by the enhanced training data set and the scene state parameter triggering the data enhancement task; if the optimization parameter does not meet the sixth threshold range, return to performing data enhancement on the training data set according to the type of the model training task, so that the target computing device performs the model training task using the enhanced training data set.

[0090] The features of the embodiments corresponding to the model training control device can be referred to the related description of the embodiments corresponding to the model training control method, which will not be repeated here.

[0091] The embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above model training control method embodiments.

[0092] The embodiments of the present application also provide a non-volatile storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above model training control method embodiments when running.

[0093] In an exemplary embodiment, the above-mentioned non-volatile storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0094] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above model training control method embodiments.

[0095] The embodiments of the present application also provide another computer program product, which comprises a non-volatile storage medium, the non-volatile storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above model training control method embodiments.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0097] The above provides a detailed description of the model training control method and electronic device provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A model training control method, characterized by comprising: The method comprises: obtaining intermediate calculation results and running state parameters of a target computing device when the target computing device performs a model training task; performing abnormality analysis on an execution state of the target computing device performing the model training task according to the intermediate calculation results and the running state parameters, to obtain an abnormality analysis result; calling corresponding self-recovery control parameters for the model training task according to the abnormality analysis result; adjusting the configuration of the target computing device performing the model training task according to the self-recovery control parameters; wherein the intermediate calculation results comprise at least one of gradient data and model learning loss values; performing abnormality analysis on an execution state of the target computing device performing the model training task according to the intermediate calculation results and the running state parameters, to obtain an abnormality analysis result, comprises: obtaining a first abnormal data range corresponding to the intermediate calculation results and a second abnormal data range corresponding to the running state parameters from an abnormal event rule library; calculating a quality evaluation parameter of the intermediate calculation results; if at least one of the quality evaluation parameter meets the first abnormal data range and the running state parameter meets the second abnormal data range, it is determined that an abnormal event occurs in the process of the target computing device performing the model training task; obtaining the intermediate calculation results comprises: obtaining gradient data to be detected obtained by the target computing device performing the model training task every first number of training steps; calculating the quality evaluation parameter of the intermediate calculation results comprises: calculating a gradient norm of the gradient data to be detected; the quality evaluation parameter meeting the first abnormal data range comprises: the gradient norm of the gradient data to be detected being greater than a first threshold value; obtaining the intermediate calculation results further comprises: obtaining a model learning loss value calculated by the target computing device after the target computing device completes a training step on a batch of data; calculating the quality evaluation parameter of the intermediate calculation results further comprises: calculating a first proportion of non-numerical data in the model learning loss value; the quality evaluation parameter meeting the first abnormal data range further comprises: the first proportion being greater than a second threshold value. 2.The model training control method of claim 1, wherein, obtaining the running state parameters of the target computing device comprises: obtaining a memory occupation parameter allocated to the model training task by the target computing device and a change rate of the memory occupation parameter; the running state parameter meeting the second abnormal data range comprises: if at least one of the memory occupation parameter being greater than a third threshold value and the change rate of the memory occupation parameter being greater than a fourth threshold value is met, it is determined that the running state parameter meets the second abnormal data range. 3.The model training control method of claim 1, wherein, performing abnormality analysis on an execution state of the target computing device performing the model training task according to the intermediate calculation results and the running state parameters, to obtain an abnormality analysis result, comprises: inputting the intermediate calculation results and the running state parameters into a pre-trained fault prediction model to output an abnormal event prediction result and a confidence score of the abnormal event prediction result; The abnormal event prediction result and the confidence score are taken as the abnormal analysis result; According to the abnormal analysis result, a corresponding self-healing control parameter for the model training task is called, including: If the abnormal event prediction result indicates that there is an abnormal event and the confidence score is greater than a fifth threshold value, a corresponding self-healing control parameter is called according to the type of abnormal event included in the abnormal event prediction result. 4.The model training control method of claim 3, wherein, The intermediate calculation result and the running state parameter are input into a pre-trained fault prediction model to output an abnormal event prediction result and a confidence score of the abnormal event prediction result, including: If the intermediate calculation result and the running state parameter do not satisfy the abnormal data range recorded in the abnormal event rule library, the intermediate calculation result and the running state parameter are input into the pre-trained fault prediction model to output the abnormal event prediction result and the confidence score. 5.The model training control method of claim 1, wherein, According to the abnormal analysis result, a corresponding self-healing control parameter for the model training task is called, including: If the fault root cause of the abnormal analysis result is training sample data abnormality, the model training task of the target computing device is controlled to roll back to a historical checkpoint when the abnormal analysis result is normal last time, and the training sample data and the corresponding intermediate calculation result used between the historical checkpoint and the current checkpoint are moved to a target isolated storage area; If the fault root cause of the abnormal analysis result is hyperparameter abnormality, an exponential decay method is used to adjust the learning rate of the target computing device executing the model training task; If the fault root cause of the abnormal analysis result is hardware abnormality, the model training task is migrated to another computing device for execution. 6.The model training control method of claim 1, wherein, Further comprising: According to the type of the model training task, data augmentation is performed on a training data set, so that the target computing device executes the model training task by using the augmented training data set. 7.The model training control method of claim 6, wherein, According to the type of the model training task, data augmentation is performed on a training data set, so that the target computing device executes the model training task by using the augmented training data set, including: When the abnormal analysis result is the intermediate calculation result abnormality, according to the type of the model training task, data augmentation is performed on a training data set, so that the target computing device executes the next subtask in the model training task by using the augmented training data set. 8.The model training control method of claim 6, wherein, According to the type of the model training task, data augmentation is performed on a training data set, including: If the type of the model training task belongs to a classification task, at least one of synonym replacement, entity replacement, local rewriting, and adversarial sample generation is used to enhance the training data set; If the type of the model training task belongs to a generation task, at least one of content expansion, sentence diversity, and reference clarification is used to enhance the training data set. 9.The model training control method of claim 6, wherein, Further comprising: After the target computing device executes the model training task by using the augmented training data set, a second task evaluation parameter of the target computing device executing the model training task is obtained; determining an optimization parameter of the second task evaluation parameter compared to a first task evaluation parameter when the target computing device adopts the training data set to perform the model training task; if the optimization parameter meets a sixth threshold range, recording an enhanced configuration parameter adopted by the enhanced training data set and a scene state parameter triggering the data enhancement task; if the optimization parameter does not meet the sixth threshold range, returning to performing data enhancement on the training data set according to the type of the model training task, so that the target computing device adopts the enhanced training data set to perform the model training task.

10. An electronic device, comprising: comprising: a memory for storing a computer program; a processor for implementing the steps of the model training control method according to any one of claims 1 to 9 when executing the computer program.

Images