A model training system configuration adjustment method

Abstract

The application provides a model training system configuration adjustment method, and relates to the field of artificial intelligence. First, the system performance parameters of the model training system are obtained; the system performance parameters include configuration parameters, actual computing power values and hardware performance indicators corresponding to historical training tasks. Based on the first prompt word engineering, the system performance parameters are input into the first specified large model to obtain the target configuration parameters after the adjustment of the configuration parameters to be optimized. The target configuration parameters are sorted according to the hardware performance indicators to obtain the target configuration sequence. According to the target configuration sequence, the target training strategy for training the target model is generated. The full-process automatic adjustment of the configuration parameters of the model training system is realized, the optimization efficiency and optimization effect of the configuration parameters are improved, and then the training efficiency and resource utilization rate of the model training system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a configuration adjustment method of a model training system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, model training business is becoming more and more complex and changeable. In order to adapt to the complex and changeable model training task, the performance of the model training system needs to be optimized. At present, the performance optimization method of the model training system is mainly manual tuning, that is, using manual experience to manually adjust the configuration parameters of the model training system, such as training batch parameters and execution strategy. However, the manual tuning method has the problems of time-consuming and labor-consuming and poor optimization effect.

[0003] Therefore, it is necessary to provide a configuration adjustment method of a model training system, which realizes automatic adjustment of configuration parameters of the model training system, improves the performance optimization effect, and further improves the training efficiency and resource utilization rate of the model training system. SUMMARY

[0004] Therefore, the present application provides a configuration adjustment method of a model training system, which comprises:

[0005] obtaining system performance parameters of the model training system; the system performance parameters include configuration parameters, actual computing power values and hardware performance indicators corresponding to historical training tasks;

[0006] based on a first prompt word engineering constructed in advance, inputting the system performance parameters into a first designated large model to obtain target configuration parameters after adjusting the configuration parameters to be optimized;

[0007] sorting the target configuration parameters according to the hardware performance indicators to obtain a target configuration sequence;

[0008] generating a target training strategy for training a target model according to the target configuration sequence; the target model is the training object of the model training system.

[0009] Optionally, the constraint conditions in the first prompt word engineering include:

[0010] determining an influence score corresponding to the configuration parameter according to the system performance parameters; the influence score is used to represent the influence degree of the configuration parameter on the actual computing power value; the influence score is positively correlated with the influence degree;

[0011] the configuration parameter with an influence score greater than a first preset score is taken as the configuration parameter to be optimized;

[0012] adjusting the configuration parameter to be optimized to obtain the target configuration parameter.

[0013] Optionally, the sorting of the target configuration parameters according to the hardware performance indicators to obtain a target configuration sequence comprises:

[0014] obtaining a preset bottleneck indicator range corresponding to each of the hardware performance indicators and a mapping relationship; the mapping relationship is a corresponding relationship between a bottleneck performance and a sorting priority;

[0015] taking an indicator in the hardware performance indicators within the corresponding bottleneck indicator range as a target bottleneck performance;

[0016] determining a target sorting priority corresponding to the target bottleneck performance based on the mapping relationship;

[0017] sorting each of the target configuration parameters based on the target sorting priority to obtain the target configuration sequence; the higher the sorting priority of the target configuration parameter, the higher the degree of improvement of the target configuration parameter on the computing power value of the model training system.

[0018] Optionally, the generating of a target training strategy for training a target model according to the target configuration sequence comprises:

[0019] inputting the target configuration sequence into a second specified large model based on a second prompt word engineering to obtain the target training strategy.

[0020] Optionally, the system performance parameters further include a historical training strategy; a historical computing power value corresponding to the historical training strategy is a computing power value determined after the model training system trains the target model based on the historical training strategy and the target model converges;

[0021] the constraint condition in the second prompt word engineering comprises:

[0022] taking a latest historical training strategy in which the historical computing power value is greater than a preset value as a target historical training strategy;

[0023] taking a preset number of configuration parameters in the target configuration sequence as first configuration parameters; and taking other configuration parameters in the configuration parameters except the first configuration parameters as to-be-adjusted configuration parameters;

[0024] adjusting the to-be-adjusted configuration parameters based on configuration parameters in the target historical training strategy to obtain second configuration parameters;

[0025] combining the first configuration parameters and the second configuration parameters to obtain the target training strategy.

[0026] Optionally, the constraint condition in the second prompt word engineering further comprises:

[0027] determining whether there is a same strategy as the target training strategy in the historical training strategies;

[0028] if not, taking the target training strategy as the final target training strategy.

[0029] Optionally, the method further comprises:

[0030] after the model training system trains the target model based on the target training strategy, obtaining a training result;

[0031] if the training result represents model convergence, determining an actual computing power value of the model training system; storing the target training strategy and the actual computing power value;

[0032] if the training result represents that the model has not converged, the following steps are performed:

[0033] obtaining an error log generated by the model training system, and a third prompt word engineering constructed in advance;

[0034] based on the third prompt word engineering, inputting the error log into a third specified large model to obtain a fault cause.

[0035] Optionally, the constraint condition in the third prompt word engineering comprises:

[0036] determining an initial fault cause based on the error log; the initial fault cause comprises at least one of a configuration parameter setting fault, a hardware fault, a communication fault, and a timeout fault;

[0037] determining a confidence score of the initial fault cause; the confidence score is used to represent the confidence of the initial fault cause; the confidence score is positively correlated with the confidence;

[0038] taking the initial fault cause with a confidence score greater than a second preset score as the fault cause.

[0039] Optionally, the constraint condition in the third prompt word engineering further comprises:

[0040] if the fault cause is the timeout fault, determining a root fault corresponding to the timeout fault; the root fault comprises at least one of the configuration parameter setting fault, the hardware fault, and the communication fault;

[0041] generating a fault repair suggestion for the root fault.

[0042] Optionally, after the system performance parameter of the model training system is acquired, the method further comprises:

[0043] acquiring a nominal computing power value of the model training system;

[0044] determining whether the actual computing power value is less than the nominal computing power value;

[0045] If the actual computing power value is less than the nominal computing power value, the step of inputting the system performance parameter into the first designated large model based on the pre-constructed first prompt word engineering is entered, and subsequent steps are executed.

[0046] In summary, the present application provides a configuration adjustment method of a model training system. First, the system performance parameter of the model training system is acquired; the system performance parameter includes the configuration parameter, the actual computing power value and the hardware performance index corresponding to the historical training task. Based on the first prompt word engineering, the system performance parameter is input into the first designated large model to obtain the target configuration parameter after the configuration parameter to be optimized is adjusted. The target configuration parameter is sorted according to the hardware performance index to obtain the target configuration sequence. According to the target configuration sequence, the target training strategy for training the target model is generated. The full-process automatic adjustment of the configuration parameter of the model training system is realized, the optimization efficiency and optimization effect of the configuration parameter are improved, and the training efficiency and resource utilization rate of the model training system are improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 a first flowchart of a configuration adjustment method of a model training system provided by the present application;

[0048] Figure 2 a first principle diagram of a configuration adjustment method of a model training system provided by the present application;

[0049] Figure 3 a second flowchart of a configuration adjustment method of a model training system provided by the present application;

[0050] Figure 4 a second principle diagram of a configuration adjustment method of a model training system provided by the present application;

[0051] Figure 5 a whole flowchart of a configuration adjustment method of a model training system provided by the present application;

[0052] Figure 6 a fault diagnosis flowchart of a configuration adjustment method of a model training system provided by the present application. DETAILED DESCRIPTION

[0053] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0054] Reference is made to Figure 1 , Figure 1 A first flowchart of a configuration adjustment method of a model training system provided by the present application is provided. The method comprises:

[0055] S1: Obtain system performance parameters of the model training system. The system performance parameters comprise configuration parameters, actual computing power values, and hardware performance indicators corresponding to historical training tasks.

[0056] The model training system described above includes various forms of training systems such as distributed training systems and single-machine training systems, which are not particularly limited by the present application. If the model training system is a distributed training system, the method provided by the present application can be deployed on a target agent in the distributed training system.

[0057] The present application first obtains the system performance parameters of the model training system, which include configuration parameters, actual computing power values, and hardware performance indicators.

[0058] The configuration parameters described above include training batch parameters and execution strategies, etc. The training batch parameters include but are not limited to sequence length, global batch size, and micro batch size. The execution strategies include but are not limited to tensor parallelism, pipeline parallelism, data parallelism, and expert parallelism. The configuration parameters can also include tensor parallelism degree, pipeline parallelism degree, data parallelism degree, and expert parallelism degree corresponding to the execution strategies, etc.

[0059] The actual computing power value described above refers to the actual computing power provided by the model training system when executing a training task. The actual computing power value can be determined according to the total number of floating point operations and the actual execution duration during the execution of the training task by the model training system, which is not described herein.

[0060] The hardware performance indicators described above include video memory performance indicators, graphics card performance indicators, GPU (Graphics Processing Unit) utilization, network bandwidth, etc. Among them, the video memory performance indicators include video memory occupancy fluctuation curve, video memory occupancy average value, video memory occupancy peak value, and video memory theoretical upper limit value, etc.; the graphics card performance indicators include graphics card temperature and power consumption curve, etc. In addition, for the case of deploying the method provided by the present application on a target agent in a distributed training system, the hardware performance indicators described above can be the hardware performance indicators of the target agent.

[0061] In addition, it should be noted that after the model training system performs each training task, the configuration parameters used in this training task, the actual computing power value provided, and the corresponding hardware performance indicators are stored in the global message pool as the system performance parameters corresponding to this training task. Based on this, the application can obtain the system performance parameters from the global message pool. The obtained system performance parameters include not only the system performance parameters corresponding to the current training task, but also the performance parameters corresponding to the historical training tasks, so as to improve the accuracy of subsequent determination of the to-be-optimized configuration parameters and the corresponding target configuration parameters.

[0062] S2: inputting the system performance parameters into the first specified large model based on the pre-constructed first prompt word engineering to obtain the target configuration parameters after adjusting the to-be-optimized configuration parameters.

[0063] The first specified large model in the application is a pre-trained large model; the first prompt word engineering is a pre-constructed prompt word engineering for guiding the first specified large model to determine the target configuration parameters. Based on the first prompt word engineering, after the system performance parameters are input into the first specified large model, the first specified large model will first determine the to-be-optimized configuration parameters that need to be optimized in the configuration parameters based on the system performance parameters, and then adjust the to-be-optimized configuration parameters to obtain the target configuration parameters. The training process of the first specified large model in the application is not particularly limited. The subsequent embodiments will describe the first prompt word engineering, which will not be described here.

[0064] S3: sorting the target configuration parameters according to the hardware performance indicators to obtain a target configuration sequence.

[0065] It is considered that the hardware performance indicators can reflect the bottleneck performance of the model training system. When the bottleneck performance of the model training system is different, the promotion degree of the target configuration parameters to the computing power value of the model training system is different. For example, when the bottleneck performance of the model training system is the display memory bottleneck, the promotion degree of the global batch size to the computing power value is the highest, and the performance optimization effect of the model training system is the most significant.

[0066] Therefore, the application sorts the target configuration parameters according to the hardware performance indicators to obtain a target configuration sequence. The target configuration sequence reflects the promotion degree of each target configuration parameter to the computing power value, and provides a reference for subsequent generation of a target training strategy based on the target configuration sequence. The subsequent embodiments will describe the process of obtaining the target configuration sequence, which will not be described here.

[0067] S4: generating a target training strategy for training a target model according to the target configuration sequence; the target model is the training object of the model training system.

[0068] After obtaining the target configuration sequence, a target training strategy for training the target model is generated according to the target configuration sequence. The target training strategy contains various configuration parameters required by the model training system to train the target model. Subsequent embodiments will describe the process of generating the target training strategy, which will not be described here.

[0069] In addition, after obtaining the target training strategy, the target training strategy can be converted into a structured target training configuration by using a predefined regular expression template and a semantic parser, so as to execute the target training configuration and automatically adjust the configuration parameters of the model training system.

[0070] Therefore, the configuration adjustment method of the model training system provided by the embodiments of the present application can determine the to-be-optimized configuration parameters of the model training system and the corresponding target configuration parameters based on the system performance parameters, generate a target training strategy, realize full-process automatic adjustment of the configuration parameters of the model training system, improve the optimization efficiency and optimization effect of the configuration parameters, and further improve the training efficiency and resource utilization rate of the model training system.

[0071] On the basis of the above embodiments:

[0072] As described above, the present application guides the first specified large model to determine the to-be-optimized configuration parameters to be optimized from the configuration parameters and adjusts the to-be-optimized configuration parameters to obtain the target configuration parameters by using the constraint condition in the first prompt word engineering. The constraint condition in the first prompt word engineering will be described in detail below.

[0073] As an optional embodiment, the constraint condition in the first prompt word engineering includes:

[0074] According to the system performance parameters, the influence score corresponding to the configuration parameter is determined; the influence score is used to represent the influence degree of the configuration parameter on the actual computing power value; the influence score is positively correlated with the influence degree;

[0075] The configuration parameter with an influence score greater than a first preset score is taken as the to-be-optimized configuration parameter;

[0076] The to-be-optimized configuration parameter is adjusted to obtain the target configuration parameter.

[0077] Please refer to Figure 2 , Figure 2 is a first principle schematic diagram of a configuration adjustment method of a model training system provided by the present application.

[0078] To ensure the effectiveness of the configuration parameter adjustment, the configuration parameters that actually affect the actual computing power value of the model training system need to be optimized and adjusted. Therefore, in the embodiment, the constraint condition in the first prompt word engineering includes determining an influence score corresponding to the configuration parameter according to the system performance parameter. As described above, the system performance parameter includes the system performance parameter corresponding to the current training task and the performance parameter corresponding to the historical training task. By comparing and analyzing the configuration parameters and the actual computing power value corresponding to the historical training task, the influence score corresponding to the configuration parameter can be determined. The specific process of determining the influence score is not particularly limited in the embodiment.

[0079] The influence score represents the influence degree of the configuration parameter on the actual computing power value, and the influence score is positively correlated with the influence degree, that is, the higher the influence score corresponding to the configuration parameter, the higher the promotion degree of the configuration parameter on the actual computing power value.

[0080] Based on this, the constraint condition in the first prompt word engineering in the embodiment further includes taking the configuration parameter with an influence score greater than a first preset score as a configuration parameter to be optimized, so as to take the configuration parameter that actually affects the actual computing power value of the model training system as the configuration parameter to be optimized. The first preset score can be set according to actual needs, and the embodiment is not particularly limited thereto.

[0081] Further, the constraint condition in the first prompt word engineering further includes adjusting the configuration parameter to be optimized to obtain a target configuration parameter, so as to guide the first specified large model to adjust the configuration parameter to be optimized to obtain the target configuration parameter.

[0082] In summary, the constraint condition for constructing the first prompt word engineering based on the thinking chain technology in the embodiment divides the task of determining the target configuration parameter into multiple steps, including determining the influence score of the configuration parameter, determining the configuration parameter to be optimized according to the influence score, and adjusting the configuration parameter to be optimized to obtain the target configuration parameter.

[0083] In addition, the first specified large model can be constrained to first show each intermediate step and then give the final target configuration parameter. Through this "thinking before answering" mode, the first specified large model can use the context information for self-checking, further improving the accuracy and interpretability of the obtained target configuration parameter.

[0084] Please refer to Figure 3 , Figure 3 for a second flowchart of a configuration adjustment method of a model training system provided by the present application. The process of determining the target configuration sequence will be described below.

[0085] As an optional embodiment, the target configuration parameters are sorted according to the hardware performance indicators to obtain the target configuration sequence, including:

[0086] S301, acquire a pre-set bottleneck index range corresponding to each hardware performance index and a mapping relationship; the mapping relationship is a corresponding relationship between bottleneck performance and ranking priority;

[0087] S302, take the index in the hardware performance index within the corresponding bottleneck index range as the target bottleneck performance;

[0088] S303, determine the target ranking priority corresponding to the target bottleneck performance based on the mapping relationship;

[0089] S304, rank each target configuration parameter based on the target ranking priority to obtain a target configuration sequence; the higher the ranking priority of the target configuration parameter, the higher the degree of improvement of the model training system's computing power value by the target configuration parameter.

[0090] As described above, the hardware performance index can represent the bottleneck performance of the model training system, and the degree of improvement of the model training system's computing power value by the target configuration parameter is different when the bottleneck performance of the model training system is different. Therefore, the target bottleneck performance needs to be determined based on the hardware performance index, and then the ranking priority of each target configuration parameter is determined for the target bottleneck performance, so as to rank the target configuration parameter and obtain the target configuration sequence.

[0091] To achieve the above purpose, in the embodiment, the bottleneck index range corresponding to each hardware performance index is pre-set, and the value of the bottleneck index range can be set according to actual needs, which is not particularly limited in this embodiment. If the hardware performance index is within the corresponding bottleneck index range, the hardware performance index is taken as the target bottleneck performance. For example, a pre-set memory mean value index range is used to represent a low memory occupancy rate, and if the memory occupancy rate mean value is within the memory mean value index range, the memory bottleneck is taken as the target bottleneck performance.

[0092] In the embodiment, a mapping relationship is also pre-set, which is a corresponding relationship between bottleneck performance and ranking priority. The degree of improvement of the model training system's computing power value by each target configuration parameter is different when the bottleneck performance is different, so the ranking priority of each target configuration parameter is different. Taking the memory bottleneck as an example, the ranking priority of the global batch size is the highest in the ranking priority corresponding to the memory bottleneck, the ranking priority of the micro batch size is the second, and the ranking priority of other configuration parameters is not described in this embodiment, which can be set according to actual experience.

[0093] Based on this, after determining the target bottleneck performance, the target ranking priority corresponding to the target bottleneck performance is determined based on the mapping relationship, and each target configuration parameter is ranked according to the target ranking priority to obtain a target configuration sequence. The target configuration sequence can reflect the degree of improvement of each target configuration parameter on the computing power value of the model training system. Taking the graphics memory bottleneck as an example, the target configuration parameter with the highest ranking priority in the target configuration sequence is the global batch size, which reflects that under the graphics memory bottleneck, the optimization potential of the global batch size is the highest, that is, improving the global batch size has the highest degree of improvement on the computing power value of the model training system, and can most significantly improve the performance of the model training system.

[0094] In summary, based on the pre-set bottleneck index range and mapping relationship, the degree of improvement of the target configuration parameter on the computing power value of the model training system is taken as the basis of the ranking priority, each target configuration parameter is ranked, and a target configuration sequence is obtained, which provides targeted guidance for subsequent generation of a target training strategy.

[0095] As an optional embodiment, a target training strategy for training a target model is generated according to the target configuration sequence, including:

[0096] Based on the pre-constructed second prompt word engineering, the target configuration sequence is input into the second specified large model to obtain the target training strategy.

[0097] In this embodiment, the pre-constructed second prompt word engineering is used to guide the pre-trained second specified large model to automatically generate a target training strategy based on a target configuration sequence, reducing the amount of manual work and improving the optimization efficiency and optimization effect of the configuration parameter. The training process of the second specified large model is not particularly limited in this embodiment.

[0098] Please refer to Figure 4 , Figure 4 is a second principle schematic diagram of a configuration adjustment method of a model training system provided by the present application. The constraint conditions in the second prompt word engineering will be described below.

[0099] As an optional embodiment, the system performance parameter further includes a historical training strategy; the historical computing power value corresponding to the historical training strategy is a computing power value determined by the model training system based on the historical training strategy when training the target model and after the target model converges;

[0100] The constraint conditions in the second prompt word engineering include:

[0101] The latest historical training strategy in the historical training strategy with a historical computing power value greater than a preset value is taken as a target historical training strategy;

[0102] The first preset number of configuration parameters in the target configuration sequence are taken as first configuration parameters, and other configuration parameters in the configuration parameters except the first configuration parameters are taken as to-be-adjusted configuration parameters;

[0103] The to-be-adjusted configuration parameters are adjusted based on the configuration parameters in the target historical training strategy to obtain second configuration parameters;

[0104] The first configuration parameters and the second configuration parameters are combined to obtain a target training strategy.

[0105] In the embodiment, first, it is considered that the target configuration sequence is only a configuration parameter sequence obtained after the to-be-optimized configuration parameters are adjusted, and the target training strategy for training the target model also needs to include other necessary configuration parameters, therefore, the target training strategy is generated by comprehensively combining the target configuration sequence and the historical training strategy.

[0106] Furthermore, if the complete target configuration sequence is directly combined with the historical training strategy to generate the target training strategy, there may be a case that the target training strategy is consistent with the historical training strategy, in which case the corresponding computing power values are also consistent, the target training strategy cannot further improve the performance of the model training system, and belongs to an invalid training strategy. In order to avoid the above-mentioned case, in the embodiment, part of the configuration parameters in the target configuration sequence are selected as the first configuration parameters, the second configuration parameters are determined based on the historical training strategy, and the first configuration parameters and the second configuration parameters are combined to obtain the target training strategy, which can reduce the possibility of repetition of the target training strategy and the historical training strategy.

[0107] According to the above idea, the constraint condition in the second prompt word engineering is constructed, which is described in detail as follows.

[0108] First of all, it needs to be explained that in the embodiment, the system performance parameters also include the historical training strategy, and when the model training system trains the target model based on the historical training strategy, the target model can converge, that is, the historical training strategy is an effective training strategy, so as to ensure the effectiveness of the target training strategy determined based on the historical training strategy. On this basis, the historical computing power value corresponding to the historical training strategy is the computing power value determined when the model training system trains the target model based on the historical training strategy and after the target model converges.

[0109] The constraint condition of the second prompt word engineering includes taking the latest historical training strategy with a historical computing power value greater than a preset value in the historical training strategy as a target historical training strategy. In this way, it is ensured that the target training strategy determined based on the target historical training strategy also corresponds to a higher computing power value, thereby ensuring the performance optimization effect of the model training system. In addition, the value of the preset value is not particularly limited in the present application, and can be set according to actual needs.

[0110] The constraint condition of the second prompt word engineering further includes taking a preset number of configuration parameters in the target configuration sequence, that is, taking a preset number of configuration parameters with high ranking priority in the target configuration sequence as the first configuration parameter, which is equivalent to selecting a configuration parameter with a higher degree of improvement of the computing power value of the model training system as the first configuration parameter. In this way, the possibility of repetition of the target training strategy and the historical training strategy can be reduced, and the performance optimization effect of the model training system can be ensured.

[0111] Further, the configuration parameters other than the first configuration parameter are taken as the to-be-adjusted configuration parameters. The to-be-adjusted configuration parameters are adjusted based on the target historical training strategy to obtain the second configuration parameter. For example, a configuration parameter mapping table is stored in advance, and all necessary configuration parameters are stored in the configuration parameter mapping table. By comparing the configuration parameter mapping table and the first configuration parameter, the to-be-adjusted configuration parameter can be determined. Then, the historical configuration parameter corresponding to the to-be-adjusted configuration parameter in the target historical training strategy is taken as the second configuration parameter.

[0112] Finally, the first configuration parameter and the second configuration parameter are combined to obtain the target training strategy.

[0113] At this point, based on the constraint condition in the above-mentioned second prompt word engineering, the second specified large model can automatically generate the target training strategy, and the possibility of repetition of the target training strategy and the historical training strategy can be reduced, and the effectiveness of the target training strategy can be improved.

[0114] In addition, as an optional embodiment, the constraint condition in the second prompt word engineering further includes determining whether there is a same strategy as the target training strategy in the historical training strategy; if not, the target training strategy is taken as the final target training strategy to further ensure the effectiveness of the target training strategy through the deduplication verification.

[0115] For the case where there is a same strategy as the target training strategy in the historical training strategy, the configuration parameter adjustment process can be ended, or the target configuration parameter can be regenerated, which is not particularly limited in this embodiment.

[0116] Please refer to Figure 5 , Figure 5 The present application further verifies the target training strategy, and the verification process is described in detail below.

[0117] As an optional embodiment, the method further includes:

[0118] After the model training system trains the target model based on the target training strategy, the training result is obtained.

[0119] If the training result indicates that the model converges, the actual computing power value of the model training system is determined, and the target training strategy and the actual computing power value are stored.

[0120] If the training result indicates that the model does not converge, the following steps are performed:

[0121] Error logs generated by the model training system are obtained, and a third prompt word engineering is constructed in advance.

[0122] Based on the third prompt word engineering, the error logs are input into a third specified large model to obtain the fault reason.

[0123] In this embodiment, after the model training system trains the target model based on the target training strategy, the training result is obtained to analyze the effectiveness of the target training strategy based on the training result. If the training result indicates that the model converges, the target training strategy is considered to be an effective training strategy. If the training result indicates that the model does not converge, the target training strategy is considered to be an ineffective training strategy.

[0124] As shown in Figure 5 , for the case where the training result indicates that the model converges, the actual computing power value of the model training system is further determined, and the target training strategy and the corresponding actual computing power value are stored. For example, the target training strategy and the corresponding actual computing power value are stored in a global message pool. On this basis, before the model training system starts to perform the next training task, the training strategy with the highest corresponding actual computing power value can be selected from the global message pool as the training strategy used for this training task, to ensure the performance of the model training system. In addition, as shown in Figure 5 , for the special case where the global message pool is empty, a set of target training strategies can be randomly initialized.

[0125] It should be noted that for the case where the model training system is a distributed training system, in addition to the target agent, other agents can also determine the training strategy used for the next training task by reading the training strategies and the corresponding actual computing power values stored in the global message pool.

[0126] In addition, as shown in Figure 5 , the hardware performance indicators during the training of the target model by the model training system can also be stored in the global message pool together with the target training strategy and the corresponding actual computing power value, to increase the amount of data of the system performance parameters in the global message pool and improve the accuracy of subsequent analysis of the to-be-optimized configuration parameters and determination of the target configuration parameters.

[0127] For the case where the training result indicates that the model does not converge, the fault reason causing the model not to converge needs to be determined. Please refer to Figure 6 , Figure 6A fault diagnosis flowchart of a configuration adjustment method of a model training system provided in the present application is shown. In this embodiment, first, error logs generated by the model training system are obtained, and then the error logs are input into a third specified large model trained in advance based on a third prompt word engineering to obtain a fault cause, thereby realizing automatic fault diagnosis.

[0128] The constraint conditions in the third prompt word engineering are described below.

[0129] As an optional embodiment, the constraint conditions in the third prompt word engineering include:

[0130] An initial fault cause is determined based on the error logs; the initial fault cause includes at least one of a configuration parameter setting fault, a hardware fault, a communication fault, and a timeout fault;

[0131] A confidence score of the initial fault cause is determined; the confidence score is used to represent the confidence of the initial fault cause; the confidence score is positively correlated with the confidence;

[0132] The initial fault cause with a confidence score greater than a second preset score is taken as the fault cause.

[0133] In this embodiment, the constraint conditions in the third prompt word engineering first include determining an initial fault cause based on the error logs. The specific manner of determining the initial fault cause by the third specified large model is not particularly limited in this embodiment. For example, error-related text in the error logs is extracted, and the initial fault cause is identified based on the error-related text.

[0134] In order to improve the reliability of fault cause diagnosis, the constraint conditions in the third prompt word engineering further include determining a confidence score of the initial fault cause; the initial fault cause with a confidence score greater than a second preset score is taken as the fault cause. The confidence score represents the confidence of the initial fault cause, and the confidence score is positively correlated with the confidence. Therefore, when the confidence score of the initial fault cause is greater than the second preset score, it is considered that the confidence of the initial fault cause is relatively high, and the reliability is high, so the initial fault cause can be taken as the fault cause. The value of the second preset score is not particularly limited in this embodiment, and can be set according to actual needs.

[0135] The above configuration parameter setting fault includes but is not limited to a fault such as a too large global batch size value, a too small global batch size value, and a too large micro batch size value. The hardware fault includes but is not limited to a fault such as insufficient video memory resources and insufficient GPU resources. The communication fault includes but is not limited to a fault such as bandwidth saturation.

[0136] After determining the fault cause, a third prompt word engineering can also be used to guide the third specified large model to generate a repair suggestion for repairing the fault cause. For example, the constraint condition in the third prompt word engineering can further include: if the fault cause is a configuration parameter setting fault, generating a repair suggestion of regenerating the target training strategy; if the fault cause is an environment fault or a communication fault, generating a suggestion of retraining the target model based on the current target training strategy.

[0137] As an optional embodiment, the constraint condition in the third prompt word engineering further includes:

[0138] If the fault cause is a timeout fault, determining a root fault corresponding to the timeout fault; the root fault includes at least one of a configuration parameter setting fault, a hardware fault and a communication fault;

[0139] Generating a fault repair suggestion for the root fault.

[0140] As shown in Figure 6 In this embodiment, root cause analysis is also performed for the timeout fault, that is, the root fault causing the timeout fault is determined. Specifically, the constraint condition in the third prompt word engineering includes: if the fault cause is a timeout fault, determining a root fault corresponding to the timeout fault. The root fault includes at least one of a configuration parameter setting fault, a hardware fault and a communication fault. A corresponding fault repair suggestion is generated for the root fault. The fault repair suggestion can refer to the foregoing, which will not be described here.

[0141] In summary, the embodiment performs root cause analysis on the timeout fault, excavates the root cause causing the timeout fault, and generates a fault repair suggestion, so as to timely and accurately adjust the related configurations of the model training system.

[0142] In addition, a VLLM (Very Large Language Model) inference service can also be deployed on the model training system to quickly and low-costly deploy each specified large model.

[0143] At this point, the present application uses multiple specified large models to work cooperatively, forming a closed-loop optimization mechanism of “analyzing system performance parameters-generating a target training strategy-verifying the target training strategy-repairing faults”, which has self-adaptive ability and can fully cope with various abnormal situations occurring in the model training process, thereby improving the performance of the model training system.

[0144] The trigger condition of adjusting the configuration parameters of the model training system will be described below.

[0145] As an optional embodiment, after obtaining the system performance parameters of the model training system, the method further includes:

[0146] obtain a nominal computing power value of the model training system;

[0147] determine whether the actual computing power value is less than the nominal computing power value;

[0148] If the actual computing power value is less than the nominal computing power value, the system performance parameters are input into the first specified large model based on the first pre-constructed prompt word engineering, and subsequent steps are performed.

[0149] In the embodiment, first, the nominal computing power value of the model training system is obtained. The nominal computing power value refers to the maximum computing power that the model training system can achieve under ideal conditions. The nominal computing power value can be determined by the number of cores, the clock frequency, and the number of floating point operations completed per clock cycle of the model training system, which is not particularly limited in the embodiment.

[0150] After obtaining the nominal computing power value, it is determined whether the actual computing power value of the model training system is less than the nominal computing power value. The actual computing power value is the actual computing power provided by the model training system when performing the current training task. If the actual computing power value is less than the nominal computing power value, it is considered that the performance of the model training system needs to be improved, and the configuration parameters need to be adjusted. Therefore, the system performance parameters are input into the first specified large model based on the first pre-constructed prompt word engineering, and subsequent steps are performed.

[0151] In summary, in the embodiment, by comparing the actual computing power value and the nominal computing power value of the model training system, it can be quickly determined whether the configuration parameters of the model training system need to be adjusted, and the performance of the model training system is optimized in time.

[0152] The application also provides a configuration adjustment device of a model training system, which comprises:

[0153] The first obtaining module is configured to obtain system performance parameters of the model training system. The system performance parameters include configuration parameters, an actual computing power value, and hardware performance indicators corresponding to historical training tasks.

[0154] The parameter determining module is configured to input the system performance parameters into a first specified large model based on a first pre-constructed prompt word engineering to obtain target configuration parameters after the to-be-optimized configuration parameters are adjusted.

[0155] The sequence determining module is configured to sort the target configuration parameters according to the hardware performance indicators to obtain a target configuration sequence.

[0156] The strategy determining module is configured to generate a target training strategy for training a target model according to the target configuration sequence. The target model is a training object of the model training system.

[0157] For detailed introduction of the configuration adjustment device of the model training system provided in the present application, please refer to the above-mentioned embodiments of the configuration adjustment method of the model training system, which will not be repeated here.

[0158] Based on the above-mentioned embodiments:

[0159] As an optional embodiment, the constraint condition in the first prompt word engineering includes:

[0160] According to the system performance parameter, determine the influence score corresponding to the configuration parameter; the influence score is used to represent the influence degree of the configuration parameter on the actual computing power value; the influence score is positively correlated with the influence degree;

[0161] The configuration parameter with an influence score greater than a first preset score is regarded as a to-be-optimized configuration parameter;

[0162] Adjust the to-be-optimized configuration parameter to obtain a target configuration parameter.

[0163] As an optional embodiment, the sequence determination module includes:

[0164] The second acquisition module is configured to acquire a pre-set bottleneck index range corresponding to each hardware performance index and a mapping relationship; the mapping relationship is a corresponding relationship between the bottleneck performance and the sorting priority;

[0165] The bottleneck determination module is configured to determine an index in the hardware performance index as a target bottleneck performance, which is within the corresponding bottleneck index range;

[0166] The priority determination module is configured to determine a target sorting priority corresponding to the target bottleneck performance based on the mapping relationship;

[0167] The sequence determination sub-module is configured to sort each target configuration parameter based on the target sorting priority to obtain a target configuration sequence; wherein, the higher the sorting priority of the target configuration parameter is, the higher the improvement degree of the target configuration parameter on the computing power value of the model training system is.

[0168] As an optional embodiment, the strategy determination module is specifically configured to:

[0169] Based on the pre-constructed second prompt word engineering, input the target configuration sequence into the second specified large model to obtain a target training strategy.

[0170] As an optional embodiment, the system performance parameter further includes a historical training strategy; the historical computing power value corresponding to the historical training strategy is the computing power value determined after the model training system trains the target model based on the historical training strategy and the target model converges;

[0171] The constraint condition in the second prompt word engineering includes:

[0172] The latest historical training strategy in the historical training strategies with a historical computing power value greater than a preset value is taken as a target historical training strategy.

[0173] The first preset number of configuration parameters in the target configuration sequence are taken as first configuration parameters, and other configuration parameters in the configuration parameters except the first configuration parameters are taken as to-be-adjusted configuration parameters.

[0174] The to-be-adjusted configuration parameters are adjusted based on the configuration parameters in the target historical training strategy to obtain second configuration parameters.

[0175] The first configuration parameters and the second configuration parameters are combined to obtain a target training strategy.

[0176] As an optional embodiment, the constraint condition in the second prompt word engineering further includes:

[0177] It is determined whether there is a same strategy as the target training strategy in the historical training strategies.

[0178] If not, the target training strategy is taken as a final target training strategy.

[0179] As an optional embodiment, the apparatus further includes:

[0180] The third acquisition module is configured to acquire a training result after the model training system trains the target model based on the target training strategy; if the training result represents that the model converges, the first processing module is triggered; if the training result represents that the model does not converge, the second processing module is triggered.

[0181] The first processing module is configured to determine an actual computing power value of the model training system; and store the target training strategy and the actual computing power value.

[0182] The second processing module is configured to acquire an error log generated by the model training system and a third prompt word engineering constructed in advance; and input the error log into a third specified large model based on the third prompt word engineering to obtain a fault cause.

[0183] As an optional embodiment, the constraint condition in the third prompt word engineering includes:

[0184] An initial fault cause is determined based on the error log; the initial fault cause includes at least one of a configuration parameter setting fault, a hardware fault, a communication fault, and a timeout fault.

[0185] A confidence score of the initial fault cause is determined; the confidence score is used to represent a confidence of the initial fault cause; the confidence score is positively correlated with the confidence.

[0186] The initial fault cause with the confidence score greater than the second preset score is taken as the fault cause.

[0187] As an optional embodiment, the constraint condition in the third prompt word engineering further includes:

[0188] If the fault cause is a timeout fault, a root fault corresponding to the timeout fault is determined, and the root fault includes at least one of a configuration parameter setting fault, a hardware fault and a communication fault.

[0189] A fault repair suggestion of the root fault is generated.

[0190] As an optional embodiment, the apparatus further includes:

[0191] The fourth acquisition module is configured to acquire a nominal computing power value of the model training system after acquiring the system performance parameter of the model training system.

[0192] The judgment module is configured to judge whether the actual computing power value is less than the nominal computing power value, and trigger the parameter determination module if the actual computing power value is less than the nominal computing power value.

[0193] The application further provides a storage medium, and the storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the configuration adjustment method of any one of the model training systems.

[0194] For detailed introduction of the storage medium provided by the application, refer to the embodiments of the configuration adjustment method of the model training system described above, which will not be repeated here.

[0195] The storage medium includes all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.

[0196] Although this specification contains many specific implementation details, these should not be construed as limiting the scope or the range of any invention, but merely as describing features that are specific to certain embodiments. Some of the features described in the specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented or practiced separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0197] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order nor that all illustrated operations be performed, to achieve desirable results. One will appreciate that a multitude of other operations can be performed or devices included in alternative embodiments can vary from those depicted in the drawings, and each iteration of an embodiment can also vary from one another. Moreover, it is recognized that some activities and / or functions can be performed in parallel, in sequence, or omitted. Likewise, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A configuration adjustment method for a model training system, characterized in that, The method includes: Obtain the system performance parameters of the model training system; the system performance parameters include configuration parameters, actual computing power values, and hardware performance indicators corresponding to historical training tasks. Based on the pre-built first prompt word project, the system performance parameters are input into the first specified large model to obtain the target configuration parameters after adjusting the configuration parameters to be optimized; the first prompt word project is used to guide the first specified large model to determine the target configuration parameters; The target configuration parameters are sorted according to the hardware performance indicators to obtain a target configuration sequence; Based on the target configuration sequence, a target training strategy for training the target model is generated; the target model is the training object of the model training system. The constraints in the first prompt word project include: Based on the system performance parameters, an impact score corresponding to the configuration parameter is determined; the impact score is used to characterize the degree of influence of the configuration parameter on the actual computing power value; the impact score is positively correlated with the degree of influence; configuration parameters with an impact score greater than a first preset score are taken as the configuration parameters to be optimized; the configuration parameters to be optimized are adjusted to obtain the target configuration parameters; The step of generating a target training strategy for training a target model based on the target configuration sequence includes: inputting the target configuration sequence into a second specified large model based on a pre-constructed second prompt word project to obtain the target training strategy; the second prompt word project is used to guide the second specified large model to generate the target training strategy based on the target configuration sequence. The system performance parameters also include historical training strategies; the historical computing power value corresponding to the historical training strategy is the computing power value determined by the model training system after the target model has converged, based on the historical training strategy. The constraints in the second prompt word project include: The latest historical training strategy in which the historical computing power value is greater than a preset value is taken as the target historical training strategy; the first preset number of configuration parameters in the target configuration sequence are taken as the first configuration parameters; and the other configuration parameters in the configuration parameters other than the first configuration parameters are taken as configuration parameters to be adjusted; the configuration parameters to be adjusted are adjusted based on the configuration parameters in the target historical training strategy to obtain the second configuration parameters; the first configuration parameters and the second configuration parameters are combined to obtain the target training strategy.

2. The configuration adjustment method for the model training system as described in claim 1, characterized in that, The step of sorting the target configuration parameters according to the hardware performance indicators to obtain the target configuration sequence includes: Obtain the bottleneck index range and mapping relationship corresponding to each of the pre-set hardware performance indicators; the mapping relationship is the correspondence between bottleneck performance and sorting priority. The hardware performance indicators that fall within the corresponding bottleneck indicator range are taken as the target bottleneck performance. Based on the mapping relationship, the target ranking priority corresponding to the target bottleneck performance is determined; Based on the target sorting priority, the target configuration parameters are sorted to obtain the target configuration sequence; wherein, the higher the sorting priority of the target configuration parameter, the greater the degree to which the target configuration parameter improves the computing power of the model training system.

3. The configuration adjustment method for the model training system as described in claim 1, characterized in that, The constraints in the second prompt word project also include: Determine whether any of the historical training strategies are identical to the target training strategy; If it does not exist, then the target training strategy will be used as the final target training strategy.

4. The configuration adjustment method for the model training system as described in claim 1, characterized in that, The method further includes: After the model training system trains the target model based on the target training strategy, the training results are obtained; If the training results indicate that the model has converged, then the actual computing power of the model training system is determined; the target training strategy and the actual computing power are stored. If the training results represent a model that has not converged, then the following steps are performed: Obtain the error logs generated by the model training system, as well as the pre-built third prompt word project; Based on the aforementioned third prompt word project, the error log is input into the third designated large model to obtain the cause of the failure.

5. The configuration adjustment method for the model training system as described in claim 4, characterized in that, The constraints in the third prompt word project include: The initial cause of the failure is determined based on the error log; the initial cause of the failure includes at least one of the following: configuration parameter setting failure, hardware failure, communication failure, and timeout failure. A confidence score is determined for the initial cause of the failure; the confidence score is used to characterize the confidence level of the initial cause of the failure; the confidence score is positively correlated with the confidence level. The initial fault cause whose confidence score is greater than the second preset score is taken as the fault cause.

6. The configuration adjustment method for the model training system as described in claim 5, characterized in that, The constraints in the third prompt word project also include: If the cause of the fault is the timeout fault, then the underlying fault corresponding to the timeout fault is determined; the underlying fault includes at least one of the configuration parameter setting fault, the hardware fault, and the communication fault. Generate troubleshooting recommendations for the underlying fault.

7. The configuration adjustment method for the model training system as described in claim 1, characterized in that, After obtaining the system performance parameters of the model training system, the following is also included: Obtain the nominal computing power value of the model training system; Determine whether the actual computing power value is less than the nominal computing power value; If the actual computing power value is less than the nominal computing power value, then proceed to the step of inputting the system performance parameters into the first specified large model based on the pre-built first prompt word project, and execute subsequent steps.

Images