Network model training method, cloud platform and related device

Abstract

The invention discloses a network model training method, a cloud platform and a related device, and belongs to the technical field of artificial intelligence. The method comprises the steps that a parameter configuration interface is provided, the parameter quantity and the type of a target network model are obtained from the parameter configuration interface, training configuration information of the target network model is determined based on the parameter quantity and the type of the target network model, the training process of the target network model is divided into a plurality of training stages, and the training stages are used for training the target network model. The model parameter quantities corresponding to different training stages are different, the model parameter quantities of the plurality of training stages are sequentially increased according to a training sequence, the training configuration information comprises configuration information of the plurality of training stages, and the target network model is trained according to the configuration information of the plurality of training stages. Therefore, staged fine control of network model training can be realized, the convergence speed of the network model can be effectively accelerated, the calculation amount required by network model training is saved, and the network model training cost is reduced.

Description

[0001] This application claims priority to Chinese patent application No. 202311061555.2, filed on August 22, 2023, and entitled “Model training method, cloud platform and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a network model training method, a cloud platform and related devices. Background Art

[0003] With the continuous development and improvement of artificial intelligence technology, neural network models have been widely used in natural language processing, computer vision, speech recognition and other fields. Since neural network models with large parameters (also known as large-scale) can better handle complex tasks and have better generalization capabilities, the number of parameters in neural network models is increasing, and the demand for training neural network models with large parameters is also gradually increasing.

[0004] Currently, neural network models are mainly trained through cloud computing platforms. However, the training functions provided by existing cloud computing platforms are relatively simple and cannot adapt to the needs brought about by the continuous expansion of parameter quantities. If a neural network model with a large number of parameters is trained, it will take longer training time and more computing power, making the model training cost high and the efficiency low. Summary of the invention

[0005] The present application provides a network model training method, device, equipment, storage medium and computer program, which can solve the problem of low model training efficiency in related technologies. The technical solution is as follows:

[0006] In a first aspect, a network model training method is provided, the method comprising: providing a parameter configuration interface, the parameter configuration interface being used to instruct a user to input parameter amounts and types of a target network model to be trained, obtaining the parameter amounts and types of the target network model from the parameter configuration interface, and determining training configuration information of the target network model based on the parameter amounts and types of the target network model, wherein the training process of the target network model is divided into a plurality of training stages, different training stages correspond to different model parameter amounts, and the model parameter amounts of the plurality of training stages are increased in sequence according to a training order, the training configuration information comprising configuration information of the plurality of training stages; and training the target network model according to the configuration information of the plurality of training stages.

[0007] In the present application, the user can input the parameter amount and type of the target network model in the parameter configuration interface, and thus determine the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models with different parameter amounts and different types in a targeted manner, so that the parameter amount of the model, the type of model and the determined training configuration information are more compatible. In addition, the present application can also divide the training process of a network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the training sequence, so that the staged fine control of the network model training can be achieved, and since the parameter amount of the network model is gradually increased with the training stage, it can effectively speed up the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the cost of network model training.

[0008] Optionally, the configuration information of the multiple training stages includes model planning information, training planning information and data planning information corresponding to the multiple training stages respectively, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration status of the training data of the corresponding training stage.

[0009] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of a model parameter in the corresponding training phase.

[0010] Optionally, the training planning information also includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0011] It should be noted that the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase.

[0012] Optionally, the training planning information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates the changing manner of the changing step size of the model parameters, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0013] Optionally, the training planning information may further include a random dropout rate, where the random dropout rate refers to the proportion of neurons that do not participate in calculations in each iteration when training a neural network.

[0014] In the case of different types of network models, the parameters included in the model planning information are different. Optionally, in the case where the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0015] Optionally, for any training stage except the first training stage in the multiple training stages, the value of any parameter included in the model planning information of the training stage is greater than or equal to the value of the corresponding parameter in the previous training stage of the training stage, and, among the parameters included in the model planning information of the training stage, there is at least one parameter whose value is greater than the value of the corresponding parameter in the previous training stage of the training stage. That is to say, for two adjacent training stages in the multiple training stages, the value of the parameter included in the model planning information of the latter training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and the values ​​of the parameters included in the model planning information of the two training stages are not completely equal.

[0016] Optionally, the model planning information also includes the model parameter quantity of the corresponding training stage. The model parameter quantity is determined based on the number of self-attention networks in the above network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. The model parameter quantity is used to intuitively determine the scale of the network model in the training stage. The difference between the model parameter quantity of the last training stage and the parameter quantity of the target network model input by the user is less than or equal to the parameter quantity difference threshold.

[0017] Optionally, the data planning information includes a ratio of a training data set of a corresponding training stage to a total training data set, where the total training data set is the sum of training data sets of multiple training stages.

[0018] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage among multiple training stages.

[0019] Optionally, the pruning configuration information includes pruning status information, and the pruning status information indicates whether to filter the training data set of the corresponding training stage.

[0020] Optionally, the data planning information may also include a storage path for the training data set.

[0021] The parameter amount and type of the target network model are input into the training prediction model to obtain the first training configuration information output by the training prediction model, and the training configuration information of the target network model is determined based on the first training configuration information. In other words, the present application can specifically determine the corresponding training configuration information for network models with different parameter amounts and different types, so that the parameter amount of the model, the type of the model and the determined training configuration information are more adapted.

[0022] Optionally, a configuration preview interface is provided, which includes first training configuration information. In response to a user's modification operation on the first training configuration information, the first training configuration information is modified to obtain second training configuration information. Based on the second training configuration information, the training configuration information of the target network model is determined.

[0023] That is to say, after determining the first training configuration information based on the parameter quantity and type of the target network model, the user can also modify the first training configuration information to obtain the training configuration information of the target network model, thereby achieving refined customization of the training plan of the target network model and effectively improving the training efficiency of the neural network model.

[0024] For the convenience of description, the number of training stages, model iterations, learning rate range, expected loss value, warm-up iterations, learning rate decay type, parameter accuracy, batch size, random inactivation rate, the number of self-attention networks in the network model, the dimension of the hidden layer in each self-attention network, the number of attention heads, the number of model parameters, the proportion of the training data set in the corresponding training stage to the total training data set, the configuration information of pruning, and the configuration information of distillation preheating are collectively referred to as planning parameters.

[0025] Based on the above description, the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase, and the model parameter quantity is determined based on the number of self-attention networks in the above network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. That is, the values ​​of planning parameters such as the expected loss value and the model parameter quantity need to be determined based on the values ​​of other planning parameters and cannot be directly configured. Therefore, planning parameters such as the expected loss value and the model parameter quantity can also be called dependent variable parameters, and the planning parameters used to determine the dependent variable parameters are called independent variable parameters. Among these planning parameters, the planning parameters that can be directly configured can be called configurable parameters, and usually, users can modify the configurable parameters.

[0026] Optionally, the first training configuration information includes values ​​of multiple planning parameters, and the second training configuration information also includes values ​​of multiple planning parameters, and the value of at least one configurable parameter among these planning parameters has been modified by the user. That is, the user modifies the value of the at least one configurable parameter in the first training configuration information, and in response to the user's modification operation, the value of the at least one configurable parameter is modified.

[0027] In practical applications, the configurable parameter may also be the number of training stages of the target network model.

[0028] Optionally, the configuration preview interface also includes a stage add button and a stage delete button. The user can add a training stage by triggering the stage add button and delete a training stage by triggering the stage delete button, thereby modifying the number of training stages.

[0029] Optionally, based on the second training configuration information, the implementation process of determining the training configuration information of the target network model includes: if the at least one configurable parameter modified by the user includes an independent variable parameter, then based on the value of the modified independent variable parameter, the value of the corresponding dependent variable parameter is re-determined, and the value of the at least one configurable parameter, the re-determined value of the dependent variable parameter, and the values ​​of other configurable parameters included in the first training configuration information except the at least one configurable parameter are determined as the training configuration information of the target network model. If the at least one configurable parameter modified by the user does not include an independent variable parameter, then the value of the at least one configurable parameter and the values ​​of the planning parameters included in the first training configuration information except the at least one configurable parameter are determined as the training configuration information of the target network model.

[0030] Next, the implementation process of training the target network model according to the configuration information of multiple training stages will be introduced through steps (1) to (4).

[0031] Step (1): Let i=1, and determine the i-th network model based on the configuration information of the i-th training stage in multiple training stages.

[0032] If the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage; if the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0033] Optionally, the configuration information of the i-th training stage includes the model planning information of the i-th training stage. In this case, based on the configuration information of the i-th training stage, the trained i-1-th network model is amplified to obtain the i-th network model. The implementation process includes: providing a parameter amplification interface, the parameter amplification interface is used to instruct the user to select a parameter amplification strategy, the parameter amplification strategy is used to determine the initial values ​​of the parameters newly added to the i-th network model relative to the i-1-th network model, obtaining the parameter amplification strategy from the parameter amplification interface, and based on the parameter amplification strategy and the model planning information of the i-th training stage, performing parameter amplification on the i-1-th network model to obtain the i-th network model.

[0034] Optionally, the parameter augmentation strategy includes a width augmentation strategy and a depth augmentation strategy, wherein the width augmentation strategy indicates a method for determining the initial values ​​of the parameters of the newly added dimensions of the hidden layer in the self-attention network, and the depth augmentation strategy refers to a method for determining the initial values ​​of the parameters in the newly added self-attention network of the i-th network model relative to the i-1-th network model. In this case, based on the parameter augmentation strategy and the model planning information of the i-th training stage, the implementation process of parameter augmentation for the i-1-th network model includes: based on the width augmentation strategy and the model planning information of the i-th training stage, the dimensions of the hidden layers in each self-attention network in the i-1-th network model are augmented to obtain an initial augmented model, and based on the depth augmentation strategy and the model planning information of the i-th training stage, the self-attention network of the initial augmented model is augmented to obtain the i-th network model.

[0035] Based on the values ​​of parameters of neurons in the hidden layer of each self-attention network in the i-1th network model, according to the width expansion strategy, the initial values ​​of parameters of newly added neurons in the hidden layer of each self-attention network in the initial augmented model relative to the i-1th network model are determined, the dimension of each hidden layer in the self-attention network in the initial augmented model is the same as the dimension of each hidden layer in the network model in the i-th training stage, and the number of self-attention networks in the initial augmented model is the same as the number of self-attention networks in the i-1th network model.

[0036] Based on the values ​​of the parameters in the self-attention network in the initial augmented model, according to the deep expansion strategy, the initial values ​​of the parameters in the newly added self-attention network in the i-th network model relative to the initial augmented model are determined, the dimension of the hidden layer in the self-attention network in the i-th network model is the same as the dimension of the hidden layer in each self-attention network in the network model in the i-th training stage, and the number of self-attention networks in the i-th network model is the same as the number of self-attention networks in the i-th network model.

[0037] Step (2): Train the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model.

[0038] Before the i-th network model is trained according to the configuration information of the i-th training stage to obtain the trained i-th network model, the user can also trigger a start training operation. In this case, in response to the start training operation triggered by the user, the i-th network model is trained according to the configuration information of the i-th training stage.

[0039] Step (3): Determine the training evaluation result of the trained i-th network model, and provide a training evaluation interface, which includes the training evaluation result of the trained i-th network model, and the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training.

[0040] Determine a loss curve of the ith network model, where the loss curve indicates how the loss value changes with the number of model iterations, where the loss value indicates the gap between the predicted value and the true value of the network model, input at least one test data set into the trained ith network model, obtain an inference result of the at least one test data set output by the trained ith network model, and determine a training evaluation result of the trained ith network model based on the loss curve and the inference result of the at least one test data set.

[0041] Optionally, during the training of the ith network model, the change of the loss value of the ith network model with the number of model iterations is recorded. In this way, the loss curve of the ith network model can be drawn according to the change of the loss value of the ith network model with the number of model iterations.

[0042] Based on the inference result of the at least one test data set, the inference accuracy corresponding to the at least one test data set is determined; based on the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model and the expected loss value of the i-th training stage, the training evaluation result of the trained i-th network model is determined.

[0043] Optionally, the training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result, wherein the loss value prediction evaluation result indicates whether the loss value of the i-th network model meets the expected prediction, and the test set prediction evaluation result indicates whether the inference result of the at least one test data set meets the expected prediction. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the loss value prediction evaluation result and the test set prediction evaluation result output by the training prediction model.

[0044] Optionally, the training evaluation result also includes training completion prediction information, which indicates a prediction of whether the trained i-th network model has completed training. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the training completion prediction information of the i-th training stage output by the training prediction model.

[0045] Since the training evaluation result also includes a prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who have no training experience, thereby improving the training efficiency of the network model.

[0046] Optionally, the training evaluation interface also includes at least one of the loss curve of the ith network model and the inference accuracy rate corresponding to the at least one test data set. In this way, the user can confirm whether the trained ith network model has completed training based on the loss curve of the ith network model and / or the inference accuracy rate corresponding to the at least one test data set, as well as the training evaluation result.

[0047] Since the training evaluation interface also includes at least one of the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, that is, the training evaluation results of the network model obtained in each training stage are displayed to the user in real time through the training evaluation interface, which enables the control of the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training.

[0048] In actual applications, after providing the training evaluation interface, the training evaluation result can also be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result.

[0049] Optionally, the training evaluation interface includes the loss curve of the i-th network model, the inference accuracy corresponding to the at least one test data set, and the training evaluation result. In this case, the user can determine whether the training of the i-th network model meets expectations based on the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set to obtain the user training evaluation result. If the user training evaluation result is inconsistent with the training evaluation result, the user can modify the training evaluation result. In response to the user's modification operation on the training evaluation result, the training evaluation result is modified to obtain the modified training evaluation result. If the user training evaluation result is consistent with the training evaluation result, the user does not modify the training evaluation result.

[0050] Step (4): If a user-triggered completion training operation is detected and the i-th training stage is not the last training stage among multiple training stages, then let i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among multiple training stages. If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among the multiple training stages, then the trained i-th network model is used as the trained target network model.

[0051] Optionally, the user can trigger the completion of training operation on the training evaluation interface after confirming that the trained i-th network model has completed training. In this case, the completion of training operation triggered by the user is detected, and when the i-th training stage is not the last training stage among the multiple training stages, let i=i+1, and return to the above-mentioned step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages. When the i-th training stage is the last training stage among the multiple training stages, the trained i-th network model is used as the trained target network model.

[0052] Optionally, the user can trigger a continue training operation or a re-planning operation on the training evaluation interface after confirming that the trained i-th network model has not completed training. The continue training operation indicates to continue training the i-th network model, and the re-planning operation indicates to return to the above step of determining the training configuration information of the target network model and re-determine the training configuration information of the target network model. If the user triggers the continue training operation, the continue training operation triggered by the user is detected, and then the i-th network model is continued to be trained based on the configuration information of the i-th training stage. If the user triggers the re-planning operation, the step of determining the training configuration information of the target network model is returned to re-determine the training configuration information of the target network model.

[0053] If a user-triggered completion training operation is detected and the i-th training stage is not after the last training stage among the multiple training stages, the configuration information of at least one training stage after the i-th training stage among the multiple training stages may also be optimized. There are multiple implementations of optimizing the configuration information of at least one training stage after the i-th training stage among the multiple training stages, and two of them are described below.

[0054] In a first implementation, configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized based on at least one of the loss curve of the i-th network model and the inference result of the at least one test data set.

[0055] Optionally, at least one of the loss curve and the inference results of the at least one test data set is input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0056] Optionally, after providing the training evaluation interface, the training evaluation result can also be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result. In this case, based on at least one of the loss curve of the i-th network model and the inference result of at least one test data set, as well as the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0057] Since the determined training evaluation results may not meet the user's expectations, in this case, the user can modify the training evaluation results. In this way, based on the interaction between the user and the execution entity of this application, the loss curve and the inference results of the test data set, the configuration information of the subsequent training stage can be adjusted in time, so that the final trained network model is more in line with the user's expectations.

[0058] Optionally, the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification are input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0059] Optionally, after obtaining the trained i-th network model, a forgetfulness evaluation is performed on the trained i-th network model to obtain a forgetfulness evaluation result, which is used to indicate the reasoning of the trained i-th network model on the training data set of the historical training stage, and the historical training stage is a training stage before the i-th training stage in multiple training stages. In this case, if it is determined based on the forgetfulness evaluation result that the reasoning accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the loss curve of the i-th network model and at least one of the reasoning results of at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification, and the forgetfulness evaluation result, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized. Otherwise, based on the loss curve and at least one of the reasoning results of at least one test data set, as well as the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized.

[0060] Optionally, the forgetting assessment result is not forgotten or forgotten, not forgotten indicates that there is at least one historical training stage whose inference accuracy is not greater than the accuracy threshold, and forgotten indicates that there is at least one historical training stage whose inference accuracy is not greater than the accuracy threshold. In this case, if the inference accuracy of at least one historical training stage corresponding to the at least one target historical data set is not greater than the accuracy threshold, the forgetting assessment result is determined to be forgotten. If the inference accuracy of at least one historical training stage corresponding to the at least one target historical data set is not greater than the accuracy threshold, the forgetting assessment result is determined to be not forgotten.

[0061] It should be noted that, when the forgetting evaluation result of the trained i-th network model indicates that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, it means that the inference accuracy of the trained i-th network model for the training data set in the historical training stage is low, and there is forgetting. Therefore, the corresponding training data set of at least one training stage after the i-th training stage includes part or all of the training data set of the forgotten training stage, which can ensure that the data set of the forgotten training stage is reviewed in the subsequent training stage, thereby improving the inference performance of the target network model.

[0062] The second implementation method is to perform a forgetfulness evaluation on the trained i-th network model after obtaining the trained i-th network model to obtain a forgetfulness evaluation result. In this case, if it is determined based on the forgetfulness evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the forgetfulness evaluation result, the loss curve of the i-th network model, and at least one of the inference results of at least one test data set, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized. Otherwise, based on at least one of the loss curve of the i-th network model and the inference results of at least one test data set, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized.

[0063] In a second aspect, a network model training device is provided, wherein the network model training device has the function of implementing the network model training method behavior in the first aspect. The network model training device includes at least one module, and the at least one module is used to implement the network model training method provided in the first aspect.

[0064] In a third aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device being used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the network model training method described in the first aspect.

[0065] Optionally, each computing device may further include a communication bus, which is used to establish a connection between the processor and the memory of each computing device.

[0066] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the steps of the network model training method described in the first aspect.

[0067] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the steps of the network model training method described in the first aspect.

[0068] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0070] Figure 2 It is a schematic diagram of the interaction between various units included in a cloud platform provided in an embodiment of the present application;

[0071] Figure 3 It is a flow chart of a network model training method provided in an embodiment of the present application;

[0072] Figure 4 is a schematic diagram of a parameter configuration interface provided in an embodiment of the present application;

[0073] Figure 5 is a schematic diagram of training planning information provided by an embodiment of the present application;

[0074] Figure 6 is a schematic diagram of another type of training planning information provided in an embodiment of the present application;

[0075] Figure 7 is a schematic diagram of model planning information provided by an embodiment of the present application;

[0076] Figure 8 is a schematic diagram of data planning information provided by an embodiment of the present application;

[0077] Fig. 9 is a schematic diagram of a configuration preview interface provided in an embodiment of the present application;

[0078] Fig.10 is a schematic diagram of a parameter amplification interface provided in an embodiment of the present application;

[0079] Fig.11 is a schematic diagram of an interface provided in an embodiment of the present application;

[0080] Fig.12 is a schematic diagram of a training evaluation interface provided in an embodiment of the present application;

[0081] Fig.13 is a schematic diagram of a training record interface provided in an embodiment of the present application;

[0082] Fig.14 is a flowchart of another network model training method provided in an embodiment of the present application;

[0083] Fig.15 It is a structural diagram of a network model training device provided in an embodiment of the present application;

[0084] Fig.16 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0085] Fig.17 It is an architecture diagram of a computing device cluster provided in an embodiment of the present application;

[0086] Fig.18 It is a schematic diagram of the connection between computing devices provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0088] For ease of understanding, before explaining in detail the training method of the network model provided in the embodiment of the present application, the application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

[0089] First, the application scenarios involved in the embodiments of the present application are introduced.

[0090] With the continuous development and improvement of artificial intelligence technology, neural network models have been widely used in natural language processing, computer vision, speech recognition and other fields. Since neural network models with large parameters (also known as large-scale) can better handle complex tasks and have better generalization capabilities, the number of parameters of neural network models is increasing, and the demand for training neural network models with large parameters is also gradually increasing. Taking the training of the GPT-3 neural network model with 175 billion parameters as an example, it takes at least more than a month to train the GPT-3 neural network model through the current most advanced graphics processor with 1024 video cards and 80GB of video memory.

[0091] Currently, neural network models are mainly trained through cloud computing platforms. However, the training functions provided by existing cloud computing platforms are relatively simple and cannot adapt to the needs brought about by the continuous expansion of parameters. Users cannot finely customize their own training plans to improve the training efficiency of neural network models. Cloud computing platforms cannot provide users with neural network model training experience to optimize the training process of neural network models, thereby accelerating the training of neural network models. If a neural network model with a large number of parameters is trained, it will take longer training time and more computing power, making the model training cost high and the efficiency low.

[0092] Based on this, an embodiment of the present application provides a network model training method, in which a user can input the parameter amount and type of the target network model to be trained in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models of different parameter amounts and different types in a targeted manner, so that the parameter amount of the model, the type of the model and the determined training configuration information are more adapted. In addition, the embodiment of the present application can also divide the training process of the network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training, so that the staged fine control of the network model training can be achieved, and because the parameter amount of the network model is gradually increased with the training stage, it can effectively speed up the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the training cost of the network model.

[0093] The executor of the network model training method provided in the embodiment of the present application can be a cloud platform or an electronic device with a model training function. The cloud platform can be a server cluster or a distributed system composed of multiple physical servers, etc., or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud server or a cloud computing service center.

[0094] Next, the implementation environment involved in the embodiment of the present application is introduced. Based on the above description, the execution subject of the network model training method provided in the embodiment of the present application can be a cloud platform or an electronic device with a model training function, and the following is introduced using a cloud platform as an example.

[0095] Please refer to Figure 1 , Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application, and the implementation environment includes a planning unit 01, a training unit 02, a recording unit 03 and a prediction unit 04. Among them, the planning unit 01 includes a training planning subunit 11, a model planning subunit 12 and a data planning subunit 13, and the training unit 02 includes a training execution subunit 21, an interaction evaluation subunit 22, a forgetting evaluation subunit 23 and a parameter amplification subunit 24.

[0096] The training planning subunit 11 is used to determine the training planning information corresponding to multiple training stages, the model planning subunit 12 is used to determine the model planning information corresponding to multiple training stages, and the data planning subunit 13 is used to determine the data planning information corresponding to multiple training stages. The training execution subunit 21 is used to train the network models of each stage, the interactive evaluation subunit 22 is used to perform training evaluation on the network models trained in each training stage, the forgetting evaluation subunit 23 is used to perform forgetting evaluation on the network models trained in each training stage, and the parameter amplification subunit 24 is used to amplify the parameters of the network models in each training stage. The recording unit 03 is used to record the model planning information, training planning information and data planning information corresponding to multiple training stages of the target network model, the change of the loss rate of the target network model during the training process, the training evaluation results of the network model trained in each training stage, the forgetting evaluation results of the network model trained in each training stage, the parameter amplification strategy of the target network model, and the interactive operation between the user and the cloud platform. The prediction unit 04 is used to determine the first training configuration information based on the parameter quantity and type of the target network model, predict the training evaluation results of the network model trained in each stage, predict the forgetting evaluation results of the network model trained in each stage, predict the parameter expansion strategy of the network model, and give suggestions for the training of the target network model.

[0097] Optionally, the prediction unit 04 can be a training prediction model. The cloud platform inputs the parameter quantity and type of the target network model into the training prediction model to obtain the first training configuration information output by the training prediction model. The training prediction model can also predict the training evaluation results of the network model trained in each stage, predict the forgetting evaluation results of the network model trained in each stage, predict the parameter amplification strategy of the network model, and give suggestions for the training of the target network model.

[0098] It should be noted that the training prediction model can be a generative model, which can be based on a deep neural network and can learn to abstract the essential laws and probability distribution of data by training large-scale data sets, thereby generating new data.

[0099] Please refer to Figure 2 , Figure 2It is a schematic diagram of the interaction between various units included in a cloud platform provided in an embodiment of the present application. The planning unit 01 can obtain the parameter quantity and type of the target network model, and send the parameter quantity and type of the target network model to the prediction unit 04. The prediction unit 04 determines the first training configuration information based on the parameter quantity and type of the target network model. After determining the first training configuration information, the first training configuration information is sent to the planning unit 01. The planning unit 01 determines the training configuration information of the target network model according to the first training configuration information, and then sends the training configuration information of the target network model to the training unit 02 and the recording unit 03. The training unit 02 receives the training configuration information of the target network model sent by the planning unit 01, and trains the target network model based on the training configuration information of the target network model. The training unit 02 can also call the prediction unit 04 to predict the training evaluation results of the network model trained at each stage, predict the forgetting evaluation results of the network model trained at each stage, predict the parameter expansion strategy of the network model, and give suggestions for the training of the target network model, and then the prediction unit 04 sends the predicted training evaluation results, forgetting evaluation results and parameter expansion strategy to the training unit 02. The training unit 02 receives the training evaluation results, forgetting evaluation results and parameter expansion strategy predicted and sent by the prediction unit 04. The training unit 02 also sends the interactive operations between the user and the cloud platform and the training status of the network model at each training stage to the recording unit 03. The recording unit 03 stores the training configuration information of the target network model sent by the planning unit 01, and stores the interactive operations between the user and the cloud platform and the specific conditions of the network model trained at each training stage sent by the training unit 02. The recording unit 03 can also send all the information stored by itself to the prediction unit 04, and the prediction unit 04 receives all the information sent by the recording unit 03.

[0100] In some embodiments, the recording unit 03 can also provide a training record interface, through which the user can view the configuration information and training records of any training stage, and the training record includes the training status of the network model in each training stage.

[0101] For example, the training situation may include a loss curve corresponding to any training stage, an inference accuracy rate corresponding to at least one test data set, a training log, etc., which is not limited in the embodiments of the present application.

[0102] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0103] Figure 3 This is a flowchart of a network model training method provided in an embodiment of the present application. Figure 3 The method includes the following steps. For the convenience of description, the training method of the network model provided in the embodiment of the present application is introduced with the cloud platform as the execution subject.

[0104] Step 301: Provide a parameter configuration interface, which is used to instruct the user to input the parameter amount and type of the target network model to be trained.

[0105] In practical applications, the type of network model can be a BERT (bidirectional encoder representations from transformers) model, a GPT (Generative Pre-trained Transformer) model, a T5 (Text-To-Text Transfer Transformer) model, a ResNet (residual network) model, etc., which is not limited to the embodiments of the present application.

[0106] Step 302: Obtain the parameter quantity and type of the target network model from the parameter configuration interface.

[0107] In some embodiments, the cloud platform completes the operation in response to the user-triggered input and obtains the parameter quantity and type of the target network model from the parameter configuration interface.

[0108] After inputting the parameter amount and type of the target network model, the user can trigger the input completion operation. The cloud platform responds to the input completion operation triggered by the user and obtains the parameter amount and type of the target network model from the parameter configuration interface.

[0109] For example, see Figure 4 , the cloud platform can provide Figure 4 The parameter configuration interface shown in the figure allows users to Figure 4 In the parameter configuration interface shown, the parameter amount of the target network model entered (also known as the scale of the target network model) is 7 billion (billion, B), and the type of the target network model is a GPT model. After the user enters the parameter amount and type of the target network model, the input is triggered to complete the operation. The cloud platform responds to the input triggered by the user to complete the operation and obtains the parameter amount and type of the target network model from the parameter configuration interface.

[0110] Step 303: Based on the parameter quantity and type of the target network model, determine the training configuration information of the target network model. The training process of the target network model is divided into multiple training stages. Different training stages correspond to different model parameter quantities, and the model parameter quantities of the multiple training stages increase in sequence according to the training sequence. The training configuration information of the target network model includes configuration information of multiple training stages.

[0111] In some embodiments, the configuration information of the multiple training stages includes model planning information, training planning information and data planning information corresponding to the multiple training stages respectively, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration status of the training data of the corresponding training stage.

[0112] Next, model planning information, training planning information, and data planning information are introduced respectively.

[0113] 1. Training planning information

[0114] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of a model parameter in the corresponding training phase.

[0115] It should be noted that, for any training stage except the first training stage in the multiple training stages, the maximum value of the learning rate interval of the training stage is the minimum value of the learning rate interval of the previous training stage of the training stage, and the minimum value of the learning rate interval of the last training stage in the multiple training stages is 0.

[0116] Optionally, the training planning information also includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0117] It should be noted that the expected loss value is determined based on the number of model iterations and the learning rate interval of the corresponding training stage. For example, the cloud platform can input the number of model iterations and the learning rate interval of each training stage into the training prediction model to obtain the expected loss value of each training stage output by the training prediction model. Of course, in practical applications, the expected loss value can also be determined by other means, which is not limited in this application.

[0118] For example, see Figure 5 , Figure 5 A schematic diagram of training planning information. Figure 5The training process of the target network model is divided into training phase 1 to training phase n. The training planning information of training phase 1 to training phase n includes the number of model iterations, learning rate range and expected loss value. Among them, the number of model iterations in training phase 1 is 25,000 times, and the learning rate range is 8e-4 (i.e. 8×10 -4 ) to 7e-4 (i.e. 7×10 -4 ), with an expected loss of 3.7. The model iterations for training stage 2 are 60,000, with a learning rate range of 7e-4 to 5e-4, and an expected loss of 2.8. The model iterations for training stage 3 are 150,000, with a learning rate range of 5e-4 to 4e-4, and an expected loss of 2.2. The model iterations for training stage n are 900,000, with a learning rate range of 8e-5 (i.e., the minimum value of the learning rate range for training stage n-1) to 0, and an expected loss of 1.8.

[0119] In some embodiments, the training planning information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter accuracy and the batch size, wherein the learning rate decay type indicates the changing manner of the change step size of the model parameters, the parameter accuracy indicates the accuracy of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0120] For example, the learning rate decay types include linear decay, exponential decay, etc., which are not limited in the embodiments of the present application.

[0121] For example, the parameter precision includes fp16, fp32, etc., which is not limited in the embodiments of the present application. Among them, fp16 is also called half-precision floating-point, which occupies 16 bits of storage space, including 1 sign bit, 5 exponent bits and 10 mantissa bits. fp32 is also called single-precision floating-point, which occupies 32 bits of storage space, including 1 sign bit, 8 exponent bits and 23 mantissa bits.

[0122] Optionally, the training planning information may further include a drop out rate, which refers to the proportion of neurons that do not participate in calculations (ie, their outputs are set to 0) in each iteration when training a neural network.

[0123] The random dropout rate ranges from 0 to 1. For example, the random dropout rate can be 0.5, in which case the output of approximately half of the neurons is set to 0 in each iteration.

[0124] It should be noted that the learning rate decay type, parameter precision, batch size, and random dropout rate are the same in each training stage, that is, during the entire training process of the target network model, the learning rate decay type, parameter precision, batch size, and random dropout rate remain unchanged.

[0125] For example, see Figure 6 The number of warm-up iterations in the training plan information is 1000, the random dropout rate is 0.1, the learning rate decay type is linear decay, and the parameter precision is fp16.

[0126] 2. Model planning information

[0127] Based on the above description, there are many types of network models. When the types of network models are different, the parameters included in the model planning information are different. For example, when the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0128] It should be noted that the dimension of the hidden layer refers to the number of neurons in the hidden layer.

[0129] The above-mentioned BERT model, GPT model, and T5 model are all pre-trained language models based on the Transformer architecture, and the BERT model, GPT model, or T5 model all include a self-attention network, or in other words, the pre-trained language model based on the Transformer architecture includes a self-attention network. In this case, if the type of the target network model is a BERT model, a GPT model, or a T5 model, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0130] Optionally, for any training stage except the first training stage in the multiple training stages, the value of any parameter included in the model planning information of the training stage is greater than or equal to the value of the corresponding parameter in the previous training stage of the training stage, and, among the parameters included in the model planning information of the training stage, there is at least one parameter whose value is greater than the value of the corresponding parameter in the previous training stage of the training stage. That is to say, for two adjacent training stages in the multiple training stages, the value of the parameter included in the model planning information of the latter training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and the values ​​of the parameters included in the model planning information of the two training stages are not completely equal.

[0131] Taking the network model of the Transformer architecture as an example, for any training stage except the first training stage in multiple training stages, the number of self-attention networks in the network model of the training stage is greater than or equal to the number of self-attention networks in the network model of the previous training stage, the dimension of the hidden layer in each self-attention network in the network model of the training stage is greater than or equal to the dimension of the hidden layer in each self-attention network in the network model of the previous training stage, the number of attention heads in the network model of the training stage is greater than or equal to the number of attention heads in the network model of the previous training stage, and the values ​​of the above three parameters are not all equal to the values ​​of the corresponding parameters in the previous training stage. In this way, it can be ensured that the scale of the network model of the multiple training stages is increased in sequence, so that the model parameter amount of the multiple training stages can be increased in sequence according to the training sequence, thereby realizing the staged fine control of the network model training, and because the parameter amount of the network model is gradually increased with the training stage, this can effectively accelerate the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the training cost of the network model.

[0132] Optionally, the model planning information also includes the model parameter quantities of the corresponding training stage.

[0133] In some embodiments, the model parameter quantity is determined based on the number of self-attention networks in the above-mentioned network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. The model parameter quantity is used to intuitively determine the scale of the network model in the training phase. The difference between the model parameter quantity of the last training phase and the parameter quantity of the target network model input by the above-mentioned user is less than or equal to the parameter quantity difference threshold.

[0134] The parameter quantity difference threshold is set in advance, for example, the parameter quantity difference can be set to 20. In different situations, it can also be adjusted according to different requirements.

[0135] Please refer to Figure 7 , Figure 7 A schematic diagram of a model planning information. Figure 7 The training process of the target network model is divided into training stage 1 to training stage n. The model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, the number of attention heads, and the model parameter amount of the corresponding training stage. Among them, the number of self-attention networks in the network model of training stage 1 is 3, the dimension of the hidden layer in each self-attention network is 384, the number of attention heads is 6, and the model parameter amount is 17 million (million, M), that is, the model parameter amount is 17×10 6 In the network model of training phase 2, the number of self-attention networks is 6, the dimension of the hidden layer in each self-attention network is 512, the number of attention heads is 8, and the number of model parameters is 35M, that is, the number of model parameters is 35×10 6 In the network model of training phase 3, the number of self-attention networks is 12, the dimension of the hidden layer in each self-attention network is 768, the number of attention heads is 12, and the number of model parameters is 110M, that is, the number of model parameters is 110×10 6 The number of self-attention networks in the network model of training stage n is 32, the dimension of the hidden layer in each self-attention network is 4096, the number of attention heads is 32, and the number of model parameters is 7B, that is, the number of model parameters is 7×10 9 .

[0136] 3. Data planning information

[0137] In some embodiments, the data planning information includes the ratio of the training data set of the corresponding training stage to the total training data set, and the total training data set is the sum of the training data sets of multiple training stages.

[0138] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage among multiple training stages.

[0139] For example, the pruning configuration information includes pruning status information, where the pruning status information indicates whether to filter the training data set in the corresponding training phase.

[0140] In some embodiments, the pruning status information may be on or off. If the pruning status information is on, it indicates that the training data set of the corresponding training stage is screened. If the pruning status information is off, it indicates that the training data set of the corresponding training stage is not screened. Of course, in practical applications, other methods may be used to indicate whether the training data set of the corresponding training stage is screened, and the embodiments of the present application are not limited to this.

[0141] In practical applications, pruning can be performed by a variety of methods, such as a pruning method based on a forgetting score, a pruning method based on an error L2 norm (error L2-norm, eL2n), etc. When there are multiple pruning methods and the pruning state information indicates that the training data set of the corresponding training stage is screened, the pruning configuration information also includes indication information of a target pruning method, and the target pruning method is one of the multiple pruning methods.

[0142] It should be noted that, when the training data set of the corresponding training stage is screened, the pruning method of each training stage is the same, that is, the target pruning method is used to screen the training data set in each training stage.

[0143] For example, the configuration information of the distillation preheating includes distillation preheating status information, which indicates whether to perform distillation preheating on the network model to determine the initial values ​​of the model parameters in the first training stage.

[0144] In some embodiments, the distillation preheating status information may be on or off. If the distillation preheating status information is on, it indicates that the network model is distilled and preheated. If the distillation preheating status information is off, it indicates that the network model is not distilled and preheated. Of course, in practical applications, other methods may be used to indicate whether the network model is distilled and preheated, and the embodiments of the present application are not limited to this.

[0145] During the training process of the network model, the appropriateness of the initial values ​​of the model parameters has a great impact on the results. Inappropriate initial values ​​may cause the network model to converge slowly or fail to converge during training. Therefore, in order to accelerate the convergence speed of the network model during the training process and improve the performance of the network model, in some embodiments, the network model can be distilled and preheated before entering the first training stage to determine the initial values ​​of the model parameters for the first training stage.

[0146] In some embodiments, a general corpus package is stored in advance in the cloud platform. When it is determined that the network model is to be distilled and preheated, the network model of the first training stage is pre-trained through the general corpus package before the first training stage to determine the initial values ​​of the model parameters of the first training stage, so as to achieve distillation and preheating of the target network model. Of course, the above-mentioned distillation and preheating of the network model through the general corpus package is only one implementation method. In practical applications, the network model can also be distilled and preheated in other ways, such as distillation and preheating through the Xavier initialization (also known as Glorot initialization) method, etc., which is not limited in the embodiments of the present application.

[0147] It should be noted that, when the distillation preheating status information indicates that the network model is to be distilled and preheated, and there are multiple distillation preheating methods, the distillation preheating configuration information also includes indication information of the target distillation preheating method, and the target distillation preheating method is one of the multiple distillation preheating methods.

[0148] In practical applications, the data planning information may also include other contents such as the storage path of the training data set, which is not limited in the embodiments of the present application.

[0149] For example, see Figure 8 , Figure 8 It is a schematic diagram of data planning information. Figure 8 The training process of the target network model is divided into training stage 1 to training stage 4. The data planning information includes the proportion of the training data set in the corresponding training stage to the total training data set, the configuration information of pruning, the configuration information of distillation preheating, and the storage path of the training data set. The configuration information of pruning includes pruning status information and target pruning method. The configuration information of distillation preheating includes distillation preheating status information. The pruning methods include pruning method 1 and pruning method 2. Pruning method 1 is a pruning method based on forgetting score, and pruning method 2 is a pruning method based on error L2 norm. Among them, the proportion of the training data set in training stage 1 to the total training data set is 25%, the proportion of the training data set in training stage 2 to the total training data set is 25%, the proportion of the training data set in training stage 3 to the total training data set is 25%, and the proportion of the training data set in training stage 4 to the total training data set is 25%. The pruning status information is on, and the target pruning method is pruning method 1. The distillation preheating status information is on. The training data set is stored in the object storage service (OBS), and the storage path of the training data set is mybucket / myfolder.

[0150] The above introduces the detailed contents of the training configuration information of the target network model. Next, the process of determining the training configuration information of the target network model is introduced.

[0151] The parameter quantity and type of the target network model are input into the training prediction model to obtain the first training configuration information output by the training prediction model, and the training configuration information of the target network model is determined based on the first training configuration information.

[0152] That is to say, the embodiments of the present application can specifically determine corresponding training configuration information for network models with different parameter quantities and different types, so that the parameter quantities of the model, the type of model and the determined training configuration information are more adapted.

[0153] In some embodiments, the first training configuration information can be directly determined as the training configuration information of the target network model. In other embodiments, a configuration preview interface can also be provided, the configuration preview interface includes the first training configuration information, and in response to the user's modification operation on the first training configuration information, the first training configuration information is modified to obtain the second training configuration information, and the training configuration information of the target network model is determined based on the second training configuration information.

[0154] That is to say, after determining the first training configuration information based on the parameter quantity and type of the target network model, the user can also modify the first training configuration information to obtain the training configuration information of the target network model, thereby achieving refined customization of the training plan of the target network model and effectively improving the training efficiency of the neural network model.

[0155] For the convenience of description, the number of training stages, model iterations, learning rate range, expected loss value, warm-up iterations, learning rate decay type, parameter accuracy, batch size, random inactivation rate, the number of self-attention networks in the network model, the dimension of the hidden layer in each self-attention network, the number of attention heads, the number of model parameters, the proportion of the training data set in the corresponding training stage to the total training data set, the configuration information of pruning, and the configuration information of distillation preheating are collectively referred to as planning parameters.

[0156] Based on the above description, the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase, and the model parameter quantity is determined based on the number of self-attention networks in the above network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. That is, the values ​​of planning parameters such as the expected loss value and the model parameter quantity need to be determined based on the values ​​of other planning parameters and cannot be directly configured. Therefore, planning parameters such as the expected loss value and the model parameter quantity can also be called dependent variable parameters, and the planning parameters used to determine the dependent variable parameters are called independent variable parameters. Among these planning parameters, the planning parameters that can be directly configured can be called configurable parameters, and usually, users can modify the configurable parameters.

[0157] In some embodiments, the first training configuration information includes values ​​of multiple planning parameters, the second training configuration information also includes values ​​of multiple planning parameters, and the value of at least one configurable parameter among these planning parameters has been modified by the user. That is, the user modifies the value of the at least one configurable parameter in the first training configuration information, and the cloud platform modifies the value of the at least one configurable parameter in response to the user's modification operation.

[0158] In some embodiments, the configurable parameters include the number of model iterations in the corresponding training stage, the learning rate range, the number of warm-up iterations, the learning rate decay type, the parameter accuracy, the batch size, the random inactivation rate, the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, the number of attention heads, the proportion of the training data set in the corresponding training stage to the total training data set, the configuration information of pruning, and at least one of the configuration information of distillation preheating.

[0159] In practical applications, the configurable parameter may also be the number of training stages of the target network model.

[0160] Optionally, the configuration preview interface also includes a stage add button and a stage delete button. The user can add a training stage by triggering the stage add button and delete a training stage by triggering the stage delete button, thereby modifying the number of training stages.

[0161] For example, the stage adding button includes an adding button corresponding to each training stage, and the stage deleting button includes a deleting button corresponding to each training stage. In this case, the training stage added by triggering the adding button corresponding to the corresponding training stage is the next training stage adjacent to the training stage corresponding to the adding button, and the training stage deleted by triggering the adding button corresponding to the corresponding training stage is the training stage corresponding to the deleting button.

[0162] For example, see Fig. 9 , Fig. 9 A schematic diagram of a configuration preview interface provided for an embodiment of the present application, the configuration preview interface includes the number of model iterations, learning rate range, expected loss value, and add buttons and delete buttons corresponding to training stages 1 to 4, respectively. The user adds a new training stage by triggering the add button corresponding to training stage 3. In this case, the training stages of the target network model include training stages 1 to 5.

[0163] It should be noted that if the user modifies the first training configuration information, it means that the user has experience in network model training. Therefore, after increasing or decreasing the training stages, the user also needs to reconfigure the values ​​of other configurable parameters corresponding to the multiple training stages according to his or her own experience. In this case, the at least one configurable parameter modified by the user includes not only the number of training stages, but also other configurable parameters reconfigured by the user except the number of the multiple training stages.

[0164] In actual applications, after modifying the number of training stages, in addition to the user reconfiguring the configurable parameters corresponding to multiple training stages after the increase or decrease of training stages, the number of training stages after the increase or decrease, the parameter amount and type of the target network model can also be input into the training prediction model to obtain the values ​​of the configurable parameters corresponding to the multiple training stages output by the training prediction model.

[0165] In some embodiments, the implementation process of determining the training configuration information of the target network model based on the second training configuration information includes: if the at least one configurable parameter modified by the user includes an independent variable parameter, then based on the value of the modified independent variable parameter, the value of the corresponding dependent variable parameter is re-determined, and the value of the at least one configurable parameter, the re-determined value of the dependent variable parameter, and the values ​​of other configurable parameters included in the first training configuration information except the at least one configurable parameter are determined as the training configuration information of the target network model. If the at least one configurable parameter modified by the user does not include an independent variable parameter, then the value of the at least one configurable parameter and the values ​​of the planning parameters included in the first training configuration information except the at least one configurable parameter are determined as the training configuration information of the target network model.

[0166] Step 304: Train the target network model according to the configuration information of multiple training stages.

[0167] Next, the implementation process of training the target network model according to the configuration information of multiple training stages will be introduced through steps (1) to (4).

[0168] Step (1): Let i=1, and determine the i-th network model based on the configuration information of the i-th training stage in multiple training stages.

[0169] In some embodiments, if the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage; if the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0170] If the i-th training stage is the first training stage among multiple training stages, the implementation process of generating the i-th network model based on the configuration information of the i-th training stage includes: determining the model structure of the first network model based on the model planning information of the first training stage, and determining the initial values ​​of the parameters of the first network model based on the configuration information of distillation preheating, so as to obtain the first network model.

[0171] In some embodiments, the configuration information of the distillation preheating indicates that the network model is distilled and preheated. In this case, before entering the first training phase, the first network model can be distilled and preheated to determine the initial values ​​of the parameters of the first network model.

[0172] If the configuration information of the distillation preheating indicates that the first network model is not to be distilled and preheated, the initial value of the parameter of the first network model can be determined by a random initialization method. Of course, in practical applications, other methods can also be used to determine the initial value of the parameter of the first network model, and the embodiment of the present application does not limit this.

[0173] In some embodiments, the configuration information of the i-th training stage includes the model planning information of the i-th training stage. In this case, based on the configuration information of the i-th training stage, the trained i-1-th network model is amplified to obtain the i-th network model. The implementation process includes: providing a parameter amplification interface, the parameter amplification interface is used to instruct the user to select a parameter amplification strategy, the parameter amplification strategy is used to determine the initial value of the newly added parameters of the i-th network model relative to the i-1-th network model, obtaining the parameter amplification strategy from the parameter amplification interface, and based on the parameter amplification strategy and the model planning information of the i-th training stage, performing parameter amplification on the i-1-th network model to obtain the i-th network model.

[0174] Optionally, the parameter amplification interface also includes a recommended parameter amplification strategy. In this case, after the cloud platform displays the recommended parameter amplification strategy on the parameter amplification interface, the user can select the recommended parameter amplification strategy as the selected parameter amplification strategy, or select a parameter amplification strategy other than the recommended parameter amplification strategy.

[0175] For example, please refer to Fig.10 The parameter amplification interface includes recommended parameter amplification strategies, which are width amplification strategy 1 and depth amplification strategy 1. The user can directly use the recommended parameter amplification strategy as the selected parameter amplification strategy.

[0176] In some embodiments, the cloud platform can store the recommended parameter augmentation strategy. In this case, the cloud platform can directly obtain the recommended parameter augmentation strategy.

[0177] In other embodiments, the cloud platform can input configuration information of multiple training stages and indication information of user interaction operations into the training prediction model to obtain a recommended parameter augmentation strategy output by the training prediction model. The indication information of the user interaction operation can represent the operation performed by the user on the cloud platform. The operation can be a user's modification operation on the first training configuration information, or a user's modification operation on the training evaluation results, etc. The user's modification operation on the training evaluation results will be introduced later and will not be elaborated here.

[0178] In practical applications, in addition to selecting a parameter augmentation strategy from the parameter augmentation interface, the parameter augmentation strategy can also be pre-set and stored in the cloud platform. In this way, the cloud platform can directly obtain the parameter augmentation strategy without the user selecting the parameter augmentation strategy.

[0179] In some embodiments, the parameter augmentation strategy includes a width augmentation strategy and a depth augmentation strategy. The width augmentation strategy indicates a method for determining the initial values ​​of the parameters of the newly added dimensions of the hidden layer in the self-attention network, and the depth augmentation strategy refers to a method for determining the initial values ​​of the parameters in the newly added self-attention network of the i-th network model relative to the i-1-th network model. In this case, based on the parameter augmentation strategy and the model planning information of the i-th training stage, the implementation process of parameter augmentation for the i-1-th network model includes: based on the width augmentation strategy and the model planning information of the i-th training stage, the dimensions of the hidden layers in each self-attention network in the i-1-th network model are augmented to obtain an initial augmented model, and based on the depth augmentation strategy and the model planning information of the i-th training stage, the self-attention network of the initial augmented model is augmented to obtain the i-th network model.

[0180] Based on the width expansion strategy and the model planning information of the i-th training stage, the dimension of the hidden layer in each self-attention network in the i-1-th network model is expanded to obtain the implementation process of the initial expansion model, including: based on the values ​​of the parameters of the neurons in the hidden layer of each self-attention network in the i-1-th network model, according to the width expansion strategy, determining the initial values ​​of the parameters of the newly added neurons in the hidden layer of each self-attention network in the initial expansion model relative to the i-1-th network model, the dimension of the hidden layer in each self-attention network in the initial expansion model is the same as the dimension of the hidden layer in each self-attention network in the network model of the i-th training stage, and the number of self-attention networks in the initial expansion model is the same as the number of self-attention networks in the i-1-th network model.

[0181] For the parameters of the target dimension in the newly added dimension, the width expansion strategy can be to perform a weighted average of the parameter values ​​of the neurons in the hidden layer of each self-attention network in the i-1th network model to obtain the initial value of the parameter of the target neuron, and the target neuron is any neuron in the newly added neurons. Of course, in practical applications, the width expansion strategy can also be other algorithms, such as determining the average value of the parameter values ​​of the neurons in the hidden layer of each self-attention network in the i-1th network model as the initial value of the parameter of the target neuron, etc., and the embodiments of the present application are not limited to this.

[0182] Based on the deep expansion strategy and the model planning information of the i-th training stage, the self-attention network of the initial expansion model is expanded to obtain the i-th network model. The implementation process includes: based on the values ​​of the parameters in the self-attention network in the initial expansion model, according to the deep expansion strategy, determining the initial values ​​of the parameters in the newly added self-attention network in the i-th network model relative to the initial expansion model, the dimension of the hidden layer in the self-attention network in the i-th network model is the same as the dimension of the hidden layer in each self-attention network in the network model in the i-th training stage, and the number of self-attention networks in the i-th network model is the same as the number of self-attention networks in the i-th network model.

[0183] For any parameter in any newly added self-attention network, the deep expansion strategy can be to use the value of the parameter in the same position as the parameter in any self-attention network in the initial amplification model as the initial value of the parameter. Of course, in practical applications, the deep expansion strategy can also be other algorithms, for example, the average value of all parameters in the self-attention network in the initial amplification model that are in the same position as the parameter is used as the initial value of the parameter. The embodiments of the present application are not limited to this.

[0184] Step (2): Train the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model.

[0185] In some embodiments, before the i-th network model is trained according to the configuration information of the i-th training stage to obtain the trained i-th network model, the user can also trigger a start training operation, and the cloud platform can also respond to the start training operation triggered by the user to train the i-th network model according to the configuration information of the i-th training stage.

[0186] For example, before the i-th network model is trained according to the configuration information of the i-th training stage, a first training execution interface may be displayed, and the user may trigger the start of the training operation on the first training execution interface.

[0187] In some embodiments, during the process of training the i-th network model according to the configuration information of the i-th training stage, the user can also trigger a pause training operation, and the cloud platform can also pause the training of the i-th training stage in response to the pause training operation triggered by the user.

[0188] Optionally, during the process of training the i-th network model according to the configuration information of the i-th training stage, the user can also trigger a stop training operation, and the cloud platform can also stop the training of the i-th training stage in response to the stop training operation triggered by the user.

[0189] For example, during the process of training the i-th network model according to the configuration information of the i-th training stage, a second training execution interface may be displayed, and the user may trigger a pause training operation or a stop training operation on the second training execution interface.

[0190] In some embodiments, during the process of training the i-th network model according to the configuration information of the i-th training stage, a third training execution interface may also be displayed, and the third training execution interface includes a training log of the i-th training stage.

[0191] Optionally, the training log includes the number of model iterations, loss function, changes in model parameters, etc. of the network model during the training process.

[0192] Optionally, the third training execution interface also includes a computing amount saving situation, which indicates the computing amount saved by the embodiment of the present application compared with related model training technologies when achieving the same training effect.

[0193] Optionally, the third training execution interface also includes training configuration information of the i-th training stage.

[0194] It should be noted that the above-mentioned first training execution interface, second training execution interface and third training execution interface can be the same interface, or any two of the first training execution interface, the second training execution interface and the third training execution interface are the same interface, or the first training execution interface, the second training execution interface and the third training execution interface are different interfaces, and the embodiments of the present application are not limited to this.

[0195] For example, please refer to Fig.11The first training execution interface, the second training execution interface and the third training execution interface can be the same interface. Before training the nth network model according to the configuration information of the nth training stage, the user can trigger the start training operation in the interface. During the training of the nth network model according to the configuration information of the nth training stage, the user can trigger the pause training operation or stop the training operation. The interface can also display the training configuration, training log and computing power saving of the nth training stage.

[0196] Step (3): Determine the training evaluation result of the trained i-th network model, and provide a training evaluation interface, which includes the training evaluation result of the trained i-th network model, and the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training.

[0197] In some embodiments, a loss curve of the i-th network model is determined, the loss curve indicating how the loss value changes with the number of model iterations, the loss value indicating the gap between the predicted value and the true value of the network model, at least one test data set is input into the trained i-th network model, and an inference result of the at least one test data set output by the trained i-th network model is obtained; based on the loss curve and the inference result of the at least one test data set, a training evaluation result of the trained i-th network model is determined.

[0198] Optionally, during the training of the i-th network model, the cloud platform can record the change of the loss value of the i-th network model with the number of model iterations. In this way, the cloud platform can draw the loss curve of the i-th network model based on the change of the loss value of the i-th network model with the number of model iterations.

[0199] In some embodiments, the implementation process of determining the training evaluation result of the trained i-th network model based on the loss curve and the inference result of at least one test data set includes: determining the inference accuracy corresponding to the at least one test data set based on the inference result of the at least one test data set, and determining the training evaluation result of the trained i-th network model based on the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage.

[0200] For any one of the at least one test data set, the proportion of correctly inferred data in the inference result of the test data set to the test data set is counted to obtain the inference accuracy rate corresponding to the test data set. Each of the at least one test data set is processed in the same manner to obtain the inference accuracy rate corresponding to each test data set.

[0201] Optionally, the training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result, wherein the loss value prediction evaluation result indicates whether the cloud platform's prediction of the loss value of the i-th network model meets the expected prediction, and the test set prediction evaluation result indicates whether the cloud platform's inference result of the at least one test data set meets the expected prediction. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the loss value prediction evaluation result and the test set prediction evaluation result output by the training prediction model.

[0202] Optionally, the training evaluation result also includes training completion prediction information, which indicates the cloud platform's prediction of whether the trained i-th network model has completed training. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the training completion prediction information of the i-th training stage output by the training prediction model.

[0203] Since the training evaluation result also includes the cloud platform's prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who have no training experience, thereby improving the training efficiency of the network model.

[0204] In some embodiments, the training evaluation interface also includes at least one of the loss curve of the ith network model and the inference accuracy rate corresponding to the at least one test data set. In this way, the user can confirm whether the trained ith network model has completed training based on the loss curve of the ith network model and / or the inference accuracy rate corresponding to the at least one test data set, as well as the training evaluation result.

[0205] Since the training evaluation interface also includes at least one of the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, that is, the training evaluation results of the network model obtained in each training stage are displayed to the user in real time through the training evaluation interface, which enables the control of the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training.

[0206] In actual applications, after providing the training evaluation interface, the training evaluation result can also be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result.

[0207] Optionally, the training evaluation interface includes the loss curve of the i-th network model, the inference accuracy corresponding to the at least one test data set, and the training evaluation result. In this case, the user can determine whether the training of the i-th network model meets expectations based on the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set to obtain the user training evaluation result. If the user training evaluation result is inconsistent with the training evaluation result, the user can modify the training evaluation result. The cloud platform responds to the user's modification operation on the training evaluation result and modifies the training evaluation result to obtain the modified training evaluation result. If the user training evaluation result is consistent with the training evaluation result, the user does not modify the training evaluation result.

[0208] Optionally, the user evaluation result includes a loss value user evaluation result and a test set user evaluation result, the loss value user evaluation result indicates whether the loss value of the i-th network model meets the user's expectations, the test set user evaluation result indicates whether the inference accuracy corresponding to the at least one test data set meets the user's expectations, and the training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result. In this case, the user can determine the loss value user evaluation result based on the loss curve of the i-th network model, and determine the test set user evaluation result based on the inference accuracy corresponding to the at least one test data set. If the loss value user evaluation result is inconsistent with the loss value prediction evaluation result, the user can modify the loss value prediction evaluation result, and the cloud platform modifies the loss value prediction evaluation result in response to the user's modification operation on the loss value prediction evaluation result to obtain a modified loss value prediction evaluation result. If the loss value user evaluation result is consistent with the loss value prediction evaluation result, the user does not modify the loss value prediction evaluation result. If the test set user evaluation result is inconsistent with the test set prediction evaluation result, the user can modify the test set prediction evaluation result. The cloud platform responds to the user's modification operation on the test set prediction evaluation result and modifies the test set prediction evaluation result to obtain the modified test set prediction evaluation result. If the test set user evaluation result is consistent with the test set prediction evaluation result, the user does not modify the test set prediction evaluation result.

[0209] For example, if the loss value prediction evaluation result is that the loss value meets expectations, the test set prediction evaluation result is that the test set reasoning accuracy meets expectations, the loss value user evaluation result is that the loss value meets expectations, and the test set user evaluation result is that the test set reasoning accuracy does not meet expectations. In this case, since the test set user evaluation result is inconsistent with the test set prediction evaluation result, the user modifies the test set prediction evaluation result to that the test set reasoning accuracy does not meet expectations. In response to the user's modification operation on the test set prediction evaluation result, the cloud platform modifies the test set prediction evaluation result to obtain a modified test set prediction evaluation result.

[0210] Step (4): If a user-triggered completion training operation is detected and the i-th training stage is not the last training stage among multiple training stages, then let i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among multiple training stages. If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among the multiple training stages, then the trained i-th network model is used as the trained target network model.

[0211] In some embodiments, the user can trigger the completion of training operation on the training evaluation interface when confirming that the trained i-th network model has completed training. In this case, the cloud platform detects the completion of training operation triggered by the user, and when the i-th training stage is not the last training stage among the multiple training stages, let i=i+1, return to the above step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages, and when the i-th training stage is the last training stage among the multiple training stages, use the trained i-th network model as the trained target network model.

[0212] In other embodiments, the user can trigger a continue training operation or a re-planning operation on the training evaluation interface after confirming that the trained i-th network model has not completed training. The continue training operation indicates to continue training the i-th network model, and the re-planning operation indicates to return to the above step of determining the training configuration information of the target network model and re-determine the training configuration information of the target network model. If the user triggers the continue training operation, the cloud platform detects the continue training operation triggered by the user, and then continues to train the i-th network model based on the configuration information of the i-th training stage. If the user triggers the re-planning operation, the cloud platform returns to the above step of determining the training configuration information of the target network model and re-determines the training configuration information of the target network model.

[0213] For example, please refer to Fig.12 The training evaluation interface includes the loss curve of the first network model ( Fig.12), the inference accuracy and training evaluation results corresponding to the test data sets 1-n respectively. The loss value prediction evaluation result is that the loss value meets expectations, the test set prediction evaluation result is that the test set inference accuracy meets expectations, the loss value user evaluation result is that the loss value meets expectations, and the test set user evaluation result is that the test set inference accuracy does not meet expectations. In this case, since the test set user evaluation result is inconsistent with the test set prediction evaluation result, the user modifies the test set prediction evaluation result to that the test set inference accuracy does not meet expectations, and the cloud platform responds to the user's modification operation on the test set prediction evaluation result, modifies the test set prediction evaluation result, and obtains the modified test set prediction evaluation result.

[0214] Of course, the above-mentioned training of the target network model by interacting with the user after obtaining the trained i-th network model is only one implementation method. In practical applications, it is also possible not to interact with the user to achieve the training of the target network model. That is, let i=1, determine the i-th network model based on the configuration information of the i-th training stage in multiple training stages, train the i-th network model according to the configuration information of the i-th training stage, and obtain the trained i-th network model. If the i-th training stage is not the last training stage in the multiple training stages, then let i=i+1, return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage in the multiple training stages, and if the i-th training stage is the last training stage in the multiple training stages, then use the trained i-th network model as the trained target network model.

[0215] For a detailed implementation process of determining the i-th network model based on the configuration information of the i-th training stage in the multiple training stages, please refer to the relevant content above and will not be repeated here.

[0216] In actual applications, if a user-triggered completion training operation is detected and the i-th training stage is not after the last training stage among multiple training stages, the configuration information of at least one training stage after the i-th training stage among multiple training stages may also be optimized. There are multiple implementation methods for optimizing the configuration information of at least one training stage after the i-th training stage among multiple training stages, and two of them are introduced below.

[0217] In a first implementation, configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized based on at least one of the loss curve of the i-th network model and the inference result of the at least one test data set.

[0218] In some embodiments, at least one of the loss curve and the inference results of the at least one test data set is input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0219] In other embodiments, after providing the training evaluation interface, the training evaluation result can also be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result. In this case, based on at least one of the loss curve of the i-th network model and the inference result of at least one test data set, as well as the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0220] In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can timely adjust the configuration information of the subsequent training stage based on the interaction between the user and the cloud platform, the loss curve and the inference results of the test data set, so that the final trained network model is more in line with the user's expectations.

[0221] In response to the user's modification operation on the training evaluation result, the training evaluation result is modified. For the detailed implementation process of obtaining the modified training evaluation result, please refer to the relevant content above and will not be repeated here.

[0222] In some embodiments, the cloud platform inputs the loss curve of the ith network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification into the training prediction model, and obtains the optimized configuration information of at least one training stage after the ith training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the ith training stage.

[0223] In practical applications, after obtaining the trained i-th network model, the cloud platform can also perform a forgetting evaluation on the trained i-th network model to obtain a forgetting evaluation result, which is used to indicate the reasoning of the trained i-th network model on the training data set of the historical training stage, and the historical training stage is a training stage before the i-th training stage in multiple training stages. In this case, if it is determined based on the forgetting evaluation result that the reasoning accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the loss curve of the i-th network model and at least one of the reasoning results of at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification, and the forgetting evaluation result, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized. Otherwise, based on the loss curve and at least one of the reasoning results of at least one test data set, as well as the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized.

[0224] The forgetting evaluation is performed on the trained i-th network model to obtain the forgetting evaluation result, which includes: determining the inference accuracy of at least one target historical data set, the at least one target historical data set corresponds one-to-one to the at least one historical training stage, and the target historical data set is part or all of the training data set of the corresponding historical training stage; based on the inference accuracy of the at least one target historical data set, determining the forgetting evaluation result of the trained i-th network model.

[0225] Optionally, the target historical data set corresponding to the target historical training stage is input into the trained i-th network model to obtain the inference result of the i-th network model on the target historical data set, the target historical training stage is any one of the at least one historical training stage, and the proportion of correctly inferred data in the inference result of the target historical data set to the target historical data set is counted to obtain the inference accuracy of the target historical data set. The at least one historical training stage is processed in the same way to obtain the inference accuracy of at least one target historical data set.

[0226] In some embodiments, the forgetting assessment result is not forgotten or forgotten, not forgotten indicates that there is at least one historical training stage whose reasoning accuracy is not greater than the accuracy threshold, and forgotten indicates that there is at least one historical training stage whose reasoning accuracy is not greater than the accuracy threshold. In this case, if there is at least one historical training stage whose reasoning accuracy is not greater than the accuracy threshold among the respectively corresponding reasoning accuracy of the at least one target historical data set, the forgetting assessment result is determined to be forgotten. If there is no at least one historical training stage whose reasoning accuracy is not greater than the accuracy threshold among the respectively corresponding reasoning accuracy of the at least one target historical data set, the forgetting assessment result is determined to be not forgotten.

[0227] It should be noted that, when the forgetting assessment result is forgotten, the forgetting assessment result also includes configuration information of the forgotten training stage, and the forgotten training stage is a historical training stage in which the inference accuracy in at least one historical training stage is not greater than the accuracy threshold.

[0228] Optionally, the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification, the training evaluation results after modification and the forgetting evaluation results can be input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0229] For a detailed implementation method of optimizing the configuration information of at least one training stage after the i-th training stage in multiple training stages based on the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification, please refer to the relevant content above and will not be repeated here.

[0230] It should be noted that, when the forgetting evaluation result of the trained i-th network model indicates that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, it means that the inference accuracy of the trained i-th network model for the training data set in the historical training stage is low, and there is forgetting. Therefore, the corresponding training data set of at least one training stage after the i-th training stage includes part or all of the training data set of the forgotten training stage, which can ensure that the data set of the forgotten training stage is reviewed in the subsequent training stage, thereby improving the inference performance of the target network model.

[0231] The second implementation method is to perform a forgetfulness evaluation on the trained i-th network model after obtaining the trained i-th network model to obtain a forgetfulness evaluation result. In this case, if it is determined based on the forgetfulness evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the forgetfulness evaluation result, the loss curve of the i-th network model, and at least one of the inference results of at least one test data set, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized. Otherwise, based on at least one of the loss curve of the i-th network model and the inference results of at least one test data set, the configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized.

[0232] For the detailed implementation process of forgetting evaluation for the trained i-th network model, please refer to the relevant content above and will not be repeated here.

[0233] In some embodiments, the loss curve of the i-th network model, at least one of the inference results of at least one test data set, and the forgetting evaluation result are input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0234] For the detailed implementation process of optimizing the configuration information of at least one training stage after the i-th training stage in multiple training stages based on the loss curve of the i-th network model and at least one of the inference results of at least one test data set, please refer to the above introduction and will not be repeated here.

[0235] In some embodiments, if a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, the trained i-th network model is directly used as the trained target network model.

[0236] In practical applications, after obtaining the trained i-th network model, the cloud platform can also perform a forget evaluation on the trained i-th network model to obtain a forget evaluation result. In this case, if a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, and it is determined based on the forget evaluation result that there is no inference accuracy of at least one historical training stage that is not greater than the accuracy threshold, the trained i-th network model is used as the trained target network model.

[0237] In some embodiments, if a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, and based on the forgetting evaluation result, it is determined that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the forgetting evaluation result, the trained i-th network model is continued to be trained to obtain the i-th network model that has been continued to be trained, and the i-th network model that has been continued to be trained is used as the trained i-th network model, and the above-mentioned step of performing forgetting evaluation on the trained i-th network model to obtain the forgetting evaluation result is returned.

[0238] In actual applications, the cloud platform can also display a training record interface, where users can view the configuration information and training records of any training stage. The training record includes the training status of the network model in each training stage.

[0239] For example, the training situation may include a loss curve corresponding to any training stage, an inference accuracy rate corresponding to at least one test data set, a training log, etc., which is not limited in the embodiments of the present application.

[0240] For example, see Fig.13 , Fig.13 A schematic diagram of a training record interface provided in an embodiment of the present application, on which a user can view the configuration information and loss curve of training stage n ( Fig.13 ). The number of model parameters in the configuration information of the training stage n is 35M, the number of self-attention networks in the network model of the nth training stage is 6, the dimension of the hidden layer in each self-attention network is 512, and the number of attention heads is 8.

[0241] Next, we will combine Fig.14 , the network model training method provided in the embodiment of the present application is introduced again.

[0242] Please refer to Fig.14 , provide a parameter configuration interface, obtain the parameter quantity and type of the target network model from the parameter configuration interface, input the parameter quantity and type of the target network model into the training prediction model, and obtain the first training configuration information output by the training prediction model. Based on the first training configuration information, determine the training configuration information of the target network model. The cloud platform provides a configuration preview interface, responds to the user's modification operation on the first training configuration information, modifies the first training configuration information, obtains the second training configuration information, and determines the model planning information, training planning information, and data planning information corresponding to the multiple training stages based on the second training configuration information.

[0243] Let i=1. If the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage. If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0244] The i-th network model is trained according to the configuration information of the i-th training stage to obtain the trained i-th network model, the training evaluation result of the trained i-th network model is determined, and a training evaluation interface is provided. The user can trigger a re-planning operation when confirming that the trained i-th network model has not completed training, return to the above step of determining the training configuration information of the target network model, and re-determine the training configuration information of the target network model.

[0245] The user can trigger the completion of training operation on the training evaluation interface after confirming that the trained ith network model has completed training. The cloud platform detects the completion of training operation triggered by the user, and when the ith training stage is not the last training stage among the multiple training stages, set i=i+1, and return to the step of determining the ith sub-model based on the configuration information of the ith training stage among the multiple training stages. In addition, after obtaining the trained ith network model, the cloud platform can also perform a forget evaluation on the trained ith network model to obtain a forget evaluation result. If the inference accuracy of at least one historical training stage is determined to be not greater than the accuracy threshold based on the forget evaluation result, then based on the loss curve of the ith network model and at least one of the inference results of at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification, and the forget evaluation result, the configuration information of at least one training stage after the ith training stage among the multiple training stages is optimized. Otherwise, based on at least one of the loss curve of the i-th network model and the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0246] In the case where the i-th training stage is the last training stage among multiple training stages, the cloud platform may also perform a forgetting evaluation on the trained i-th network model after obtaining the trained i-th network model to obtain a forgetting evaluation result. If it is determined based on the forgetting evaluation result that there is no inference accuracy of at least one historical training stage that is not greater than the accuracy threshold, then the trained i-th network model is used as the trained target network model, thereby completing the training of the target network model.

[0247] In an embodiment of the present application, a user can input the parameter amount and type of the target network model in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for different parameter amounts and different types of network models in a targeted manner, so that the parameter amount of the model, the type of model and the determined training configuration information are more compatible. In addition, the embodiment of the present application can also divide the training process of a network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training, so that stage-by-stage fine control of network model training can be achieved, and since the parameter amount of the network model is gradually increased with the training stage, it can effectively speed up the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the cost of network model training.

[0248] Since the training evaluation results in the embodiment of the present application also include the cloud platform's prediction of whether the trained i-th network model has completed training, it is possible to provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model. When the training evaluation interface also includes the loss curve of the i-th network model and / or the reasoning accuracy corresponding to at least one test data set, the training evaluation results of the network model obtained in each training stage can be displayed to the user in real time through the training evaluation interface, so that the user can control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training. In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can adjust the configuration information of the subsequent training stages in a timely manner based on the interaction between the user and the cloud platform, the loss curve and the reasoning results of the test data set, so that the final trained network model is more in line with user expectations.

[0249] Fig.15 : is a structural diagram of a network model training device provided in an embodiment of the present application. The network model training device can be implemented by software, hardware or a combination of both to become a cloud platform or part or all of an electronic device with a model training function. Fig.15 The device includes: a providing module 1501, an acquiring module 1502, a determining module 1503 and a training module 1504.

[0250] A module 1501 is provided for providing a parameter configuration interface, which is used to instruct the user to input the parameter amount and type of the target network model to be trained. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0251] The acquisition module 1502 is used to acquire the parameter quantity and type of the target network model from the parameter configuration interface. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0252] The determination module 1503 is used to determine the training configuration information of the target network model based on the parameter amount and type of the target network model; wherein the training process of the target network model is divided into multiple training stages, and the model parameter amounts corresponding to different training stages are different, and the model parameter amounts of the multiple training stages are increased in sequence according to the training sequence, and the training configuration information includes the configuration information of the multiple training stages. The detailed implementation process refers to the corresponding content in the above-mentioned embodiments, which will not be repeated here.

[0253] The training module 1504 is used to train the target network model according to the configuration information of multiple training stages. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0254] Optionally, the determining module 1503 is specifically configured to:

[0255] Inputting the parameter quantity and type of the target network model into the training prediction model to obtain the first training configuration information output by the training prediction model;

[0256] Based on the first training configuration information, training configuration information of the target network model is determined.

[0257] Optionally, the determining module 1503 is specifically configured to:

[0258] Providing a configuration preview interface, the configuration preview interface including first training configuration information;

[0259] In response to a user's modification operation on the first training configuration information, modify the first training configuration information to obtain second training configuration information;

[0260] Based on the second training configuration information, training configuration information of the target network model is determined.

[0261] Optionally, the configuration information of the multiple training stages includes model planning information, training planning information and data planning information corresponding to the multiple training stages respectively;

[0262] Among them, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration of the training data of the corresponding training stage.

[0263] Optionally, when the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0264] Optionally, the model planning information also includes the model parameter quantities of the corresponding training stage.

[0265] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of a model parameter in the corresponding training phase.

[0266] Optionally, the training planning information also includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0267] Optionally, the training planning information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates the changing manner of the changing step size of the model parameters, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0268] Optionally, the data planning information includes a ratio of a training data set of a corresponding training stage to a total training data set, where the total training data set is the sum of training data sets of multiple training stages.

[0269] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage among multiple training stages.

[0270] Optionally, the training module 1504 is specifically used for:

[0271] Let i=1, determine the i-th network model based on the configuration information of the i-th training stage in the multiple training stages;

[0272] Training the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model;

[0273] Determine the training evaluation result of the trained i-th network model;

[0274] Providing a training evaluation interface, the training evaluation interface including the training evaluation result of the trained i-th network model, the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training;

[0275] If a user-triggered completion training operation is detected and the i-th training stage is not the last training stage among the multiple training stages, then i=i+1 is set, and the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages is returned;

[0276] If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, the trained i-th network model is used as the trained target network model.

[0277] Optionally, the training module 1504 is specifically used for:

[0278] If the i-th training stage is the first training stage among multiple training stages, generating the i-th network model based on the configuration information of the i-th training stage;

[0279] If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0280] Optionally, the configuration information of the i-th training stage includes model planning information of the i-th training stage;

[0281] The training module 1504 is specifically used for:

[0282] Providing a parameter amplification interface, the parameter amplification interface is used to instruct the user to select a parameter amplification strategy, the parameter amplification strategy is used to determine the initial value of the parameter newly added to the i-th network model relative to the i-1-th network model;

[0283] Obtain parameter amplification strategy from the parameter amplification interface;

[0284] Based on the parameter augmentation strategy and the model planning information of the i-th training stage, the parameters of the i-1-th network model are augmented to obtain the i-th network model.

[0285] Optionally, the training module 1504 is specifically used for:

[0286] Determine the loss curve of the i-th network model, where the loss curve indicates how the loss value changes with the number of model iterations, and the loss value indicates the gap between the predicted value and the true value of the network model;

[0287] Inputting at least one test data set into the trained i-th network model to obtain an inference result of at least one test data set output by the trained i-th network model;

[0288] Based on the loss curve and the inference result of at least one test data set, a training evaluation result of the trained i-th network model is determined.

[0289] Optionally, the training module 1504 is specifically used for:

[0290] Based on at least one of the loss curve and the inference result of at least one test data set, configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized.

[0291] Optionally, the training module 1504 is specifically used for:

[0292] In response to a modification operation of the user on the training evaluation result, the training evaluation result is modified to obtain a modified training evaluation result;

[0293] Based on at least one of the loss curve and the inference result of at least one test data set, optimizing configuration information of at least one training stage after the i-th training stage among the multiple training stages, including:

[0294] Based on at least one of the loss curve and the inference result of at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized.

[0295] Optionally, the device further comprises:

[0296] A forgetting evaluation module is used to perform a forgetting evaluation on the trained ith network model to obtain a forgetting evaluation result, wherein the forgetting evaluation result is used to indicate the reasoning of the trained ith network model on the training data set of a historical training stage, wherein the historical training stage is a training stage before the ith training stage among multiple training stages;

[0297] The training module 1504 is specifically used for:

[0298] If it is determined based on the forgetting evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on at least one of the loss curve and the inference results of at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification, and the forgetting evaluation result, the configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

[0299] In an embodiment of the present application, a user can input the parameter amount and type of the target network model in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for different parameter amounts and different types of network models in a targeted manner, so that the parameter amount of the model, the type of model and the determined training configuration information are more compatible. In addition, the embodiment of the present application can also divide the training process of a network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training, so that stage-by-stage fine control of network model training can be achieved, and since the parameter amount of the network model is gradually increased with the training stage, it can effectively speed up the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the cost of network model training.

[0300] Since the training evaluation results in the embodiment of the present application also include the cloud platform's prediction of whether the trained i-th network model has completed training, it is possible to provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model. When the training evaluation interface also includes the loss curve of the i-th network model and / or the reasoning accuracy corresponding to at least one test data set, the training evaluation results of the network model obtained in each training stage can be displayed to the user in real time through the training evaluation interface, so that the user can control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training. In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can adjust the configuration information of the subsequent training stages in a timely manner based on the interaction between the user and the cloud platform, the loss curve and the reasoning results of the test data set, so that the final trained network model is more in line with user expectations.

[0301] It should be noted that: the network model training device provided in the above embodiment only uses the division of the above functional modules as an example when training the network model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the network model training device provided in the above embodiment and the network model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0302] Fig.15The providing module, acquiring module, determining module and training module in the training device of the network model shown can be implemented by software, or can be implemented by hardware, or can be implemented by a combination of software and hardware. Next, taking the providing module in the training device of the network model as an example, the implementation method of the providing module is introduced. Similarly, the implementation methods of the above-mentioned other modules can refer to the implementation method of the providing module.

[0303] As an example of a software functional unit, a module may include code running on a computing instance. The computing instance may include at least one of a physical host, a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, a module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, a region may include multiple AZs.

[0304] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple virtual private clouds (VPCs). Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0305] As an example of a hardware functional unit, the module provided may include at least one computing device, such as a server, etc. Alternatively, the module provided may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0306] The multiple computing devices included in the provision module can be distributed in the same region or in different regions. The multiple computing devices included in the provision module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the provision module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0307] In other embodiments, the providing module in the training device for the network model can be used to execute any step in the training method for the network model mentioned above, the acquiring module in the training device for the network model can be used to execute any step in the training method for the network model mentioned above, the determining module in the training device for the network model can be used to execute any step in the training method for the network model mentioned above, and the training module in the training device for the network model can be used to execute any step in the training method for the network model mentioned above. The steps that the providing module, acquiring module, determining module and training module in the training device for the network model are responsible for implementing can be specified as needed. Different steps in the training method for the network model mentioned above are respectively implemented by the providing module, acquiring module, determining module and training module in the training device for the network model, thereby realizing all the functions of the training device for providing the network model.

[0308] The present application embodiment also provides a computing device 100. Fig.16 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0309] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.18 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0310] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an integrated circuit, etc. The integrated circuit may be an ASIC, a PLD, or a combination thereof. Optionally, the PLD may be a CPLD, an FPGA, a GAL, or any combination thereof.

[0311] The memory 106 may include a volatile memory, such as a random access memory (RAM). The memory 106 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 106 exists independently and is connected to the processor 104 through the bus 102, or the memory 106 is integrated with the processor 104.

[0312] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to implement the aforementioned network model training method. That is, the memory 106 stores instructions for executing the network model training method.

[0313] For example, the memory 106 stores executable code, and the processor 104 executes the executable code to implement the above Fig.16 The functions of the providing module, the acquiring module, the determining module and the training module are shown. That is, the memory 106 stores instructions for executing the training method of the network model provided in the embodiment of the present application.

[0314] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or communication networks. The network interface includes a wired communication interface, or also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0315] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes multiple computing devices. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0316] like Fig.17 As shown, the computing device cluster includes multiple computing devices 100. The memory 106 in the multiple computing devices 100 in the computing device cluster may store the same instructions for executing the training method of the above network model.

[0317] In some possible implementations, the memory 106 of the multiple computing devices 100 in the computing device cluster may also respectively store partial instructions for executing the training method of the above network model. In other words, the combination of the multiple computing devices 100 can jointly execute all instructions for executing the training method of the above network model.

[0318] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing part of the functions of the above-mentioned network model training method. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of some or all modules included in the above-mentioned network model training device.

[0319] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. Fig.18 A possible implementation is shown, such as Fig.18 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of a portion of the modules included in the training device for the network model described above. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of another portion of the modules included in the training device for the network model described above.

[0320] It should be understood that Fig.18 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0321] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.17 and Fig.18 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the training method of the network model.

[0322] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the training method of the network model. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the training method of the network model.

[0323] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a computer, the computer executes the steps of the network model training method described in the above embodiment, or executes the steps of the network model training method described in the above embodiment.

[0324] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the network model training method described in the above embodiment. In other words, a computer program is provided, which, when executed on a computer, enables the computer to execute the steps of the network model training method described in the above embodiment.

[0325] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0326] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0327] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the parameter quantities and types of the target network model involved in the embodiments of the present application are obtained with full authorization.

[0328] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A network model training method, characterized in that: The method comprises: Providing a parameter configuration interface, wherein the parameter configuration interface is used to instruct the user to input the parameter amount and type of the target network model to be trained; Acquire the parameter quantity and type of the target network model from the parameter configuration interface; Based on the parameter amount and type of the target network model, determine the training configuration information of the target network model; wherein the training process of the target network model is divided into multiple training stages, different training stages correspond to different model parameter amounts, and the model parameter amounts of the multiple training stages are increased in sequence according to the training sequence, and the training configuration information includes the configuration information of the multiple training stages; The target network model is trained according to the configuration information of the multiple training stages.

2. The method according to claim 1, characterized in that The determining, based on the parameter amount and type of the target network model, training configuration information of the target network model includes: Inputting the parameter quantity and type of the target network model into a training prediction model to obtain first training configuration information output by the training prediction model; Based on the first training configuration information, training configuration information of the target network model is determined.

3. The method according to claim 2, characterized in that The determining, based on the first training configuration information, the training configuration information of the target network model includes: Providing a configuration preview interface, wherein the configuration preview interface includes the first training configuration information; In response to the user's modification operation on the first training configuration information, modify the first training configuration information to obtain second training configuration information; Based on the second training configuration information, training configuration information of the target network model is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The configuration information of the multiple training stages includes model planning information, training planning information and data planning information respectively corresponding to the multiple training stages; Among them, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration of the training data of the corresponding training stage.

5. The method according to claim 4, characterized in that When the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

6. The method according to claim 5, characterized in that The model planning information also includes the model parameter quantities of the corresponding training stage.

7. The method according to any one of claims 4 to 6, characterized in that: The training planning information includes the number of model iterations and the learning rate interval of the corresponding training stage, and the learning rate interval indicates the range of the change step size of the model parameters of the corresponding training stage.

8. The method according to claim 7, characterized in that The training planning information also includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

9. The method according to claim 7 or 8, characterized in that The training planning information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates the changing manner of the changing step size of the model parameters, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage among the multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

10. The method according to any one of claims 4 to 9, characterized in that: The data planning information includes the ratio of the training data set of the corresponding training stage to the total training data set, and the total training data set is the sum of the training data sets of the multiple training stages.

11. The method according to claim 10, characterized in that The data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage among the multiple training stages.

12. The method according to claim 1, characterized in that The training of the target network model according to the configuration information of the multiple training stages includes: Let i=1, determine the i-th network model based on the configuration information of the i-th training stage in the multiple training stages; Training the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model; Determine the training evaluation result of the trained i-th network model; Providing a training evaluation interface, the training evaluation interface including the training evaluation result of the trained i-th network model, the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training; If the training completion operation triggered by the user is detected and the i-th training stage is not the last training stage among the multiple training stages, then set i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages; If the completion of the training operation triggered by the user is detected and the i-th training stage is the last training stage among the multiple training stages, the trained i-th network model is used as the trained target network model.

13. The method according to claim 12, characterized in that The determining, based on the configuration information of the i-th training stage in the multiple training stages, the i-th network model comprises: If the i-th training stage is the first training stage among the multiple training stages, generating the i-th network model based on the configuration information of the i-th training stage; If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

14. The method according to claim 13, characterized in that The configuration information of the i-th training stage includes the model planning information of the i-th training stage; The step of amplifying the trained (i-1)th network model based on the configuration information of the (i)th training stage to obtain the (i)th network model comprises: Providing a parameter augmentation interface, the parameter augmentation interface is used to instruct the user to select a parameter augmentation strategy, the parameter augmentation strategy is used to determine the initial value of the parameter newly added to the i-th network model relative to the i-1-th network model; Acquiring the parameter amplification strategy from the parameter amplification interface; Based on the parameter augmentation strategy and the model planning information of the i-th training stage, the parameters of the i-1-th network model are augmented to obtain the i-th network model.

15. The method according to claim 12, characterized in that The step of determining the training evaluation result of the trained i-th network model comprises: Determine a loss curve of the i-th network model, wherein the loss curve indicates how a loss value changes with the number of model iterations, and the loss value indicates a gap between a predicted value and a true value of the network model; Inputting at least one test data set into the trained i-th network model to obtain an inference result of the at least one test data set output by the trained i-th network model; Based on the loss curve and the inference result of the at least one test data set, a training evaluation result of the trained i-th network model is determined.

16. The method according to claim 15, characterized in that If the completion of the training operation triggered by the user is detected and the i-th training stage is not the last training stage among the multiple training stages, the method further includes: Based on at least one of the loss curve and the inference result of the at least one test data set, configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

17. The method according to claim 16, characterized in that After providing the training evaluation interface, the method further includes: In response to the user's modification operation on the training evaluation result, modify the training evaluation result to obtain a modified training evaluation result; The optimizing, based on at least one of the loss curve and the inference result of the at least one test data set, configuration information of at least one training stage after the i-th training stage among the multiple training stages comprises: Based on at least one of the loss curve and the inference result of the at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

18. The method according to claim 17, characterized in that The method further comprises: Performing a forgetting evaluation on the trained i-th network model to obtain a forgetting evaluation result, wherein the forgetting evaluation result is used to indicate the reasoning of the trained i-th network model on the training data set of a historical training stage, wherein the historical training stage is a training stage before the i-th training stage among the multiple training stages; The optimizing, based on at least one of the loss curve and the inference result of the at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, configuration information of at least one training stage after the i-th training stage in the multiple training stages comprises: If it is determined based on the forgetting evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on at least one of the loss curve and the inference results of the at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification and the forgetting evaluation result, the configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

19. A network model training device, characterized in that: The device comprises: A module is provided for providing a parameter configuration interface, wherein the parameter configuration interface is used to instruct a user to input the parameter amount and type of a target network model to be trained; An acquisition module, used to acquire the parameter quantity and type of the target network model from the parameter configuration interface; A determination module, configured to determine the training configuration information of the target network model based on the parameter amount and type of the target network model; wherein the training process of the target network model is divided into a plurality of training stages, the model parameter amounts corresponding to different training stages are different, and the model parameter amounts of the plurality of training stages are increased in sequence according to the training sequence, and the training configuration information includes the configuration information of the plurality of training stages; A training module is used to train the target network model according to the configuration information of the multiple training stages.

20. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-18.

21. A computer-readable storage medium, characterized in that: The storage medium includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 18.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 18.

Images