Network model training method, cloud platform and related apparatus

Abstract

The present application belongs to the technical field of artificial intelligence. Disclosed are a network model training method, a cloud platform and a related apparatus. The method comprises: providing a parameter configuration interface, and acquiring from the parameter configuration interface a parameter quantity of and the type of a target network model; on the basis of the parameter quantity and the type of the target network model, determining training configuration information of the target network model, a training process of the target network model being divided into a plurality of training stages, different training stages corresponding to different model parameter quantities, the model parameter quantities of the plurality of training stages being successively increased in the order of training, and the training configuration information comprising configuration information of the plurality of training stages; and, according to the configuration information of the plurality of training stages, training the target network model. Thus, the present invention can implement stage-based fine control of network model training, and can also effectively increase the convergence speed of network models, save the amount of calculation required by network model training, and reduce network model training costs.

Description

Network model training method, cloud platform and related devices

[0001] This application claims priority to Chinese patent application No. 202311061555.2 filed on August 22, 2023, with invention name “Model training method, cloud platform and related devices”, and claims priority to Chinese patent application No. 202311393706.4 filed on October 25, 2023, with invention name “Network model training method, cloud platform and related devices”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a network model training method, cloud platform and related devices. Background Art

[0003] With the continuous development and improvement of artificial intelligence technology, neural network models have been widely used in fields such as natural language processing, computer vision, and speech recognition. Because large-parameter (also known as large-scale) neural network models can better handle complex tasks and have better generalization capabilities, the number of parameters in neural network models is increasing, and the demand for training large-parameter neural network models is also gradually increasing.

[0004] Currently, neural network models are primarily trained through cloud computing platforms. However, existing cloud computing platforms offer limited training capabilities and are unable to adapt to the demands of the ever-expanding number of parameters. Training a neural network model with a large number of parameters requires extended training time and a greater computational load, resulting in high training costs and low efficiency.

[0005] Summary of the Invention

[0006] This application provides a network model training method, apparatus, device, storage medium, and computer program that can solve the problem of low model training efficiency in related technologies. The technical solution is as follows:

[0007] In a first aspect, a network model training method is provided, the method comprising: providing a parameter configuration interface, the parameter configuration interface being used to instruct a user to input parameter amounts and types of a target network model to be trained, obtaining the parameter amounts and types of the target network model from the parameter configuration interface, and determining training configuration information of the target network model based on the parameter amounts and types of the target network model, wherein the training process of the target network model is divided into multiple training stages, different training stages correspond to different model parameter amounts, and the model parameter amounts of the multiple training stages increase sequentially in a training order, the training configuration information including configuration information of the multiple training stages; and training the target network model according to the configuration information of the multiple training stages.

[0008] In this application, the user can input the parameter amount and type of the target network model in the parameter configuration interface, and thus determine the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models with different parameter amounts and different types in a targeted manner, so that the parameter amount of the model, the type of model and the determined training configuration information are more adapted. In addition, the present application can also divide the training process of a network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training. In this way, fine control of the network model training can be achieved in stages, and since the parameter amount of the network model gradually increases with the training stage, it can effectively speed up the convergence speed of the network model, save the amount of calculation required for network model training, and reduce the cost of network model training.

[0009] Optionally, the configuration information of the multiple training stages includes model planning information, training planning information and data planning information corresponding to the multiple training stages respectively, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration status of the training data of the corresponding training stage.

[0010] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of the model parameters in the corresponding training phase.

[0011] Optionally, the training planning information further includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0012] It should be noted that the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase.

[0013] Optionally, the training plan information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates how the step size of the model parameters changes, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0014] Optionally, the training planning information may further include a random dropout rate, where the random dropout rate refers to the proportion of neurons that do not participate in calculations in each iteration when training a neural network.

[0015] When the type of the network model is different, the parameters included in the model planning information are different. Optionally, when the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0016] Optionally, for any training stage other than the first training stage among the multiple training stages, the value of any parameter included in the model planning information of the training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and, among the parameters included in the model planning information of the training stage, there is at least one parameter whose value is greater than the value of the corresponding parameter in the previous training stage. In other words, for two adjacent training stages among the multiple training stages, the value of the parameter included in the model planning information of the latter training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and the values ​​of the parameters included in the model planning information of the two training stages are not completely equal.

[0017] Optionally, the model planning information also includes the model parameter quantity for the corresponding training stage. The model parameter quantity is determined based on the number of self-attention networks in the above-mentioned network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. The model parameter quantity is used to intuitively determine the scale of the network model in the training stage. The difference between the model parameter quantity of the last training stage and the parameter quantity of the target network model input by the user is less than or equal to the parameter quantity difference threshold.

[0018] Optionally, the data planning information includes a ratio of the training data set of the corresponding training stage to the total training data set, where the total training data set is the sum of the training data sets of multiple training stages.

[0019] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage in multiple training stages.

[0020] Optionally, the pruning configuration information includes pruning status information, where the pruning status information indicates whether to filter the training data set of the corresponding training phase.

[0021] Optionally, the data planning information may also include a storage path for the training data set.

[0022] The parameter quantity and type of the target network model are input into the training prediction model to obtain the first training configuration information output by the training prediction model. Based on the first training configuration information, the training configuration information of the target network model is determined. In other words, the present application can specifically determine the corresponding training configuration information for network models with different parameter quantities and different types, so that the parameter quantity of the model, the type of the model and the determined training configuration information are more adapted.

[0023] Optionally, a configuration preview interface is provided, which includes first training configuration information. In response to the user's modification operation on the first training configuration information, the first training configuration information is modified to obtain second training configuration information. Based on the second training configuration information, the training configuration information of the target network model is determined.

[0024] That is to say, after determining the first training configuration information based on the parameter quantity and type of the target network model, the user can also modify the first training configuration information to obtain the training configuration information of the target network model, thereby realizing the refined customization of the training plan of the target network model and effectively improving the training efficiency of the neural network model.

[0025] For ease of description, the number of training stages, number of model iterations, learning rate range, expected loss value, number of warm-up iterations, learning rate decay type, parameter accuracy, batch size, random inactivation rate, number of self-attention networks in the network model, dimension of hidden layers in each self-attention network, number of attention heads, number of model parameters, proportion of the training data set in the corresponding training stage to the total training data set, pruning configuration information, and distillation preheating configuration information are collectively referred to as planning parameters.

[0026] Based on the description above, the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase. The number of model parameters is determined based on the number of self-attention networks in the above network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. In other words, the values ​​of planning parameters such as the expected loss value and the number of model parameters need to be determined based on the values ​​of other planning parameters and cannot be directly configured. Therefore, planning parameters such as the expected loss value and the number of model parameters can also be called dependent variable parameters, and the planning parameters used to determine the dependent variable parameters are called independent variable parameters. Among these planning parameters, those that can be directly configured are called configurable parameters. Generally, users can modify these configurable parameters.

[0027] Optionally, the first training configuration information includes values ​​of multiple planning parameters, and the second training configuration information also includes values ​​of multiple planning parameters, and the value of at least one configurable parameter among these planning parameters has been modified by the user. That is, the user modifies the value of the at least one configurable parameter in the first training configuration information, and in response to the user's modification operation, the value of the at least one configurable parameter is modified.

[0028] In practical applications, the configurable parameter may also be the number of training stages of the target network model.

[0029] Optionally, the configuration preview interface also includes a stage add button and a stage delete button. The user can add a training stage by triggering the stage add button and delete a training stage by triggering the stage delete button, thereby modifying the number of training stages.

[0030] Optionally, the implementation process of determining the training configuration information of the target network model based on the second training configuration information includes: if the at least one configurable parameter modified by the user includes an independent variable parameter, then based on the value of the modified independent variable parameter, the value of the corresponding dependent variable parameter is re-determined, and the value of the at least one configurable parameter, the value of the re-determined dependent variable parameter, and the values ​​of other configurable parameters included in the first training configuration information other than the at least one configurable parameter are determined as the training configuration information of the target network model. If the at least one configurable parameter modified by the user does not include an independent variable parameter, then the value of the at least one configurable parameter and the values ​​of the planning parameters included in the first training configuration information other than the at least one configurable parameter are determined as the training configuration information of the target network model.

[0031] Next, the implementation process of training the target network model according to the configuration information of multiple training stages will be introduced through steps (1) to (4).

[0032] Step (1): Let i=1, and determine the i-th network model based on the configuration information of the i-th training stage in multiple training stages.

[0033] If the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage; if the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0034] Optionally, the configuration information of the i-th training stage includes the model planning information of the i-th training stage. In this case, based on the configuration information of the i-th training stage, the trained i-1-th network model is amplified to obtain the i-th network model. The implementation process includes: providing a parameter amplification interface, the parameter amplification interface is used to instruct the user to select a parameter amplification strategy, the parameter amplification strategy is used to determine the initial values ​​of the parameters newly added to the i-th network model relative to the i-1-th network model, obtaining the parameter amplification strategy from the parameter amplification interface, and based on the parameter amplification strategy and the model planning information of the i-th training stage, performing parameter amplification on the i-1-th network model to obtain the i-th network model.

[0035] Optionally, the parameter augmentation strategy includes a width augmentation strategy and a depth augmentation strategy, wherein the width augmentation strategy indicates a method for determining the initial values ​​of the parameters of the newly added dimensions of the hidden layer in the self-attention network, and the depth augmentation strategy refers to a method for determining the initial values ​​of the parameters in the newly added self-attention network of the i-th network model relative to the i-1-th network model. In this case, based on the parameter augmentation strategy and the model planning information of the i-th training stage, the implementation process of parameter augmentation for the i-1-th network model includes: based on the width augmentation strategy and the model planning information of the i-th training stage, augmenting the dimensions of the hidden layers in each self-attention network in the i-1-th network model to obtain an initial augmented model, and based on the depth augmentation strategy and the model planning information of the i-th training stage, augmenting the self-attention network of the initial augmented model to obtain the i-th network model.

[0036] Based on the values ​​of the parameters of the neurons in the hidden layer of each self-attention network in the i-1th network model, according to the width expansion strategy, the initial values ​​of the parameters of the newly added neurons in the hidden layer of each self-attention network in the initial augmented model relative to the i-1th network model are determined, the dimension of the hidden layer of each self-attention network in the initial augmented model is the same as the dimension of the hidden layer of each self-attention network in the network model in the i-th training stage, and the number of self-attention networks in the initial augmented model is the same as the number of self-attention networks in the i-1th network model.

[0037] Based on the values ​​of the parameters in the self-attention network in the initial augmented model, according to the depth expansion strategy, determine the initial values ​​of the parameters in the newly added self-attention network in the i-th network model relative to the initial augmented model, the dimension of the hidden layer in the self-attention network in the i-th network model is the same as the dimension of the hidden layer in each self-attention network in the network model in the i-th training stage, and the number of self-attention networks in the i-th network model is the same as the number of self-attention networks in the i-th network model.

[0038] Step (2): Train the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model.

[0039] Before the i-th network model is trained according to the configuration information of the i-th training stage to obtain the trained i-th network model, the user can also trigger the start training operation. In this case, in response to the start training operation triggered by the user, the i-th network model is trained according to the configuration information of the i-th training stage.

[0040] Step (3): Determine the training evaluation result of the trained i-th network model and provide a training evaluation interface, which includes the training evaluation result of the trained i-th network model. The training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training.

[0041] Determine a loss curve for the i-th network model, where the loss curve indicates how the loss value changes with the number of model iterations, and the loss value indicates the gap between the predicted value and the true value of the network model. Input at least one test data set into the trained i-th network model, and obtain an inference result of the at least one test data set output by the trained i-th network model. Based on the loss curve and the inference result of the at least one test data set, determine a training evaluation result of the trained i-th network model.

[0042] Optionally, during the training of the i-th network model, the change of the loss value of the i-th network model with the number of model iterations is recorded. In this way, the loss curve of the i-th network model can be drawn according to the change of the loss value of the i-th network model with the number of model iterations.

[0043] Based on the inference result of the at least one test data set, determine the inference accuracy corresponding to the at least one test data set respectively; based on the inference accuracy corresponding to the at least one test data set respectively, the loss curve of the i-th network model and the expected loss value of the i-th training stage, determine the training evaluation result of the trained i-th network model.

[0044] Optionally, the training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result, wherein the loss value prediction evaluation result indicates whether the loss value of the i-th network model meets the expected prediction, and the test set prediction evaluation result indicates whether the inference result of the at least one test data set meets the expected prediction. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the loss value prediction evaluation result and the test set prediction evaluation result output by the training prediction model.

[0045] Optionally, the training evaluation result also includes training completion prediction information, which indicates a prediction of whether the trained i-th network model has completed training. In this case, the inference accuracy corresponding to the at least one test dataset, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model, and the training completion prediction information of the i-th training stage is output by the training prediction model.

[0046] Since the training evaluation result also includes a prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model.

[0047] Optionally, the training evaluation interface further includes at least one of a loss curve of the i-th network model and the inference accuracy corresponding to the at least one test dataset. In this way, the user can confirm whether the trained i-th network model has completed training based on the loss curve of the i-th network model and / or the inference accuracy corresponding to the at least one test dataset, as well as the training evaluation results.

[0048] Since the training evaluation interface also includes at least one of the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, that is, the training evaluation results of the network model obtained in each training stage are displayed to the user in real time through the training evaluation interface. This makes it possible to control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training.

[0049] In actual applications, after providing the training evaluation interface, the training evaluation result can be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result.

[0050] Optionally, the training evaluation interface includes the loss curve of the i-th network model, the inference accuracy corresponding to the at least one test data set, and the training evaluation result. In this case, the user can determine whether the training of the i-th network model meets expectations based on the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, so as to obtain the user training evaluation result. If the user training evaluation result is inconsistent with the training evaluation result, the user can modify the training evaluation result. In response to the user's modification operation on the training evaluation result, the training evaluation result is modified to obtain the modified training evaluation result. If the user training evaluation result is consistent with the training evaluation result, the user does not modify the training evaluation result.

[0051] Step (4): If a user-triggered completion training operation is detected and the i-th training stage is not the last training stage among multiple training stages, then let i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among multiple training stages. If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among the multiple training stages, then the trained i-th network model is used as the trained target network model.

[0052] Optionally, the user can trigger the completion of training operation on the training evaluation interface when confirming that the trained i-th network model has completed training. In this case, the completion of training operation triggered by the user is detected, and when the i-th training stage is not the last training stage among the multiple training stages, let i=i+1, and return to the above-mentioned step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages. When the i-th training stage is the last training stage among the multiple training stages, the trained i-th network model is used as the trained target network model.

[0053] Optionally, when confirming that the trained i-th network model has not completed training, the user can trigger a continue training operation or a re-planning operation on the training evaluation interface. The continue training operation indicates to continue training the i-th network model, and the re-planning operation indicates to return to the above-mentioned step of determining the training configuration information of the target network model and re-determine the training configuration information of the target network model. If the user triggers the continue training operation, the user-triggered continue training operation is detected, and then the i-th network model is continued to be trained based on the configuration information of the i-th training stage. If the user triggers the re-planning operation, the step of determining the training configuration information of the target network model is returned to and the training configuration information of the target network model is re-determined.

[0054] If a user-triggered training completion operation is detected and the i-th training phase is not the last training phase among the multiple training phases, the configuration information of at least one training phase after the i-th training phase among the multiple training phases may be optimized. There are multiple implementations for optimizing the configuration information of at least one training phase after the i-th training phase among the multiple training phases, two of which are described below.

[0055] In a first implementation, configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized based on the loss curve of the i-th network model and at least one of the inference results of the at least one test data set.

[0056] Optionally, at least one of the loss curve and the inference results of the at least one test data set is input into a training prediction model to obtain the optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby optimizing the configuration information of at least one training stage after the i-th training stage.

[0057] Optionally, after providing the training evaluation interface, the training evaluation result can be modified in response to a user's modification operation on the training evaluation result to obtain a modified training evaluation result. In this case, based on at least one of the loss curve of the i-th network model and the inference result of at least one test dataset, as well as the pre-modification training evaluation result and the modified training evaluation result, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0058] Since the determined training evaluation results may not meet the user's expectations, in this case, the user can modify the training evaluation results. In this way, based on the interaction between the user and the execution entity of this application, the loss curve and the inference results of the test data set, the configuration information of the subsequent training stage can be adjusted in time, so that the final trained network model is more in line with the user's expectations.

[0059] Optionally, the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification are input into the training prediction model to obtain the optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby achieving optimization of the configuration information of at least one training stage after the i-th training stage.

[0060] Optionally, after obtaining the trained i-th network model, a forgetfulness assessment is performed on the trained i-th network model to obtain a forgetfulness assessment result. The forgetfulness assessment result is used to indicate the inference performance of the trained i-th network model on a training dataset from a historical training phase, where the historical training phase is a training phase prior to the i-th training phase in the multiple training phases. In this case, if the inference accuracy of at least one historical training phase is determined to be no greater than an accuracy threshold based on the forgetfulness assessment result, then the configuration information of at least one training phase subsequent to the i-th training phase in the multiple training phases is optimized based on the loss curve of the i-th network model and at least one of the inference results of at least one test dataset, as well as the pre-modification training assessment result, the post-modification training assessment result, and the forgetfulness assessment result. Otherwise, the configuration information of at least one training phase subsequent to the i-th training phase in the multiple training phases is optimized based on the loss curve and at least one of the inference results of at least one test dataset, as well as the pre-modification training assessment result and the post-modification training assessment result.

[0061] Optionally, the forgetting assessment result is not forgotten or forgotten, not forgotten indicates that there is at least one historical training stage with an inference accuracy rate not greater than the accuracy threshold, and forgotten indicates that there is at least one historical training stage with an inference accuracy rate not greater than the accuracy threshold. In this case, if the inference accuracy rates corresponding to the at least one target historical data set include at least one historical training stage with an inference accuracy rate not greater than the accuracy threshold, the forgetting assessment result is determined to be forgotten. If the inference accuracy rates corresponding to the at least one target historical data set do not include at least one historical training stage with an inference accuracy rate not greater than the accuracy threshold, the forgetting assessment result is determined to be not forgotten.

[0062] It should be noted that, when the forgetting evaluation result of the trained i-th network model indicates that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, it means that the inference accuracy of the trained i-th network model for the training data set of the historical training stage is low, and there is forgetting. Therefore, the corresponding training data set of at least one training stage after the i-th training stage includes part or all of the training data set of the forgotten training stage. This can ensure that the data set of the forgotten training stage is reviewed in the subsequent training stage, thereby improving the inference performance of the target network model.

[0063] In a second implementation, after obtaining the trained i-th network model, a forgetfulness assessment is performed on the trained i-th network model to obtain a forgetfulness assessment result. In this case, if the inference accuracy of at least one historical training stage is determined to be no greater than an accuracy threshold based on the forgetfulness assessment result, then the configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized based on the forgetfulness assessment result, the loss curve of the i-th network model, and at least one of the inference results of at least one test dataset. Otherwise, the configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized based on at least one of the loss curve of the i-th network model and the inference results of at least one test dataset.

[0064] In a second aspect, a network model training device is provided, wherein the network model training device has the function of implementing the network model training method described in the first aspect. The network model training device includes at least one module, which is used to implement the network model training method described in the first aspect.

[0065] In a third aspect, a computing device cluster is provided, which includes at least one computing device, each computing device including a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the network model training method described in the first aspect.

[0066] Optionally, each computing device may further include a communication bus, which is used to establish a connection between the processor and the memory of each computing device.

[0067] In a fourth aspect, a computer-readable storage medium is provided, which includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the steps of the network model training method described in the first aspect.

[0068] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the steps of the network model training method described in the first aspect.

[0069] The technical effects obtained in the above-mentioned second, third, fourth and fifth aspects are similar to those obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0071] FIG2 is a schematic diagram of the interaction between various units included in a cloud platform provided in an embodiment of the present application;

[0072] FIG3 is a flow chart of a network model training method provided in an embodiment of the present application;

[0073] FIG4 is a schematic diagram of a parameter configuration interface provided in an embodiment of the present application;

[0074] FIG5 is a schematic diagram of training planning information provided in an embodiment of the present application;

[0075] FIG6 is a schematic diagram of another training plan information provided in an embodiment of the present application;

[0076] FIG7 is a schematic diagram of model planning information provided in an embodiment of the present application;

[0077] FIG8 is a schematic diagram of data planning information provided in an embodiment of the present application;

[0078] FIG9 is a schematic diagram of a configuration preview interface provided in an embodiment of the present application;

[0079] FIG10 is a schematic diagram of a parameter amplification interface provided in an embodiment of the present application;

[0080] FIG11 is a schematic diagram of an interface provided in an embodiment of the present application;

[0081] FIG12 is a schematic diagram of a training evaluation interface provided in an embodiment of the present application;

[0082] FIG13 is a schematic diagram of a training record interface provided in an embodiment of the present application;

[0083] FIG14 is a flowchart of another network model training method provided in an embodiment of the present application;

[0084] FIG15 is a schematic diagram of the structure of a network model training device provided in an embodiment of the present application;

[0085] FIG16 is a schematic structural diagram of a computer device provided in an embodiment of the present application;

[0086] FIG17 is an architectural diagram of a computing device cluster provided in an embodiment of the present application;

[0087] FIG18 is a schematic diagram of a connection between computing devices provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0089] For ease of understanding, before explaining in detail the training method of the network model provided in the embodiment of the present application, the application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

[0090] First, the application scenarios involved in the embodiments of the present application are introduced.

[0091] With the continuous development and improvement of artificial intelligence technology, neural network models have been widely used in fields such as natural language processing, computer vision, and speech recognition. Because large-parameter (also known as large-scale) neural network models can better handle complex tasks and have better generalization capabilities, the number of parameters in neural network models is increasing, and the demand for training large-parameter neural network models is also gradually increasing. Taking the training of the GPT-3 neural network model with 175 billion parameters as an example, it would take at least more than a month to train the GPT-3 neural network model using a state-of-the-art graphics processor with 1024 video cards and 80GB of video memory.

[0092] Currently, neural network models are mainly trained through cloud computing platforms. However, the training functions provided by existing cloud computing platforms are relatively simple and cannot adapt to the needs brought about by the continuous expansion of parameter quantities. Users cannot fine-tune their own training plans to improve the training efficiency of neural network models. Cloud computing platforms are also unable to provide users with neural network model training experience to optimize the training process of neural network models, thereby accelerating the training of neural network models. If a neural network model with a large number of parameters is trained, it will take longer training time and a greater amount of computing power, making the model training cost high and the efficiency low.

[0093] Based on this, an embodiment of the present application provides a network model training method, in which the user can input the parameter amount and type of the target network model to be trained in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models with different parameter amounts and different types in a targeted manner, so that the parameter amount of the model, the type of model and the determined training configuration information are more adapted. In addition, the embodiment of the present application can also divide the training process of the network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training, so that the staged and fine control of the network model training can be achieved, and since the parameter amount of the network model is gradually increased with the training stage, the convergence speed of the network model can be effectively accelerated, the amount of calculation required for network model training can be saved, and the training cost of the network model can be reduced.

[0094] The execution entity of the network model training method provided in the embodiment of the present application can be a cloud platform or an electronic device with model training function. The cloud platform can be a server cluster or distributed system composed of multiple physical servers, etc., or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or it can be a cloud computing service center.

[0095] Next, the implementation environment involved in the embodiment of the present application is introduced. Based on the above description, the execution subject of the network model training method provided in the embodiment of the present application can be a cloud platform or an electronic device with a model training function. Next, the cloud platform is used as an example for introduction.

[0096] Please refer to Figure 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a planning unit 01, a training unit 02, a recording unit 03, and a prediction unit 04. Specifically, the planning unit 01 includes a training planning subunit 11, a model planning subunit 12, and a data planning subunit 13. The training unit 02 includes a training execution subunit 21, an interaction evaluation subunit 22, a forgetfulness evaluation subunit 23, and a parameter amplification subunit 24.

[0097] The training planning subunit 11 is used to determine the training planning information corresponding to multiple training stages, the model planning subunit 12 is used to determine the model planning information corresponding to multiple training stages, and the data planning subunit 13 is used to determine the data planning information corresponding to multiple training stages. The training execution subunit 21 is used to train the network models of each stage, the interactive evaluation subunit 22 is used to perform training evaluation on the network models trained in each training stage, the forgetting evaluation subunit 23 is used to perform forgetting evaluation on the network models trained in each training stage, and the parameter amplification subunit 24 is used to amplify the parameters of the network models in each training stage. The recording unit 03 is used to record the model planning information, training planning information and data planning information corresponding to the multiple training stages of the target network model, the changes in the loss rate of the target network model during the training process, the training evaluation results of the network models trained in each training stage, the forgetting evaluation results of the network models trained in each training stage, the parameter amplification strategy of the target network model, and the interactive operations between the user and the cloud platform. The prediction unit 04 is used to determine the first training configuration information based on the parameter quantity and type of the target network model, predict the training evaluation results of the network model trained in each stage, predict the forgetting evaluation results of the network model trained in each stage, predict the parameter expansion strategy of the network model, and provide suggestions for the training of the target network model.

[0098] Optionally, the prediction unit 04 can be a training prediction model. The cloud platform inputs the parameter quantity and type of the target network model into the training prediction model to obtain the first training configuration information output by the training prediction model. The training prediction model can also predict the training evaluation results of the network model trained in each stage, predict the forgetting evaluation results of the network model trained in each stage, predict the parameter amplification strategy of the network model, and give suggestions for the training of the target network model.

[0099] It should be noted that the training prediction model can be a generative model, which can be based on a deep neural network and can learn to abstract the essential laws and probability distribution of data by training large-scale data sets, thereby generating new data.

[0100] Please refer to Figure 2, which is a schematic diagram of the interaction between various units included in a cloud platform provided in an embodiment of the present application. The planning unit 01 can obtain the parameter quantity and type of the target network model and send the parameter quantity and type of the target network model to the prediction unit 04. The prediction unit 04 determines first training configuration information based on the parameter quantity and type of the target network model. After determining the first training configuration information, the prediction unit 04 sends the first training configuration information to the planning unit 01. The planning unit 01 determines the training configuration information of the target network model based on the first training configuration information, and then sends the training configuration information of the target network model to the training unit 02 and the recording unit 03. The training unit 02 receives the training configuration information of the target network model sent by the planning unit 01, and trains the target network model based on the training configuration information of the target network model. Training unit 02 can also call prediction unit 04 to predict the training evaluation results of the network model trained at each stage, predict the forgetfulness evaluation results of the network model trained at each stage, predict the parameter augmentation strategy for the network model, and provide recommendations for the training of the target network model. Prediction unit 04 then sends the predicted training evaluation results, forgetfulness evaluation results, and parameter augmentation strategy to training unit 02. Training unit 02 receives the training evaluation results, forgetfulness evaluation results, and parameter augmentation strategy predicted and sent by prediction unit 04. Training unit 02 also sends user interactions with the cloud platform and the training status of the network model at each training stage to recording unit 03. Recording unit 03 stores the training configuration information of the target network model sent by planning unit 01, as well as user interactions with the cloud platform and the specific details of the network model trained at each training stage sent by training unit 02. Recording unit 03 can also send all its stored information to prediction unit 04, which then receives all information sent by recording unit 03.

[0101] In some embodiments, the recording unit 03 can also provide a training record interface, through which the user can view the configuration information and training records of any training stage, and the training records include the training status of the network model in each training stage.

[0102] For example, the training situation may include a loss curve corresponding to any training stage, an inference accuracy rate corresponding to at least one test data set, a training log, etc., which is not limited in this embodiment of the present application.

[0103] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0104] Figure 3 is a flow chart of a network model training method provided in an embodiment of the present application. Referring to Figure 3 , the method includes the following steps. For ease of description, the network model training method provided in an embodiment of the present application will be described below using the cloud platform as the execution entity.

[0105] Step 301: providing a parameter configuration interface, which is used to instruct the user to input the parameter amount and type of the target network model to be trained.

[0106] In practical applications, the type of network model may be a BERT (bidirectional encoder representations from transformers) model, a GPT (Generative Pre-trained Transformer) model, a T5 (Text-To-Text Transfer Transformer) model, a ResNet (residual network) model, etc., which is not limited in the embodiments of the present application.

[0107] Step 302: Obtain the parameter quantity and type of the target network model from the parameter configuration interface.

[0108] In some embodiments, the cloud platform completes the operation in response to the user-triggered input and obtains the parameter quantity and type of the target network model from the parameter configuration interface.

[0109] After inputting the parameter amount and type of the target network model, the user can trigger the input completion operation. The cloud platform responds to the input completion operation triggered by the user and obtains the parameter amount and type of the target network model from the parameter configuration interface.

[0110] For example, please refer to Figure 4. The cloud platform can provide a parameter configuration interface as shown in Figure 4. The user can enter the parameter quantity of the target network model (also known as the scale of the target network model) as 7 billion (billion, B) in the parameter configuration interface shown in Figure 4. The type of the target network model is a GPT model. After the user enters the parameter quantity and type of the target network model, the input is triggered to complete the operation. The cloud platform completes the operation in response to the input triggered by the user, and obtains the parameter quantity and type of the target network model from the parameter configuration interface.

[0111] Step 303: Based on the parameter quantity and type of the target network model, determine the training configuration information of the target network model. The training process of the target network model is divided into multiple training stages. Different training stages correspond to different model parameter quantities, and the model parameter quantities of the multiple training stages increase in sequence according to the training order. The training configuration information of the target network model includes configuration information of multiple training stages.

[0112] In some embodiments, the configuration information of the multiple training stages includes model planning information, training planning information and data planning information corresponding to the multiple training stages respectively. The model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration status of the training data of the corresponding training stage.

[0113] Next, we will introduce model planning information, training planning information, and data planning information respectively.

[0114] 1. Training plan information

[0115] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of the model parameters in the corresponding training phase.

[0116] It should be noted that, for any training stage except the first training stage in the multiple training stages, the maximum value of the learning rate interval of the training stage is the minimum value of the learning rate interval of the previous training stage of the training stage, and the minimum value of the learning rate interval of the last training stage in the multiple training stages is 0.

[0117] Optionally, the training planning information further includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0118] It should be noted that the expected loss value is determined based on the number of model iterations and the learning rate range of the corresponding training stage. For example, the cloud platform can input the number of model iterations and the learning rate range of each training stage into the training prediction model to obtain the expected loss value of each training stage output by the training prediction model. Of course, in practical applications, the expected loss value can also be determined by other means, which is not limited in this application.

[0119] For example, please refer to Figure 5, which is a schematic diagram of training planning information. In Figure 5, the training process of the target network model is divided into training phase 1 to training phase n. The training planning information of training phase 1 to training phase n includes the number of model iterations, learning rate range and expected loss value. Among them, the number of model iterations in training phase 1 is 25,000 times, and the learning rate range is 8e-4 (i.e. 8×10 -4 ) to 7e-4 (i.e. 7×10 -4 ), with an expected loss of 3.7. The model iteration count for phase 2 is 60,000, with a learning rate range of 7e-4 to 5e-4, and an expected loss of 2.8. The model iteration count for phase 3 is 150,000, with a learning rate range of 5e-4 to 4e-4, and an expected loss of 2.2. The model iteration count for phase n is 900,000, with a learning rate range of 8e-5 (the minimum of the learning rate range for phase n-1) to 0, and an expected loss of 1.8.

[0120] In some embodiments, the training plan information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter accuracy and the batch size, wherein the learning rate decay type indicates how the step size of the model parameters changes, the parameter accuracy indicates the accuracy of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0121] For example, the learning rate decay types include linear decay, exponential decay, etc., which are not limited in the embodiments of the present application.

[0122] For example, the parameter precision includes fp16, fp32, etc., which is not limited in the embodiments of the present application. Among them, fp16 is also called half-precision floating-point, which occupies 16 bits of storage space, including 1 sign bit, 5 exponent bits and 10 mantissa bits. fp32 is also called single-precision floating-point, which occupies 32 bits of storage space, including 1 sign bit, 8 exponent bits and 23 mantissa bits.

[0123] Optionally, the training planning information may further include a dropout rate, which refers to the proportion of neurons that do not participate in calculations (ie, their outputs are set to 0) in each iteration when training a neural network.

[0124] The dropout rate ranges from 0 to 1. For example, the dropout rate can be 0.5, in which case the outputs of approximately half of the neurons are set to 0 at each iteration.

[0125] It should be noted that the learning rate decay type, parameter precision, batch size, and random activation rate are the same in each training stage, that is, the learning rate decay type, parameter precision, batch size, and random activation rate remain unchanged throughout the training process of the target network model.

[0126] For example, refer to Figure 6. The number of warm-up iterations in the training plan information is 1000, the random dropout rate is 0.1, the learning rate decay type is linear decay, and the parameter precision is fp16.

[0127] 2. Model planning information

[0128] Based on the above description, there are various types of network models. Depending on the type of network model, the model planning information includes different parameters. For example, if the target network model type indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model at the corresponding training stage, the dimensions of the hidden layers in each self-attention network, and the number of attention heads.

[0129] It should be noted that the dimension of the hidden layer refers to the number of neurons in the hidden layer.

[0130] The aforementioned BERT, GPT, and T5 models are all pre-trained language models based on the Transformer architecture. These BERT, GPT, and T5 models all include self-attention networks. In other words, pre-trained language models based on the Transformer architecture include self-attention networks. In this case, if the target network model is a BERT, GPT, or T5 model, the model planning information includes the number of self-attention networks in the network model at the corresponding training stage, the dimensions of the hidden layers in each self-attention network, and the number of attention heads.

[0131] Optionally, for any training stage other than the first training stage among the multiple training stages, the value of any parameter included in the model planning information of the training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and, among the parameters included in the model planning information of the training stage, there is at least one parameter whose value is greater than the value of the corresponding parameter in the previous training stage. In other words, for two adjacent training stages among the multiple training stages, the value of the parameter included in the model planning information of the latter training stage is greater than or equal to the value of the corresponding parameter in the previous training stage, and the values ​​of the parameters included in the model planning information of the two training stages are not completely equal.

[0132] Taking a network model with a Transformer architecture as an example, for any training stage other than the first training stage in multiple training stages, the number of self-attention networks in the network model of that training stage is greater than or equal to the number of self-attention networks in the network model trained in the previous training stage, the dimension of each hidden layer in each self-attention network in the network model of that training stage is greater than or equal to the dimension of each hidden layer in the network model trained in the previous training stage, the number of attention heads in the network model of that training stage is greater than or equal to the number of attention heads in the network model trained in the previous training stage, and the values ​​of the above three parameters are not all equal to the values ​​of the corresponding parameters in the previous training stage. In this way, the scale of the network model in the multiple training stages can be guaranteed to increase sequentially, so that the model parameters in the multiple training stages can be increased sequentially according to the training sequence, thereby achieving stage-by-stage fine control of network model training. Moreover, since the parameters of the network model gradually increase with the training stage, this can effectively accelerate the convergence of the network model, save the computational effort required for network model training, and reduce the training cost of the network model.

[0133] Optionally, the model planning information also includes the model parameter quantities of the corresponding training stage.

[0134] In some embodiments, the model parameter quantity is determined based on the number of self-attention networks in the above-mentioned network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. The model parameter quantity is used to intuitively determine the scale of the network model in the training stage. The difference between the model parameter quantity of the last training stage and the parameter quantity of the target network model input by the above-mentioned user is less than or equal to the parameter quantity difference threshold.

[0135] The parameter difference threshold is set in advance, for example, the parameter difference can be set to 20. In addition, it can be adjusted according to different requirements in different situations.

[0136] Please refer to Figure 7, which is a schematic diagram of model planning information. In Figure 7, the training process of the target network model is divided into training stages 1 to n. The model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer and the number of attention heads in each self-attention network, and the model parameter amount of the corresponding training stage. Among them, the number of self-attention networks in the network model of training stage 1 is 3, the dimension of the hidden layer in each self-attention network is 384, the number of attention heads is 6, and the model parameter amount is 17 million (million, M), that is, the model parameter amount is 17×10 6 In the network model of training phase 2, the number of self-attention networks is 6, the dimension of the hidden layer in each self-attention network is 512, the number of attention heads is 8, and the number of model parameters is 35M, that is, the number of model parameters is 35×10 6 The number of self-attention networks in the network model of training stage 3 is 12, the dimension of the hidden layer in each self-attention network is 768, the number of attention heads is 12, and the number of model parameters is 110M, that is, the number of model parameters is 110×10 6 The number of self-attention networks in the network model of training stage n is 32, the dimension of the hidden layer in each self-attention network is 4096, the number of attention heads is 32, and the number of model parameters is 7B, that is, the number of model parameters is 7×10 9 .

[0137] 3. Data planning information

[0138] In some embodiments, the data planning information includes the proportion of the training data set of the corresponding training stage to the total training data set, and the total training data set is the sum of the training data sets of multiple training stages.

[0139] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage in multiple training stages.

[0140] For example, the pruning configuration information includes pruning status information, where the pruning status information indicates whether to filter the training data set of the corresponding training phase.

[0141] In some embodiments, the pruning status information can be on or off. If the pruning status information is on, it indicates that the training data set of the corresponding training phase is filtered. If the pruning status information is off, it indicates that the training data set of the corresponding training phase is not filtered. Of course, in actual applications, other methods can also be used to indicate whether the training data set of the corresponding training phase is filtered, and the embodiments of the present application are not limited to this.

[0142] In practical applications, pruning can be performed using a variety of methods, such as a pruning method based on a forgetting fraction, a pruning method based on an error L2 norm (eL2n), etc. When multiple pruning methods are used and the pruning status information indicates that the training data set of the corresponding training phase is to be screened, the pruning configuration information also includes information indicating a target pruning method, where the target pruning method is one of the multiple pruning methods.

[0143] It should be noted that, when the training data set of the corresponding training stage is screened, the pruning method of each training stage is the same, that is, the target pruning method is used to screen the training data set in each training stage.

[0144] For example, the configuration information of the distillation preheating includes distillation preheating status information, which indicates whether to perform distillation preheating on the network model to determine the initial values ​​of the model parameters in the first training stage.

[0145] In some embodiments, the distillation preheating status information can be on or off. If the distillation preheating status information is on, it indicates that the network model is being distilled and preheated. If the distillation preheating status information is off, it indicates that the network model is not being distilled and preheated. Of course, in practical applications, other methods can also be used to indicate whether the network model is being distilled and preheated, and this embodiment of the application is not limited to this.

[0146] During the training process of the network model, the appropriateness of the initial values ​​of the model parameters has a great impact on the results. Inappropriate initial values ​​may cause the network model to converge slowly or fail to converge during training. Therefore, in order to accelerate the convergence speed of the network model during training and improve the performance of the network model, in some embodiments, the network model can be distilled and preheated before entering the first training stage to determine the initial values ​​of the model parameters for the first training stage.

[0147] In some embodiments, a general corpus package is pre-stored in the cloud platform. When it is determined that the network model is to be distilled and preheated, the network model of the first training phase is pre-trained using the general corpus package before the first training phase to determine the initial values ​​of the model parameters of the first training phase, thereby achieving distillation and preheating of the target network model. Of course, the above-mentioned distillation and preheating of the network model using the general corpus package is only one implementation method. In actual applications, the network model can also be distilled and preheated in other ways, such as distillation and preheating through the Xavier initialization (also known as Glorot initialization) method, etc. This embodiment of the present application does not limit this.

[0148] It should be noted that when the distillation preheating status information indicates that the network model should be distilled and preheated, and there are multiple distillation preheating methods, the distillation preheating configuration information also includes indication information of the target distillation preheating method, and the target distillation preheating method is one of the multiple distillation preheating methods.

[0149] In practical applications, the data planning information may also include other contents such as the storage path of the training data set, which is not limited in the embodiments of the present application.

[0150] For example, please refer to Figure 8, which is a schematic diagram of data planning information. In Figure 8, the training process of the target network model is divided into training stages 1 to 4. The data planning information includes the proportion of the training dataset in the corresponding training stage to the total training dataset, pruning configuration information, distillation preheating configuration information, and the storage path of the training dataset. The pruning configuration information includes pruning status information and target pruning method. The distillation preheating configuration information includes distillation preheating status information. The pruning methods include pruning method 1 and pruning method 2. Pruning method 1 is a pruning method based on the forgetting score, and pruning method 2 is a pruning method based on the error L2 norm. The training dataset of training stage 1 accounts for 25% of the total training dataset, the training dataset of training stage 2 accounts for 25% of the total training dataset, the training dataset of training stage 3 accounts for 25% of the total training dataset, and the training dataset of training stage 4 accounts for 25% of the total training dataset. The pruning status information is enabled, and the target pruning method is pruning method 1. The distillation preheating status information is enabled. The training dataset is stored in the object storage service (OBS). The storage path of the training dataset is mybucket / myfolder.

[0151] The above describes the details of the training configuration information of the target network model. Next, the process of determining the training configuration information of the target network model will be described.

[0152] The parameter quantity and type of the target network model are input into the training prediction model to obtain the first training configuration information output by the training prediction model, and the training configuration information of the target network model is determined based on the first training configuration information.

[0153] That is to say, the embodiments of the present application can specifically determine corresponding training configuration information for network models with different parameter amounts and different types, so that the parameter amounts of the model, the type of model and the determined training configuration information are more adapted.

[0154] In some embodiments, the first training configuration information can be directly determined as the training configuration information of the target network model. In other embodiments, a configuration preview interface can also be provided, the configuration preview interface including the first training configuration information. In response to a user's modification operation on the first training configuration information, the first training configuration information is modified to obtain second training configuration information. Based on the second training configuration information, the training configuration information of the target network model is determined.

[0155] That is to say, after determining the first training configuration information based on the parameter quantity and type of the target network model, the user can also modify the first training configuration information to obtain the training configuration information of the target network model, thereby realizing the refined customization of the training plan of the target network model and effectively improving the training efficiency of the neural network model.

[0156] For ease of description, the number of training stages, number of model iterations, learning rate range, expected loss value, number of warm-up iterations, learning rate decay type, parameter accuracy, batch size, random inactivation rate, number of self-attention networks in the network model, dimension of hidden layers in each self-attention network, number of attention heads, number of model parameters, proportion of the training data set in the corresponding training stage to the total training data set, pruning configuration information, and distillation preheating configuration information are collectively referred to as planning parameters.

[0157] Based on the description above, the expected loss value is determined based on the number of model iterations and the learning rate range in the corresponding training phase. The number of model parameters is determined based on the number of self-attention networks in the above network model, the dimension of the hidden layer in each self-attention network, and the number of attention heads. In other words, the values ​​of planning parameters such as the expected loss value and the number of model parameters need to be determined based on the values ​​of other planning parameters and cannot be directly configured. Therefore, planning parameters such as the expected loss value and the number of model parameters can also be called dependent variable parameters, and the planning parameters used to determine the dependent variable parameters are called independent variable parameters. Among these planning parameters, those that can be directly configured are called configurable parameters. Generally, users can modify these configurable parameters.

[0158] In some embodiments, the first training configuration information includes values ​​for multiple planning parameters, and the second training configuration information also includes values ​​for multiple planning parameters. The value of at least one configurable parameter among these planning parameters has been modified by the user. Specifically, the user modifies the value of the at least one configurable parameter in the first training configuration information, and the cloud platform modifies the value of the at least one configurable parameter in response to the user's modification operation.

[0159] In some embodiments, the configurable parameters include the number of model iterations in the corresponding training stage, the learning rate range, the number of warm-up iterations, the learning rate decay type, the parameter accuracy, the batch size, the random inactivation rate, the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, the number of attention heads, the proportion of the training data set in the corresponding training stage to the total training data set, the configuration information of pruning, and at least one of the configuration information of distillation preheating.

[0160] In practical applications, the configurable parameter may also be the number of training stages of the target network model.

[0161] Optionally, the configuration preview interface also includes a stage add button and a stage delete button. The user can add a training stage by triggering the stage add button and delete a training stage by triggering the stage delete button, thereby modifying the number of training stages.

[0162] For example, the stage add button includes an add button corresponding to each training stage, and the stage delete button includes a delete button corresponding to each training stage. In this case, the training stage added by triggering the add button corresponding to the corresponding training stage is the next training stage adjacent to the training stage corresponding to the add button, and the training stage deleted by triggering the add button corresponding to the corresponding training stage is the training stage corresponding to the delete button.

[0163] For example, please refer to Figure 9, which is a schematic diagram of a configuration preview interface provided in an embodiment of the present application. The configuration preview interface includes the number of model iterations, learning rate range, expected loss value, and add buttons and delete buttons corresponding to training stages 1 to training stages 4, respectively. The user adds a new training stage by triggering the add button corresponding to training stage 3. In this case, the training stages of the target network model include training stages 1 to training stage 5.

[0164] It should be noted that if a user modifies the first training configuration information, it indicates that the user has experience in network model training. Therefore, after adding or removing training stages, the user needs to reconfigure the values ​​of other configurable parameters corresponding to the multiple training stages based on their experience. In this case, the at least one configurable parameter modified by the user includes not only the number of training stages, but also other configurable parameters reconfigured by the user in addition to the number of training stages.

[0165] In actual applications, after modifying the number of training stages, in addition to the user reconfiguring the configurable parameters corresponding to the multiple training stages after the increase or decrease of the training stages, the number of training stages after the increase or decrease, the parameter amount and type of the target network model can also be input into the training prediction model to obtain the values ​​of the configurable parameters corresponding to the multiple training stages output by the training prediction model.

[0166] In some embodiments, the implementation process of determining the training configuration information of the target network model based on the second training configuration information includes: if the at least one configurable parameter modified by the user includes an independent variable parameter, then based on the value of the modified independent variable parameter, the value of the corresponding dependent variable parameter is re-determined, and the value of the at least one configurable parameter, the value of the re-determined dependent variable parameter, and the values ​​of other configurable parameters included in the first training configuration information other than the at least one configurable parameter are determined as the training configuration information of the target network model. If the at least one configurable parameter modified by the user does not include an independent variable parameter, then the value of the at least one configurable parameter and the values ​​of the planning parameters included in the first training configuration information other than the at least one configurable parameter are determined as the training configuration information of the target network model.

[0167] Step 304: Train the target network model according to the configuration information of multiple training stages.

[0168] Next, the implementation process of training the target network model according to the configuration information of multiple training stages will be introduced through steps (1) to (4).

[0169] Step (1): Let i=1, and determine the i-th network model based on the configuration information of the i-th training stage in multiple training stages.

[0170] In some embodiments, if the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage; if the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0171] If the i-th training stage is the first training stage among multiple training stages, the implementation process of generating the i-th network model based on the configuration information of the i-th training stage includes: determining the model structure of the first network model based on the model planning information of the first training stage, and determining the initial values ​​of the parameters of the first network model based on the distillation preheating configuration information, thereby obtaining the first network model.

[0172] In some embodiments, the distillation preheating configuration information indicates that the network model is to be distilled and preheated. In this case, before entering the first training phase, the first network model can be distilled and preheated to determine the initial values ​​of the parameters of the first network model.

[0173] If the distillation preheating configuration information indicates that distillation preheating is not performed on the first network model, the initial parameter values ​​of the first network model can be determined through random initialization. Of course, in actual applications, other methods can also be used to determine the initial parameter values ​​of the first network model, and this embodiment of the application is not limited to this.

[0174] In some embodiments, the configuration information of the i-th training stage includes the model planning information of the i-th training stage. In this case, based on the configuration information of the i-th training stage, the trained i-1-th network model is amplified to obtain the i-th network model. The implementation process includes: providing a parameter amplification interface, the parameter amplification interface is used to instruct the user to select a parameter amplification strategy, the parameter amplification strategy is used to determine the initial value of the newly added parameters of the i-th network model relative to the i-1-th network model, obtaining the parameter amplification strategy from the parameter amplification interface, and based on the parameter amplification strategy and the model planning information of the i-th training stage, performing parameter amplification on the i-1-th network model to obtain the i-th network model.

[0175] Optionally, the parameter amplification interface also includes a recommended parameter amplification strategy. In this case, after the cloud platform displays the recommended parameter amplification strategy on the parameter amplification interface, the user can select the recommended parameter amplification strategy as the selected parameter amplification strategy, or select a parameter amplification strategy other than the recommended parameter amplification strategy.

[0176] For example, referring to Figure 10, the parameter amplification interface includes recommended parameter amplification strategies, which are width amplification strategy 1 and depth amplification strategy 1. The user can directly select the recommended parameter amplification strategy as the selected parameter amplification strategy.

[0177] In some embodiments, the cloud platform can store the recommended parameter augmentation strategy. In this case, the cloud platform can directly obtain the recommended parameter augmentation strategy.

[0178] In other embodiments, the cloud platform can input the configuration information of multiple training stages and the indication information of user interaction operations into the training prediction model to obtain the recommended parameter augmentation strategy output by the training prediction model. The indication information of the user interaction operation can represent the operation performed by the user on the cloud platform. The operation can be the user's modification operation of the first training configuration information, or the user's modification operation of the training evaluation results, etc. The user's modification operation of the training evaluation results will be introduced later and will not be explained in detail here.

[0179] In practical applications, in addition to selecting a parameter augmentation strategy from the parameter augmentation interface, the parameter augmentation strategy can also be pre-set and stored in the cloud platform. In this way, the cloud platform can directly obtain the parameter augmentation strategy without the user having to select the parameter augmentation strategy.

[0180] In some embodiments, the parameter augmentation strategy includes a width augmentation strategy and a depth augmentation strategy. The width augmentation strategy indicates a method for determining the initial values ​​of the parameters of the newly added dimensions of the hidden layer in the self-attention network, and the depth augmentation strategy refers to a method for determining the initial values ​​of the parameters in the newly added self-attention network of the i-th network model relative to the i-1-th network model. In this case, based on the parameter augmentation strategy and the model planning information of the i-th training stage, the implementation process of parameter augmentation for the i-1-th network model includes: based on the width augmentation strategy and the model planning information of the i-th training stage, the dimensions of the hidden layers in each self-attention network in the i-1-th network model are augmented to obtain an initial augmented model; based on the depth augmentation strategy and the model planning information of the i-th training stage, the self-attention network of the initial augmented model is augmented to obtain the i-th network model.

[0181] Based on the width expansion strategy and the model planning information of the i-th training stage, the dimension of the hidden layer in each self-attention network in the i-1-th network model is expanded to obtain the implementation process of the initial expansion model, including: based on the value of the parameter of the neuron in the hidden layer of each self-attention network in the i-1-th network model, according to the width expansion strategy, determining the initial value of the parameter of the newly added neuron in the hidden layer of each self-attention network in the initial expansion model relative to the i-1-th network model, the dimension of each hidden layer in the self-attention network in the initial expansion model is the same as the dimension of each hidden layer in the network model in the i-th training stage, and the number of self-attention networks in the initial expansion model is the same as the number of self-attention networks in the i-1-th network model.

[0182] For the parameters of the target dimension in the newly added dimension, the width expansion strategy can be to perform a weighted average of the parameter values ​​of the neurons in the hidden layer of each self-attention network in the i-1th network model to obtain the initial value of the parameter of the target neuron, and the target neuron is any one of the newly added neurons. Of course, in practical applications, the width expansion strategy can also be other algorithms, such as determining the average value of the parameter values ​​of the neurons in the hidden layer of each self-attention network in the i-1th network model as the initial value of the parameter of the target neuron, etc., and the embodiments of the present application do not limit this.

[0183] Based on the deep expansion strategy and the model planning information of the i-th training stage, the self-attention network of the initial expansion model is expanded to obtain the i-th network model. The implementation process includes: based on the values ​​of the parameters in the self-attention network in the initial expansion model, according to the deep expansion strategy, determining the initial values ​​of the parameters in the newly added self-attention network in the i-th network model relative to the initial expansion model, the dimension of the hidden layer in the self-attention network in the i-th network model is the same as the dimension of the hidden layer in each self-attention network in the network model of the i-th training stage, and the number of self-attention networks in the i-th network model is the same as the number of self-attention networks in the i-th network model.

[0184] For any parameter in any newly added self-attention network, the deep expansion strategy can be to use the value of the parameter in the same position as the parameter in any self-attention network in the initial expansion model as the initial value of the parameter. Of course, in practical applications, the deep expansion strategy can also be other algorithms, for example, using the average value of all parameters in the same position as the parameter in the self-attention network in the initial expansion model as the initial value of the parameter. This embodiment of the application is not limited to this.

[0185] Step (2): Train the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model.

[0186] In some embodiments, before training the i-th network model according to the configuration information of the i-th training stage to obtain the trained i-th network model, the user can also trigger the start training operation, and the cloud platform can also respond to the start training operation triggered by the user to train the i-th network model according to the configuration information of the i-th training stage.

[0187] For example, before the i-th network model is trained according to the configuration information of the i-th training stage, a first training execution interface may be displayed, and the user may trigger a training start operation on the first training execution interface.

[0188] In some embodiments, during the process of training the i-th network model according to the configuration information of the i-th training stage, the user can also trigger a pause training operation, and the cloud platform can also pause the training of the i-th training stage in response to the pause training operation triggered by the user.

[0189] Optionally, during the process of training the i-th network model according to the configuration information of the i-th training stage, the user can also trigger a stop training operation, and the cloud platform can also stop the training of the i-th training stage in response to the stop training operation triggered by the user.

[0190] For example, during the process of training the i-th network model according to the configuration information of the i-th training stage, a second training execution interface may be displayed, and the user may trigger a pause training operation or a stop training operation on the second training execution interface.

[0191] In some embodiments, during the process of training the i-th network model according to the configuration information of the i-th training stage, a third training execution interface may also be displayed, where the third training execution interface includes a training log of the i-th training stage.

[0192] Optionally, the training log includes the number of model iterations, loss function, changes in model parameters, etc. during the training process of the network model.

[0193] Optionally, the third training execution interface also includes a computational savings situation, which indicates the computational savings of the embodiment of the present application relative to related model training technologies when achieving the same training effect.

[0194] Optionally, the third training execution interface also includes training configuration information of the i-th training stage.

[0195] It should be noted that the above-mentioned first training execution interface, second training execution interface and third training execution interface can be the same interface, or any two of the first training execution interface, second training execution interface and third training execution interface are the same interface, or the first training execution interface, second training execution interface and third training execution interface are different interfaces respectively. The embodiments of the present application are not limited to this.

[0196] For example, referring to FIG11 , the first training execution interface, the second training execution interface, and the third training execution interface can be the same interface. Before training the nth network model according to the configuration information of the nth training stage, the user can trigger the start of training in the interface. During the training of the nth network model according to the configuration information of the nth training stage, the user can trigger the pause or stop of training. The interface can also display the training configuration, training log, and computational savings of the nth training stage.

[0197] Step (3): Determine the training evaluation result of the trained i-th network model and provide a training evaluation interface, which includes the training evaluation result of the trained i-th network model. The training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training.

[0198] In some embodiments, a loss curve of the i-th network model is determined, where the loss curve indicates how the loss value changes with the number of model iterations, and the loss value indicates the gap between the predicted value and the true value of the network model. At least one test data set is input into the trained i-th network model, and an inference result of the at least one test data set output by the trained i-th network model is obtained. Based on the loss curve and the inference result of the at least one test data set, a training evaluation result of the trained i-th network model is determined.

[0199] Optionally, during the training of the i-th network model, the cloud platform can record the change of the loss value of the i-th network model with the number of model iterations. In this way, the cloud platform can draw the loss curve of the i-th network model based on the change of the loss value of the i-th network model with the number of model iterations.

[0200] In some embodiments, the implementation process of determining the training evaluation result of the trained i-th network model based on the loss curve and the inference result of at least one test data set includes: determining the inference accuracy corresponding to the at least one test data set based on the inference result of the at least one test data set, and determining the training evaluation result of the trained i-th network model based on the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage.

[0201] For any one of the at least one test datasets, the proportion of correctly inferred data in the inference results of the test dataset is counted to obtain the inference accuracy corresponding to the test dataset. Each of the at least one test datasets is processed in the same manner to obtain the inference accuracy corresponding to each test dataset.

[0202] Optionally, the training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result, wherein the loss value prediction evaluation result indicates whether the cloud platform's prediction of the loss value of the i-th network model meets the expected prediction, and the test set prediction evaluation result indicates whether the cloud platform's inference result of the at least one test data set meets the expected prediction. In this case, the inference accuracy corresponding to the at least one test data set, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model to obtain the loss value prediction evaluation result and the test set prediction evaluation result output by the training prediction model.

[0203] Optionally, the training evaluation result also includes training completion prediction information, which indicates the cloud platform's prediction of whether the trained i-th network model has completed training. In this case, the inference accuracy corresponding to the at least one test dataset, the loss curve of the i-th network model, and the expected loss value of the i-th training stage are input into the training prediction model, and the training completion prediction information of the i-th training stage is output by the training prediction model.

[0204] Since the training evaluation results also include the cloud platform's prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model.

[0205] In some embodiments, the training evaluation interface further includes at least one of a loss curve for the i-th network model and the inference accuracy corresponding to the at least one test dataset. Thus, the user can determine whether the trained i-th network model has completed training based on the loss curve for the i-th network model and / or the inference accuracy corresponding to the at least one test dataset, as well as the training evaluation results.

[0206] Since the training evaluation interface also includes at least one of the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, that is, the training evaluation results of the network model obtained in each training stage are displayed to the user in real time through the training evaluation interface. This makes it possible to control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training.

[0207] In actual applications, after providing the training evaluation interface, the training evaluation result can be modified in response to the user's modification operation on the training evaluation result to obtain a modified training evaluation result.

[0208] Optionally, the training evaluation interface includes the loss curve of the i-th network model, the inference accuracy corresponding to the at least one test data set, and the training evaluation result. In this case, the user can determine whether the training of the i-th network model meets expectations based on the loss curve of the i-th network model and the inference accuracy corresponding to the at least one test data set, so as to obtain the user training evaluation result. If the user training evaluation result is inconsistent with the training evaluation result, the user can modify the training evaluation result. The cloud platform responds to the user's modification operation on the training evaluation result and modifies the training evaluation result to obtain the modified training evaluation result. If the user training evaluation result is consistent with the training evaluation result, the user does not modify the training evaluation result.

[0209] Optionally, the user evaluation result includes a loss value user evaluation result and a test set user evaluation result. The loss value user evaluation result indicates whether the loss value of the i-th network model meets the user's expectations. The test set user evaluation result indicates whether the inference accuracy corresponding to the at least one test data set meets the user's expectations. The training evaluation result includes a loss value prediction evaluation result and a test set prediction evaluation result. In this case, the user can determine the loss value user evaluation result based on the loss curve of the i-th network model, and determine the test set user evaluation result based on the inference accuracy corresponding to the at least one test data set. If the loss value user evaluation result is inconsistent with the loss value prediction evaluation result, the user can modify the loss value prediction evaluation result. The cloud platform modifies the loss value prediction evaluation result in response to the user's modification operation on the loss value prediction evaluation result to obtain a modified loss value prediction evaluation result. If the loss value user evaluation result is consistent with the loss value prediction evaluation result, the user does not modify the loss value prediction evaluation result. If the user evaluation result of the test set is inconsistent with the test set prediction evaluation result, the user can modify the test set prediction evaluation result. In response to the user's modification operation on the test set prediction evaluation result, the cloud platform modifies the test set prediction evaluation result to obtain the modified test set prediction evaluation result. If the user evaluation result of the test set is consistent with the test set prediction evaluation result, the user does not modify the test set prediction evaluation result.

[0210] For example, if the loss value prediction evaluation result is that the loss value meets expectations, the test set prediction evaluation result is that the test set inference accuracy meets expectations, and the loss value user evaluation result is that the loss value meets expectations, and the test set user evaluation result is that the test set inference accuracy does not meet expectations. In this case, since the test set user evaluation result is inconsistent with the test set prediction evaluation result, the user modifies the test set prediction evaluation result to that the test set inference accuracy does not meet expectations. In response to the user's modification operation on the test set prediction evaluation result, the cloud platform modifies the test set prediction evaluation result to obtain a modified test set prediction evaluation result.

[0211] Step (4): If a user-triggered completion training operation is detected and the i-th training stage is not the last training stage among multiple training stages, then let i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among multiple training stages. If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among the multiple training stages, then the trained i-th network model is used as the trained target network model.

[0212] In some embodiments, the user can trigger the completion of training operation on the training evaluation interface when confirming that the trained i-th network model has completed training. In this case, the cloud platform detects the completion of training operation triggered by the user, and when the i-th training stage is not the last training stage among the multiple training stages, let i=i+1, and return to the above-mentioned step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages. When the i-th training stage is the last training stage among the multiple training stages, the trained i-th network model is used as the trained target network model.

[0213] In other embodiments, the user can trigger a continue training operation or a re-planning operation on the training evaluation interface when confirming that the trained i-th network model has not completed training. The continue training operation indicates to continue training the i-th network model, and the re-planning operation indicates to return to the above-mentioned step of determining the training configuration information of the target network model and re-determine the training configuration information of the target network model. If the user triggers the continue training operation, the cloud platform detects the user-triggered continue training operation, and then continues to train the i-th network model based on the configuration information of the i-th training stage. If the user triggers the re-planning operation, the cloud platform returns to the above-mentioned step of determining the training configuration information of the target network model and re-determines the training configuration information of the target network model.

[0214] For example, please refer to Figure 12. The training evaluation interface includes the loss curve of the first network model (not shown in Figure 12), the reasoning accuracy corresponding to the test data sets 1-n, and the training evaluation results. The loss value prediction evaluation result is that the loss value is in line with expectations, the test set prediction evaluation result is that the test set reasoning accuracy is in line with expectations, the loss value user evaluation result is that the loss value is in line with expectations, and the test set user evaluation result is that the test set reasoning accuracy is not in line with expectations. In this case, since the test set user evaluation result is inconsistent with the test set prediction evaluation result, the user modifies the test set prediction evaluation result to that the test set reasoning accuracy is not in line with expectations. The cloud platform responds to the user's modification operation on the test set prediction evaluation result and modifies the test set prediction evaluation result to obtain a modified test set prediction evaluation result.

[0215] Of course, the above-mentioned training of the target network model by interacting with the user after obtaining the trained i-th network model is only one implementation method. In actual applications, it is also possible to train the target network model without interacting with the user. That is, let i = 1, determine the i-th network model based on the configuration information of the i-th training stage among multiple training stages, train the i-th network model according to the configuration information of the i-th training stage, and obtain the trained i-th network model. If the i-th training stage is not the last training stage among the multiple training stages, then let i = i + 1, return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages, and if the i-th training stage is the last training stage among the multiple training stages, then use the trained i-th network model as the trained target network model.

[0216] For a detailed implementation process of determining the i-th network model based on the configuration information of the i-th training stage in multiple training stages, please refer to the relevant content above and will not be repeated here.

[0217] In actual applications, if a user-triggered training completion operation is detected and the i-th training phase is not the last training phase among multiple training phases, the configuration information of at least one training phase after the i-th training phase among the multiple training phases may be optimized. There are multiple implementations for optimizing the configuration information of at least one training phase after the i-th training phase among the multiple training phases, two of which are described below.

[0218] In a first implementation, configuration information of at least one training stage after the i-th training stage in multiple training stages is optimized based on the loss curve of the i-th network model and at least one of the inference results of the at least one test data set.

[0219] In some embodiments, at least one of the loss curve and the inference results of the at least one test data set is input into a training prediction model to obtain optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby achieving optimization of the configuration information of at least one training stage after the i-th training stage.

[0220] In other embodiments, after providing the training evaluation interface, the training evaluation result can be modified in response to a user's modification operation on the training evaluation result to obtain a modified training evaluation result. In this case, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized based on at least one of the loss curve of the i-th network model and the inference result of at least one test dataset, as well as the pre-modification training evaluation result and the modified training evaluation result.

[0221] In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can timely adjust the configuration information of the subsequent training stage based on the interaction between the user and the cloud platform, the loss curve and the inference results of the test data set, so that the final trained network model is more in line with the user's expectations.

[0222] In response to the user's modification operation on the training evaluation result, the training evaluation result is modified. For the detailed implementation process of obtaining the modified training evaluation result, please refer to the relevant content above and will not be repeated here.

[0223] In some embodiments, the cloud platform inputs the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification into the training prediction model, and obtains the optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby achieving optimization of the configuration information of at least one training stage after the i-th training stage.

[0224] In practical applications, after obtaining the trained i-th network model, the cloud platform may also perform a forgetfulness assessment on the trained i-th network model to obtain a forgetfulness assessment result. The forgetfulness assessment result indicates the inference performance of the trained i-th network model on a training dataset from a historical training phase, where the historical training phase is a training phase prior to the i-th training phase in the multiple training phases. In this case, if the inference accuracy of at least one historical training phase is determined to be no greater than an accuracy threshold based on the forgetfulness assessment result, the configuration information of at least one training phase subsequent to the i-th training phase in the multiple training phases is optimized based on the loss curve of the i-th network model and at least one of the inference results of at least one test dataset, as well as the pre-modification training assessment result, the post-modification training assessment result, and the forgetfulness assessment result. Otherwise, the configuration information of at least one training phase subsequent to the i-th training phase in the multiple training phases is optimized based on the loss curve and at least one of the inference results of at least one test dataset, as well as the pre-modification training assessment result and the post-modification training assessment result.

[0225] The forgetting evaluation result of the trained i-th network model is obtained by performing a forgetting evaluation on the model, and the implementation process includes: determining the inference accuracy of at least one target historical data set, the at least one target historical data set corresponding one-to-one to the at least one historical training stage, and the target historical data set being part or all of the training data set of the corresponding historical training stage; and determining the forgetting evaluation result of the trained i-th network model based on the inference accuracy of the at least one target historical data set.

[0226] Optionally, a target historical dataset corresponding to a target historical training stage is input into the trained i-th network model to obtain an inference result of the i-th network model on the target historical dataset, where the target historical training stage is any one of at least one historical training stage. The proportion of correctly inferred data in the inference results of the target historical dataset is counted to obtain an inference accuracy rate of the target historical dataset. The at least one historical training stage is processed in the same manner to obtain an inference accuracy rate of the at least one target historical dataset.

[0227] In some embodiments, the forgetting assessment result is not forgotten or forgotten. Not forgotten indicates that there is at least one historical training stage with an inference accuracy rate that is not greater than the accuracy threshold, and forgotten indicates that there is at least one historical training stage with an inference accuracy rate that is not greater than the accuracy threshold. In this case, if the inference accuracy rates of the at least one target historical data set respectively corresponding to the inference accuracy rate of at least one historical training stage are not greater than the accuracy threshold, the forgetting assessment result is determined to be forgotten. If the inference accuracy rates of the at least one target historical data set respectively corresponding to the inference accuracy rate of at least one historical training stage are not greater than the accuracy threshold, the forgetting assessment result is determined to be not forgotten.

[0228] It should be noted that when the forgetting assessment result is forgotten, the forgetting assessment result also includes the configuration information of the forgotten training stage, and the forgotten training stage is a historical training stage in which the inference accuracy in at least one historical training stage is not greater than the accuracy threshold.

[0229] Optionally, the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification, the training evaluation results after modification and the forgetting evaluation results can be input into the training prediction model to obtain the optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby achieving optimization of the configuration information of at least one training stage after the i-th training stage.

[0230] For a detailed implementation method of optimizing the configuration information of at least one training stage after the i-th training stage in multiple training stages based on the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the training evaluation results before modification and the training evaluation results after modification, please refer to the relevant content above and will not be repeated here.

[0231] It should be noted that, when the forgetting evaluation result of the trained i-th network model indicates that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, it means that the inference accuracy of the trained i-th network model for the training data set of the historical training stage is low, and there is forgetting. Therefore, the corresponding training data set of at least one training stage after the i-th training stage includes part or all of the training data set of the forgotten training stage. This can ensure that the data set of the forgotten training stage is reviewed in the subsequent training stage, thereby improving the inference performance of the target network model.

[0232] In a second implementation, after obtaining the trained i-th network model, a forgetfulness assessment is performed on the trained i-th network model to obtain a forgetfulness assessment result. In this case, if the inference accuracy of at least one historical training stage is determined to be no greater than an accuracy threshold based on the forgetfulness assessment result, then the configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized based on the forgetfulness assessment result, the loss curve of the i-th network model, and at least one of the inference results of at least one test dataset. Otherwise, the configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized based on at least one of the loss curve of the i-th network model and the inference results of at least one test dataset.

[0233] For the detailed implementation process of forgetting evaluation for the trained i-th network model, please refer to the relevant content above and will not be repeated here.

[0234] In some embodiments, the loss curve of the i-th network model and at least one of the inference results of at least one test data set, as well as the forgetting assessment result are input into a training prediction model to obtain the optimized configuration information of at least one training stage after the i-th training stage output by the training prediction model, thereby achieving optimization of the configuration information of at least one training stage after the i-th training stage.

[0235] For the detailed implementation process of optimizing the configuration information of at least one training stage after the i-th training stage in multiple training stages based on the loss curve of the i-th network model and at least one of the inference results of at least one test dataset, please refer to the above introduction and will not be repeated here.

[0236] In some embodiments, if a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, the trained i-th network model is directly used as the trained target network model.

[0237] In practical applications, after obtaining the trained i-th network model, the cloud platform may also perform a forgetfulness assessment on the trained i-th network model to obtain a forgetfulness assessment result. In this case, if a user-triggered training completion operation is detected and the i-th training phase is the last of multiple training phases, and based on the forgetfulness assessment result, it is determined that there is no historical training phase with an inference accuracy that is not greater than the accuracy threshold, then the trained i-th network model is used as the trained target network model.

[0238] In some embodiments, if a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, and based on the forgetting evaluation result, it is determined that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on the forgetting evaluation result, the trained i-th network model is continued to be trained to obtain the i-th network model that has been continued to be trained, and the i-th network model that has been continued to be trained is used as the trained i-th network model, and the above-mentioned step of performing forgetting evaluation on the trained i-th network model to obtain the forgetting evaluation result is returned.

[0239] In actual applications, the cloud platform can also display a training record interface, where users can view the configuration information and training records of any training stage. The training record includes the training status of the network model in each training stage.

[0240] For example, the training situation may include a loss curve corresponding to any training stage, an inference accuracy rate corresponding to at least one test data set, a training log, etc., which is not limited in this embodiment of the present application.

[0241] For example, please refer to Figure 13, which is a schematic diagram of a training record interface provided in an embodiment of the present application. The user can view the configuration information and loss curve of training stage n on the training record interface (not shown in Figure 13). Among them, the number of model parameters in the configuration information of training stage n is 35M, the number of self-attention networks in the network model of the nth training stage is 6, the dimension of the hidden layer in each self-attention network is 512, and the number of attention heads is 8.

[0242] Next, the network model training method provided in the embodiment of the present application will be introduced again in conjunction with Figure 14.

[0243] Please refer to Figure 14. A parameter configuration interface is provided. The parameter quantities and types of the target network model are obtained from the parameter configuration interface. The parameter quantities and types of the target network model are input into the training prediction model to obtain the first training configuration information output by the training prediction model. Based on the first training configuration information, the training configuration information of the target network model is determined. The cloud platform provides a configuration preview interface. In response to the user's modification operation on the first training configuration information, the first training configuration information is modified to obtain the second training configuration information. Based on the second training configuration information, the model planning information, training planning information, and data planning information corresponding to the multiple training stages are determined.

[0244] Let i = 1. If the i-th training stage is the first training stage among multiple training stages, the i-th network model is generated based on the configuration information of the i-th training stage. If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0245] The i-th network model is trained according to the configuration information of the i-th training stage to obtain a trained i-th network model, a training evaluation result of the trained i-th network model is determined, and a training evaluation interface is provided. If the user confirms that the i-th network model has not completed training, a re-planning operation can be triggered, returning to the above step of determining the training configuration information of the target network model and re-determining the training configuration information of the target network model.

[0246] After confirming that the trained i-th network model has completed training, the user can trigger a training completion operation on the training evaluation interface. The cloud platform detects the user-triggered training completion operation and, if the i-th training stage is not the last training stage among the multiple training stages, sets i=i+1 and returns to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages. Furthermore, after obtaining the trained i-th network model, the cloud platform can also perform a forgetfulness evaluation on the trained i-th network model to obtain a forgetfulness evaluation result. If, based on the forgetfulness evaluation result, it is determined that the inference accuracy of at least one historical training stage is not greater than an accuracy threshold, then, based on at least one of the loss curve of the i-th network model and the inference results of at least one test dataset, as well as the training evaluation result before modification, the training evaluation result after modification, and the forgetfulness evaluation result, the configuration information of at least one training stage after the i-th training stage among the multiple training stages is optimized. Otherwise, based on the loss curve of the i-th network model and at least one of the inference results of the at least one test data set, and the training evaluation results before modification and the training evaluation results after modification, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0247] In the case where the i-th training stage is the last training stage among multiple training stages, the cloud platform may also perform a forgetting evaluation on the trained i-th network model after obtaining the trained i-th network model to obtain a forgetting evaluation result. If it is determined based on the forgetting evaluation result that there is no inference accuracy of at least one historical training stage that is not greater than the accuracy threshold, then the trained i-th network model is used as the trained target network model, that is, the training of the target network model is completed.

[0248] In an embodiment of the present application, the user can input the parameter amount and type of the target network model in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models with different parameter amounts and different types, so that the parameter amount of the model, the type of model and the determined training configuration information are more adapted. In addition, the embodiment of the present application can also divide the training process of the network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training. In this way, fine control of the network model training can be achieved in stages, and since the parameter amount of the network model is gradually increased with the training stage, the convergence speed of the network model can be effectively accelerated, the amount of calculation required for network model training can be saved, and the cost of network model training can be reduced.

[0249] Since the training evaluation results in the embodiment of the present application also include the cloud platform's prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model. When the training evaluation interface also includes the loss curve of the i-th network model and / or the inference accuracy corresponding to at least one test data set, the training evaluation results of the network model obtained in each training stage can be displayed to the user in real time through the training evaluation interface, so that the user can control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training. In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can adjust the configuration information of the subsequent training stages in a timely manner based on the interaction between the user and the cloud platform, the loss curve and the inference results of the test data set, so that the final trained network model is more in line with user expectations.

[0250] Figure 15 is a schematic diagram of the structure of a network model training device provided in an embodiment of the present application. The network model training device can be implemented by software, hardware, or a combination of both to form a cloud platform or part or all of an electronic device with model training capabilities. Referring to Figure 15 , the device includes: a providing module 1501, an acquiring module 1502, a determining module 1503, and a training module 1504.

[0251] The module 1501 is provided to provide a parameter configuration interface, which is used to instruct the user to input the parameter amount and type of the target network model to be trained. The detailed implementation process refers to the corresponding content in the above embodiments and will not be repeated here.

[0252] The acquisition module 1502 is used to acquire the parameter quantity and type of the target network model from the parameter configuration interface. The detailed implementation process is referred to the corresponding content of each embodiment above and will not be repeated here.

[0253] Determination module 1503 is configured to determine training configuration information for the target network model based on the parameter quantity and type of the target network model. The training process of the target network model is divided into multiple training stages, where different training stages correspond to different model parameter quantities, and the model parameter quantities of the multiple training stages increase in sequence according to the training sequence. The training configuration information includes configuration information for the multiple training stages. The detailed implementation process is referenced to the corresponding content in the above-mentioned embodiments and will not be repeated here.

[0254] The training module 1504 is used to train the target network model according to the configuration information of multiple training stages. The detailed implementation process is referred to the corresponding content of each embodiment above and will not be repeated here.

[0255] Optionally, the determining module 1503 is specifically configured to:

[0256] Inputting the parameter quantity and type of the target network model into the training prediction model to obtain first training configuration information output by the training prediction model;

[0257] Based on the first training configuration information, training configuration information of the target network model is determined.

[0258] Optionally, the determining module 1503 is specifically configured to:

[0259] Providing a configuration preview interface, the configuration preview interface including first training configuration information;

[0260] In response to a user's modification operation on the first training configuration information, modify the first training configuration information to obtain second training configuration information;

[0261] Based on the second training configuration information, training configuration information of the target network model is determined.

[0262] Optionally, the configuration information of the multiple training stages includes model planning information, training planning information, and data planning information corresponding to the multiple training stages respectively;

[0263] Among them, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration of the training data of the corresponding training stage.

[0264] Optionally, when the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

[0265] Optionally, the model planning information also includes the model parameter quantities of the corresponding training stage.

[0266] Optionally, the training planning information includes the number of model iterations and a learning rate interval in a corresponding training phase, where the learning rate interval indicates a range of a change step size of the model parameters in the corresponding training phase.

[0267] Optionally, the training planning information further includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

[0268] Optionally, the training plan information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates how the step size of the model parameters changes, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage in multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

[0269] Optionally, the data planning information includes a ratio of the training data set of the corresponding training stage to the total training data set, where the total training data set is the sum of the training data sets of multiple training stages.

[0270] Optionally, the data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage in multiple training stages.

[0271] Optionally, the training module 1504 is specifically configured to:

[0272] Let i=1, determine the i-th network model based on the configuration information of the i-th training stage in the multiple training stages;

[0273] Training the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model;

[0274] Determine the training evaluation result of the trained i-th network model;

[0275] Providing a training evaluation interface, which includes the training evaluation results of the trained i-th network model, and the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training;

[0276] If a user-triggered training completion operation is detected and the i-th training stage is not the last training stage among the multiple training stages, then set i=i+1 and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages;

[0277] If a user-triggered completion training operation is detected and the i-th training stage is the last training stage among multiple training stages, the trained i-th network model is used as the trained target network model.

[0278] Optionally, the training module 1504 is specifically configured to:

[0279] If the i-th training stage is the first training stage among multiple training stages, generating the i-th network model based on the configuration information of the i-th training stage;

[0280] If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

[0281] Optionally, the configuration information of the i-th training stage includes model planning information of the i-th training stage;

[0282] The training module 1504 is specifically used to:

[0283] Providing a parameter expansion interface, the parameter expansion interface is used to instruct the user to select a parameter expansion strategy, the parameter expansion strategy is used to determine the initial value of the newly added parameter of the i-th network model relative to the i-1-th network model;

[0284] Get the parameter amplification strategy from the parameter amplification interface;

[0285] Based on the parameter augmentation strategy and the model planning information of the i-th training stage, the parameters of the i-1-th network model are augmented to obtain the i-th network model.

[0286] Optionally, the training module 1504 is specifically configured to:

[0287] Determine the loss curve of the i-th network model, which indicates how the loss value changes with the number of model iterations. The loss value indicates the gap between the predicted value of the network model and the true value;

[0288] Inputting at least one test data set into the trained i-th network model, and obtaining an inference result of at least one test data set output by the trained i-th network model;

[0289] Determine a training evaluation result of the trained i-th network model based on the loss curve and the inference result of the at least one test data set.

[0290] Optionally, the training module 1504 is specifically configured to:

[0291] Based on at least one of the loss curve and the inference result of the at least one test data set, configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0292] Optionally, the training module 1504 is specifically configured to:

[0293] In response to a user's modification operation on the training evaluation result, the training evaluation result is modified to obtain a modified training evaluation result;

[0294] Optimizing configuration information of at least one training stage after the i-th training stage among the plurality of training stages based on at least one of a loss curve and an inference result of at least one test data set, including:

[0295] Based on at least one of the loss curve and the inference result of at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0296] Optionally, the device further comprises:

[0297] a forgetfulness assessment module, configured to perform a forgetfulness assessment on the trained i-th network model to obtain a forgetfulness assessment result, wherein the forgetfulness assessment result is used to indicate the reasoning status of the trained i-th network model on the training data set of a historical training phase, where the historical training phase is a training phase that is prior to the i-th training phase among multiple training phases;

[0298] The training module 1504 is specifically used to:

[0299] If it is determined based on the forgetting evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on at least one of the loss curve and the inference results of at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification, and the forgetting evaluation result, the configuration information of at least one training stage after the i-th training stage in the multiple training stages is optimized.

[0300] In an embodiment of the present application, the user can input the parameter amount and type of the target network model in the parameter configuration interface, thereby determining the training configuration information of the target network model based on the parameter amount and type of the target network model. In this way, the corresponding training configuration information can be determined for network models with different parameter amounts and different types, so that the parameter amount of the model, the type of model and the determined training configuration information are more adapted. In addition, the embodiment of the present application can also divide the training process of the network model with a large parameter amount into multiple stages, and the model parameter amounts of multiple training stages are increased in sequence according to the order of training. In this way, fine control of the network model training can be achieved in stages, and since the parameter amount of the network model is gradually increased with the training stage, the convergence speed of the network model can be effectively accelerated, the amount of calculation required for network model training can be saved, and the cost of network model training can be reduced.

[0301] Since the training evaluation results in the embodiment of the present application also include the cloud platform's prediction of whether the trained i-th network model has completed training, this can provide reliable training completion prediction information for users who do not have training experience, thereby improving the training efficiency of the network model. When the training evaluation interface also includes the loss curve of the i-th network model and / or the inference accuracy corresponding to at least one test data set, the training evaluation results of the network model obtained in each training stage can be displayed to the user in real time through the training evaluation interface, so that the user can control the training effect of each training stage, so that the user can promptly discover possible problems in the training process and improve the efficiency of network model training. In actual applications, the training evaluation results determined by the cloud platform may not meet the user's expectations. In this case, the user can modify the training evaluation results. In this way, the cloud platform can adjust the configuration information of the subsequent training stages in a timely manner based on the interaction between the user and the cloud platform, the loss curve and the inference results of the test data set, so that the final trained network model is more in line with user expectations.

[0302] It should be noted that the network model training device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the training of the network model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the network model training device provided in the above embodiment and the network model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0303] The providing module, acquiring module, determining module, and training module in the network model training device shown in FIG15 can all be implemented via software, hardware, or a combination of software and hardware. Next, the implementation of the providing module in the network model training device is described using the providing module as an example. Similarly, the implementation of the other modules described above can refer to the implementation of the providing module.

[0304] As an example of a software functional unit, a module may include code running on a computing instance. The computing instance may include at least one of a physical host, a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0305] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same VPC or across multiple virtual private clouds (VPCs). Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0306] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, the module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0307] The multiple computing devices included in the provisioning module can be distributed in the same region or in different regions. The multiple computing devices included in the provisioning module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the provisioning module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0308] In other embodiments, the providing module in the training device of the network model can be used to execute any step in the training method of the above-mentioned network model, the acquiring module in the training device of the network model can be used to execute any step in the training method of the above-mentioned network model, the determining module in the training device of the network model can be used to execute any step in the training method of the above-mentioned network model, and the training module in the training device of the network model can be used to execute any step in the training method of the above-mentioned network model. The steps that the providing module, acquiring module, determining module and training module in the training device of the network model are responsible for implementing can be specified as needed. Different steps in the training method of the above-mentioned network model are respectively implemented through the providing module, acquiring module, determining module and training module in the training device of the network model, thereby realizing all the functions of the training device for providing the network model.

[0309] The present application also provides a computing device 100. As shown in FIG16 , computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0310] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG18 shows only one line, but this does not imply that there is only one bus or only one type of bus. Bus 102 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0311] Processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an integrated circuit, and the like. The integrated circuit may include an ASIC, a programmable logic device (PLD), or a combination thereof. Optionally, the PLD may be a CPLD, an FPGA, a GAL, or any combination thereof.

[0312] The memory 106 may include volatile memory, such as random access memory (RAM). The memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid state drive (SSD), or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 106 exists independently and is connected to the processor 104 via the bus 102, or the memory 106 is integrated with the processor 104.

[0313] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to implement the aforementioned network model training method. In other words, the memory 106 stores instructions for executing the network model training method.

[0314] For example, the memory 106 stores executable code, and the processor 104 executes the executable code to implement the functions of the providing module, obtaining module, determining module, and training module as shown in Figure 16 above. In other words, the memory 106 stores instructions for executing the training method of the network model provided in the embodiment of the present application.

[0315] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 100 and other devices or communication networks. Network interfaces include wired communication interfaces or wireless communication interfaces. Examples of wired communication interfaces include Ethernet interfaces. Ethernet interfaces can be optical interfaces, electrical interfaces, or a combination thereof. Wireless communication interfaces can include wireless local area network (WLAN) interfaces, cellular network communication interfaces, or a combination thereof.

[0316] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes multiple computing devices. The computing devices can be servers, such as central servers, edge servers, or local servers in a local data center. In some embodiments, the computing devices can also be terminal devices such as desktop computers, laptop computers, or smartphones.

[0317] As shown in Figure 17, the computing device cluster includes multiple computing devices 100. The same instructions for executing the training method of the above network model may be stored in the memory 106 of the multiple computing devices 100 in the computing device cluster.

[0318] In some possible implementations, the memories 106 of the multiple computing devices 100 in the computing device cluster may each store a portion of the instructions for executing the aforementioned network model training method. In other words, the combination of the multiple computing devices 100 may collectively execute all the instructions for executing the aforementioned network model training method.

[0319] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing part of the functions of the above-mentioned network model training method. In other words, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of some or all of the modules included in the above-mentioned network model training apparatus.

[0320] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG18 shows a possible implementation. As shown in FIG18 , two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is made through a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of a portion of the modules included in the training device for the network model described above. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of another portion of the modules included in the training device for the network model described above.

[0321] It should be understood that the functions of the computing device 100A shown in FIG18 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0322] The present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection method of the computing device cluster described in Figures 17 and 18. The difference is that the memory 106 of one or more computing devices 100 in the computing device cluster can store the same instructions for executing the network model training method.

[0323] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the network model training method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the network model training method.

[0324] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a computer, the computer executes the steps of the network model training method described in the above embodiment, or executes the steps of the network model training method described in the above embodiment.

[0325] The present application also provides a computer program product comprising instructions that, when executed on a computer, cause the computer to execute the steps of the network model training method described in the above embodiment. Alternatively, the present application also provides a computer program that, when executed on a computer, causes the computer to execute the steps of the network model training method described in the above embodiment.

[0326] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0327] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0328] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the parameter quantities and types of the target network model involved in the embodiments of this application are obtained with full authorization.

[0329] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A network model training method, characterized in that: The method comprises: Providing a parameter configuration interface, wherein the parameter configuration interface is used to instruct the user to input the parameter amount and type of the target network model to be trained; Acquire the parameter quantity and type of the target network model from the parameter configuration interface; Based on the parameter amount and type of the target network model, determine the training configuration information of the target network model; wherein the training process of the target network model is divided into multiple training stages, different training stages correspond to different model parameter amounts, and the model parameter amounts of the multiple training stages are increased in sequence according to the training sequence, and the training configuration information includes the configuration information of the multiple training stages; The target network model is trained according to the configuration information of the multiple training stages.

2. The method according to claim 1, characterized in that The determining, based on the parameter amount and type of the target network model, training configuration information of the target network model includes: Inputting the parameter quantity and type of the target network model into a training prediction model to obtain first training configuration information output by the training prediction model; Based on the first training configuration information, training configuration information of the target network model is determined.

3. The method according to claim 2, characterized in that The determining, based on the first training configuration information, the training configuration information of the target network model includes: Providing a configuration preview interface, wherein the configuration preview interface includes the first training configuration information; In response to the user's modification operation on the first training configuration information, modify the first training configuration information to obtain second training configuration information; Based on the second training configuration information, training configuration information of the target network model is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The configuration information of the multiple training stages includes model planning information, training planning information and data planning information respectively corresponding to the multiple training stages; Among them, the model planning information indicates the model structure of the corresponding training stage, the training planning information indicates the training configuration parameters of the corresponding training stage, and the data planning information indicates the configuration of the training data of the corresponding training stage.

5. The method according to claim 4, characterized in that When the type of the target network model indicates that the target network model includes a self-attention network, the model planning information includes the number of self-attention networks in the network model of the corresponding training stage, the dimension of the hidden layer in each self-attention network, and the number of attention heads.

6. The method according to claim 5, characterized in that The model planning information also includes the model parameter quantities of the corresponding training stage.

7. The method according to any one of claims 4 to 6, characterized in that: The training planning information includes the number of model iterations and the learning rate interval of the corresponding training stage, and the learning rate interval indicates the range of the change step size of the model parameters of the corresponding training stage.

8. The method according to claim 7, characterized in that The training planning information also includes an expected loss value of the corresponding training stage, where the expected loss value indicates an expected gap between a predicted value and a true value of the network model of the corresponding training stage.

9. The method according to claim 7 or 8, characterized in that The training planning information also includes at least one of the number of warm-up iterations, the learning rate decay type, the parameter precision and the batch size, wherein the learning rate decay type indicates the changing manner of the changing step size of the model parameters, the parameter precision indicates the precision of the model parameters, the batch size indicates the size of the training data used for one iteration, the number of warm-up iterations indicates the number of iterations required for the learning rate of the first training stage among the multiple training stages to change from zero to the maximum learning rate, and the maximum learning rate is the maximum value in the learning rate range of the first training stage.

10. The method according to any one of claims 4 to 9, characterized in that: The data planning information includes the ratio of the training data set of the corresponding training stage to the total training data set, and the total training data set is the sum of the training data sets of the multiple training stages.

11. The method according to claim 10, characterized in that The data planning information also includes at least one of pruning configuration information and distillation preheating configuration information, wherein the pruning is used to screen the training data set of the corresponding training stage to reduce redundant information, and the distillation preheating is used to determine the initial values ​​of the model parameters of the first training stage among the multiple training stages.

12. The method according to claim 1, characterized in that The training of the target network model according to the configuration information of the multiple training stages includes: Let i=1, determine the i-th network model based on the configuration information of the i-th training stage in the multiple training stages; Training the i-th network model according to the configuration information of the i-th training stage to obtain a trained i-th network model; Determine the training evaluation result of the trained i-th network model; Providing a training evaluation interface, the training evaluation interface including the training evaluation result of the trained i-th network model, the training evaluation interface is also used to instruct the user to confirm whether the trained i-th network model has completed training; If the training completion operation triggered by the user is detected and the i-th training stage is not the last training stage among the multiple training stages, then set i=i+1, and return to the step of determining the i-th sub-model based on the configuration information of the i-th training stage among the multiple training stages; If the completion of the training operation triggered by the user is detected and the i-th training stage is the last training stage among the multiple training stages, the trained i-th network model is used as the trained target network model.

13. The method according to claim 12, characterized in that The determining, based on the configuration information of the i-th training stage in the multiple training stages, the i-th network model comprises: If the i-th training stage is the first training stage among the multiple training stages, generating the i-th network model based on the configuration information of the i-th training stage; If the i-th training stage is not the first training stage, the trained i-1-th network model is amplified based on the configuration information of the i-th training stage to obtain the i-th network model.

14. The method according to claim 13, characterized in that The configuration information of the i-th training stage includes the model planning information of the i-th training stage; The step of amplifying the trained (i-1)th network model based on the configuration information of the (i)th training stage to obtain the (i)th network model comprises: Providing a parameter augmentation interface, the parameter augmentation interface is used to instruct the user to select a parameter augmentation strategy, the parameter augmentation strategy is used to determine the initial value of the parameter newly added to the i-th network model relative to the i-1-th network model; Acquiring the parameter amplification strategy from the parameter amplification interface; Based on the parameter augmentation strategy and the model planning information of the i-th training stage, the parameters of the i-1-th network model are augmented to obtain the i-th network model.

15. The method according to claim 12, characterized in that The step of determining the training evaluation result of the trained i-th network model comprises: Determine a loss curve of the i-th network model, wherein the loss curve indicates how a loss value changes with the number of model iterations, and the loss value indicates a gap between a predicted value and a true value of the network model; Inputting at least one test data set into the trained i-th network model to obtain an inference result of the at least one test data set output by the trained i-th network model; Based on the loss curve and the inference result of the at least one test data set, a training evaluation result of the trained i-th network model is determined.

16. The method according to claim 15, characterized in that If the completion of the training operation triggered by the user is detected and the i-th training stage is not the last training stage among the multiple training stages, the method further includes: Based on at least one of the loss curve and the inference result of the at least one test data set, The configuration information of at least one training stage after the i-th training stage is optimized.

17. The method according to claim 16, characterized in that After providing the training evaluation interface, the method further includes: In response to the user's modification operation on the training evaluation result, modify the training evaluation result to obtain a modified training evaluation result; The optimizing, based on at least one of the loss curve and the inference result of the at least one test data set, configuration information of at least one training stage after the i-th training stage among the multiple training stages comprises: Based on at least one of the loss curve and the inference result of the at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, the configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

18. The method according to claim 17, characterized in that The method further comprises: Performing a forgetting evaluation on the trained i-th network model to obtain a forgetting evaluation result, wherein the forgetting evaluation result is used to indicate the reasoning of the trained i-th network model on the training data set of a historical training stage, wherein the historical training stage is a training stage before the i-th training stage among the multiple training stages; The optimizing, based on at least one of the loss curve and the inference result of the at least one test data set, and the training evaluation result before modification and the training evaluation result after modification, configuration information of at least one training stage after the i-th training stage in the multiple training stages comprises: If it is determined based on the forgetting evaluation result that the inference accuracy of at least one historical training stage is not greater than the accuracy threshold, then based on at least one of the loss curve and the inference results of the at least one test data set, as well as the training evaluation result before modification, the training evaluation result after modification and the forgetting evaluation result, the configuration information of at least one training stage in the multiple training stages that is located after the i-th training stage is optimized.

19. A network model training device, characterized in that: The device comprises: A module is provided for providing a parameter configuration interface, wherein the parameter configuration interface is used to instruct a user to input the parameter amount and type of a target network model to be trained; An acquisition module, used to acquire the parameter quantity and type of the target network model from the parameter configuration interface; A determination module, configured to determine the training configuration information of the target network model based on the parameter amount and type of the target network model; wherein the training process of the target network model is divided into a plurality of training stages, the model parameter amounts corresponding to different training stages are different, and the model parameter amounts of the plurality of training stages are increased in sequence according to the training sequence, and the training configuration information includes the configuration information of the plurality of training stages; A training module is used to train the target network model according to the configuration information of the multiple training stages.

20. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-18.

21. A computer-readable storage medium, characterized in that: The storage medium includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 18.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 18.

Images