Model training method and device

Abstract

The invention provides a model training method and device, and relates to the field of machine learning. The method and the device are used for improving model training efficiency. The method comprises: training a first model; a loss function value of the first model is monitored. And after determining that the loss function value of the first model meets a preset condition, amplifying the first model to obtain a second model. Wherein the amplified parameters in the second model are obtained by copying the parameters in the first model. And training the second model.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a model training method and device. Background Art

[0002] As the mainstream method in the field of machine learning, deep learning has been widely used in many fields such as natural language processing, computer vision, and search recommendation. With the development of deep learning technology, the performance of the model is getting better and better, but at the same time, the scale of model parameters has increased from millions to billions or even hundreds of billions, so the amount of computation required to train the model is extremely large, which brings great time and resource requirements for model training.

[0003] In order to speed up the model training process, the prior art can first build an initial model with fewer parameters and train the initial model. Since there are fewer parameters, the amount of calculation required to train the model is also less. Then, during the training process, the model is gradually expanded in stages, and finally a model that meets the usage requirements is trained.

[0004] So, in the process of training the model by building an initial model and then gradually expanding the model in stages, how to further improve the efficiency of model training is a problem that needs to be solved at present. Summary of the invention

[0005] The present application provides a model training method and device for improving the efficiency of model training.

[0006] In a first aspect, a model training method is provided, comprising: training a first model. Monitoring a loss function value of the first model. After determining that the loss function value of the first model meets a preset condition, amplifying the first model to obtain a second model. The amplified parameters in the second model are obtained by copying the parameters in the first model. Training the second model.

[0007] In the above method, in the scenario where model training is performed by gradually expanding the model in stages, the method can monitor the loss function value (which can be referred to as Loss value) of the first model during the training of the model to be expanded (the model to be expanded can be the initial model or the model of any intermediate stage in the training process. For the sake of distinction, it is referred to as the "first model" below), and after determining that the Loss value of the first model meets the preset conditions, expand the first model to obtain the second model. Among them, the parameters expanded in the second model are obtained by copying the parameters in the first model (for example, the first model includes x coding layers, and the second model includes 2x coding layers, then the parameters in the 1st to xth coding layers and the parameters in the (x+1)th to 2xth coding layers in the second model are respectively obtained by copying the parameters in the x coding layers in the first model). Then the second model is trained. Similarly, for the second model and the subsequent model expanded from the second model, the timing of expansion can also be determined in a similar manner as above (that is, expansion after determining that the loss function value of the model meets the preset conditions). This allows the appropriate time for amplification to be determined quickly without the need for multiple trials.

[0008] In one implementation, training the second model includes: obtaining a first loss function value of the second model when the second model is trained using a preset learning rate. Determining a first learning rate corresponding to the first loss function value according to a correspondence between a learning rate and a loss function value during training of the first model. Using the first learning rate as an initial learning rate, training the second model.

[0009] Through the above implementation method, the present application can adaptively determine the initial learning rate that matches the current model, and use the initial learning rate to train the augmented model, which can achieve the effect of improving the convergence speed of the model and saving the model training overhead.

[0010] In one implementation, the method further includes: determining the preset learning rate corresponding to the target number of iterations in a preset correspondence between the learning rate and the number of iterations; the target number of iterations is the number of model iterations that have been performed by the time the first model is amplified to obtain the second model.

[0011] In one implementation, the preset condition includes: the change amplitude of the loss function value of the first model monitored in a plurality of consecutive sliding windows is less than the amplitude threshold. Alternatively, the preset condition includes: within p consecutive sliding windows, the number of sliding windows in which the change amplitude of the loss function value of the first model monitored is less than the amplitude threshold exceeds the number threshold q.

[0012] In the above implementation, on the one hand, it can be determined whether the current model has reached a suitable convergence degree; on the other hand, it can avoid the interference of unstable factors on the judgment result.

[0013] In one implementation, the first model includes n coding layers constructed in a stacking manner, where n is a positive integer. After determining that the loss function value of the first model meets a preset condition, the first model is amplified to obtain a second model, which includes: after determining that the loss function value of the first model meets the preset condition, m coding layers are added to the end of the n coding layers included in the first model close to the decoding end to obtain the second model; the parameters in the m coding layers are the parameters in the m coding layers close to the decoding end among the n coding layers, which are copied from shallow to deep, and m is a positive integer less than or equal to n.

[0014] In one implementation, the method further includes: obtaining the number of coding layers L of the target model and the number of stages S in the model training process. According to the number of coding layers L and the number of stages S, determining the number of coding layers n of the first model.

[0015] In one implementation, after determining that the loss function value of the first model meets a preset condition, the first model is expanded to obtain a second model, including: after determining that the loss function value of the first model meets the preset condition, the width of the first model is expanded to obtain the second model.

[0016] In one implementation, after determining that the loss function value of the first model meets a preset condition, the first model is amplified to obtain a second model, including: after determining that the loss function value of the first model meets the preset condition, data amplification of the first model is performed to obtain the second model.

[0017] In a second aspect, a model training device is provided, comprising: a training unit for training a first model; a monitoring unit for monitoring the loss function value of the first model; an amplification unit for amplifying the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition; wherein the amplified parameters in the second model are obtained by copying the parameters in the first model; and the training unit is also used to train the second model.

[0018] In one implementation, the training unit is also used to train the second model, including: a training unit, specifically used to obtain a first loss function value of the second model when the second model is trained using a preset learning rate; a training unit, specifically used to determine a first learning rate corresponding to the first loss function value based on the correspondence between the learning rate and the loss function value during the training of the first model; a training unit, specifically used to use the first learning rate as the initial learning rate to train the second model.

[0019] In one implementation, the training unit is further used to determine the preset learning rate corresponding to the target number of iterations in the preset correspondence between the learning rate and the number of iterations; the target number of iterations is the number of model iterations that have been performed by the time the first model is amplified to obtain the second model.

[0020] In one implementation, the preset condition includes: in a plurality of consecutive sliding windows, the change amplitude of the loss function value of the first model is monitored to be less than the amplitude threshold. Alternatively, the preset condition includes: in p consecutive sliding windows, the number of sliding windows in which the change amplitude of the loss function value of the first model is monitored to be less than the amplitude threshold exceeds the number threshold q.

[0021] In one implementation, the first model includes n coding layers constructed in a stacking manner, where n is a positive integer.

[0022] An amplification unit, used to amplify the first model after determining that the loss function value of the first model meets a preset condition, so as to obtain a second model, including: an amplification unit, used to add m coding layers to the end of the n coding layers included in the first model close to the decoding end after determining that the loss function value of the first model meets the preset condition, so as to obtain the second model; the parameters in the m coding layers are the parameters in the m coding layers close to the decoding end among the n coding layers, which are copied in a direction from shallow to deep, and m is a positive integer less than or equal to n.

[0023] In one implementation, the training unit is further used to obtain the number of coding layers L of the target model and the number of stages S in the model training process. The training unit is further used to determine the number of coding layers n of the first model according to the number of coding layers L and the number of stages S.

[0024] In one implementation, an amplification unit is used to amplify the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition, including: an amplification unit is used to perform width amplification on the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition.

[0025] In one implementation, an amplification unit is used to amplify the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition, including: an amplification unit is used to perform data amplification on the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition.

[0026] In a third aspect, a model training device is provided, comprising: a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to implement a method as described in the first aspect or any implementation method in the first aspect.

[0027] According to a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a processor, a method as in the first aspect or any implementation method in the first aspect is implemented.

[0028] According to a fifth aspect, a computer program product is provided, the computer program product comprising instructions, which, when executed on a processor, implement the method according to the first aspect or any one of the implementations of the first aspect.

[0029] The technical effects produced by any implementation method in the above-mentioned second to fifth aspects and each aspect can refer to the above-mentioned first aspect and the corresponding implementation method in the first aspect, and the repetitive parts will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 One of the flowchart diagrams of a model training process provided in an embodiment of the present application;

[0031] Figure 2 A second flowchart of a model training process provided in an embodiment of the present application;

[0032] Figure 3 A third flowchart of a model training process provided in an embodiment of the present application;

[0033] Figure 4 One of the flowcharts of a model training method provided in an embodiment of the present application;

[0034] Figure 5 A second flowchart of a model training method provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of width expansion in a model training method provided in an embodiment of the present application;

[0036] Figure 7A schematic diagram of data augmentation in a model training method provided in an embodiment of the present application;

[0037] Figure 8 A third flowchart of a model training method provided in an embodiment of the present application;

[0038] Fig. 9 A fourth flowchart of a model training method provided in an embodiment of the present application;

[0039] Fig.10 One of the structural schematic diagrams of a model training device provided in an embodiment of the present application;

[0040] Fig.11 The second structural diagram of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0042] At present, in order to accelerate the model training process, the related technologies often adopt the method of first building an initial model with fewer parameters, and then gradually expanding the model in stages to perform model training. The following is an example of the specific implementation process of model training used in the related technologies:

[0043] In the first related technology, a method for accelerating the training of a neural machine translation system based on a stacking algorithm is provided. Figure 1 As shown, the method includes: first constructing an initial transformer model (such as Figure 1 The transformer model includes an encoder 111 and a decoder 112. The encoder 111 includes an encoding layer ( Figure 1 Then, the model 11 is trained, and the parameters of each encoding layer in the encoder 111 can be learned during the training process. After the model 11 is trained for a preset number of times, the model is amplified. The amplified model is as follows: Figure 1 As shown in model 12. Specifically, model 12 includes an encoder 121 and a decoder 122. The encoder 121 includes 12 coding layers, which are obtained by copying and stacking the 6 coding layers in model 11 (the parameters of the 6 coding layers have been trained). In addition, the decoder 122 can be obtained by copying the decoder 112.

[0044] Then, the model 12 is trained, and the parameters of each encoding layer in the encoder 121 can be learned during the training process. After the model 12 is trained for a preset number of times, the model is amplified. Figure 1 As shown in model 13. Specifically, model 13 includes an encoder 131 and a decoder 132. The encoder 131 includes 18 encoding layers, which are obtained by copying and stacking the 12 encoding layers in model 12 (the parameters of the 12 encoding layers have been trained).

[0045] Afterwards, the model 13 is trained, and after the training of the model 13 is completed, the model is expanded. Figure 1 In the example shown, during the amplification process, the top 6 coding layers (i.e., the 6 coding layers close to the decoder end) in the encoder of the model are copied twice each time and stacked on the previous coding layers. For example, in model 12, the 6 coding layers close to the decoder end are copied twice and stacked on the 6 coding layers of model 11, thereby constructing encoder 121 in model 12. The process of amplifying other models can be understood in the same principle, and the repeated content will not be repeated here.

[0046] like Figure 1 As shown, after k times of step-wise expansion according to the above method, a model 1k including 6k encoding layers (i.e., a model whose model depth meets the requirements) can be constructed. After training the model 1k, the trained model can be used for translation.

[0047] In the embodiments of the present application, it is considered that: on the one hand, in the implementation process of the above-mentioned related technology 1, it is usually necessary to go through multiple tests to obtain a model with accuracy that meets the requirements. Figure 1 Take k=4 as an example (i.e., from model 11 to model 14 as the target model, four stages of model training (model 11-model 12-model 13-model 14) are required), as Figure 2As shown: In the first test, the technicians first need to set the number of iterations corresponding to each stage of model training and the initial learning rate after each amplification. For example, the number of iterations of the 4-stage model training is set to [30, 30, 30, 30], and the initial learning rate after 3 amplifications is set to [0.75, 0.50, 0.25]. Next, in the process of model training, the initial model 11 is amplified after 30 iterations, and the initial learning rate of the amplified model 12 is 0.75; then the model 12 is amplified after 30 iterations, and the initial learning rate of the amplified model 13 is 0.5; then the model 13 is amplified after 30 iterations, and the initial learning rate of the amplified model 14 is 0.25; then the model 14 is iterated 30 times to complete the model training process. Then the accuracy of the trained model 14 is tested. After determining that the accuracy of model 14 does not meet the standard, adjust the number of iterations and initial learning rate of the model training at each stage. For example, adjust the number of iterations to [40, 20, 20, 40], adjust the initial learning rate to [0.7, 0.5, 0.2], and conduct a second test. And so on, until the trained model reaches the accuracy target.

[0048] On the other hand, in the implementation process of the above-mentioned related technology 1, it is usually necessary for technicians to first formulate an expansion strategy, and then train the model according to the expansion strategy. For example, if the target model required is a model including 48 coding layers, the technicians need to first formulate an expansion strategy such as the number of coding layers of the initial model and the number of coding layers for each expansion, and then train the model according to the expansion strategy. Therefore, there is a problem of poor algorithm versatility and ease of use.

[0049] In a second related technology, a method for efficient training of bidirectional encoder representations from transformers (BERT) by progressively stacking is provided. The method comprises: step 1, constructing a BERT model including L encoding layers and performing a certain number of rounds of training. For example, the constructed BERT model is as follows Figure 3 As shown in the model 21, the model 21 includes an encoder 211 and a classifier 212 for encoding the embedding layer information, and the encoder 211 includes L encoding layers. Step 2, the encoder parameters of the trained BERT model are copied in sequence to construct a BERT model containing 2L encoding layers (such as Figure 3 The model is shown in model 22) and the model is trained. Repeat the above steps n times to obtain a L×2 nBERT model with 4 encoding layers.

[0050] For the above-mentioned second related technology, the embodiments of the present application take into account: on the one hand, similar to the first related technology, in the second related technology, multiple experiments are also required to determine the appropriate hyperparameters (including the number of iterations corresponding to each stage of model training and the initial learning rate after each amplification), which requires more overhead. On the other hand, similar to the first related technology, the second related technology also requires technicians to first formulate an amplification strategy and then perform model training according to the amplification strategy.

[0051] In addition, the above-mentioned related technologies can also use a learning rate baseline to determine the initial learning rate after each expansion. For example, the learning rate baseline can be the corresponding relationship between the initial learning rate and the number of iterations. Exemplarily, the learning rate baseline can be specifically: when the 1st to 30th iteration, the corresponding initial learning rate is 0.66; when the 31st to 60th iteration, the corresponding initial learning rate is 0.33. Then, if the model 21 is expanded after 20 iterations, since 20 iterations belong to the range of "1st to 30th iterations", the expanded model 22 uses 0.66 as the initial learning rate for training. However, when the learning rate baseline is used to determine the initial learning rate after each expansion, although multiple experimental adjustments are not required to determine the initial learning rate, this method cannot determine the preferred initial learning rate corresponding to each stage model, so it may result in a lower accuracy of the model. For example, in the above example, when the model 21 is expanded after 20 iterations, the preferred initial learning rate may not be 0.66, which may result in a lower accuracy of the model.

[0052] In view of the above technical problems, the embodiment of the present application considers that if the training of the model can be monitored in real time during the model training process, and the amplification time is determined according to the monitoring information. In this way, the appropriate amplification time can be quickly determined without multiple experiments. Based on the above considerations, the embodiment of the present application provides a model training method. In the scenario of model training by gradually amplifying the model in stages, the method can monitor the loss function value of the first model (which can be referred to as the Loss value for short) during the training of the model to be amplified (the model to be amplified can be the initial model or the model of any intermediate stage in the training process. For the convenience of distinction, it is referred to as the "first model" below), and after determining that the Loss value of the first model meets the preset conditions, the first model is amplified to obtain the second model. Among them, the parameters amplified in the second model are obtained by copying the parameters in the first model (for example, the first model includes x coding layers, and the second model includes 2x coding layers, then the parameters in the 1st to x coding layers and the parameters in the (x+1)th to 2x coding layers in the second model are respectively obtained by copying the parameters in the x coding layers in the first model). Then the second model is trained. Similarly, for the second model and subsequent models obtained by amplifying the second model, the timing of amplification can also be determined in a similar manner as above (i.e., amplification is performed after determining that the loss function value of the model meets the preset conditions). In this way, the appropriate amplification timing can be quickly determined without multiple experiments.

[0053] The model training method provided in the embodiment of the present application is introduced below with reference to specific examples.

[0054] In the actual application process, the model training method provided in the embodiment of the present application can be specifically applied to various model training devices, so that the model training device can perform model training according to the model training method provided in the embodiment of the present application during operation. In the actual application process, the model training device, the functions of the above-mentioned model training device can be implemented by a personal computer (including desktop computers, laptop computers, handheld computers and notebook computers, etc.), an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), or a smart phone, a server and other devices; or, the functions of the model training device can also be implemented by some hardware / software devices in the above-mentioned devices. The embodiment of the present application does not impose any special restrictions on the specific form of the model training device.

[0055] Specifically, when the model training method provided in the embodiment of the present application is applied to the model training process based on deep amplification, such as Figure 4 As shown, the model training method provided in the embodiment of the present application may include:

[0056] S302: The model training device constructs an initial model.

[0057] In one implementation, before constructing the initial model, the model training device may determine the number n of coding layers of the first model based on the required number L of coding layers of the target model and the number S of stages in the model training process.

[0058] For example, when the number of coding layers in the model is expanded to twice the number of original coding layers each time the model is expanded, the number of coding layers n of the first model can be determined using the following formula (1):

[0059]

[0060] For another example, when the number of coding layers in the model is increased by m coding layers each time the model is expanded, the number of coding layers n of the first model can be determined using the following formula (2):

[0061] n=L-(S-1)m (2)

[0062] Therefore, before constructing the initial model, the method may further include:

[0063] S301a, the model training device obtains the number L of coding layers of the target model and the number S of stages in the model training process.

[0064] Exemplarily, after starting the model training process, the model training device can output prompt information on the display interface, which is used to prompt the user to input the number of coding layers of the target model and the number of stages in the model training process. Then, after receiving the user's input operation, the model training device can determine the number of coding layers L and the number of stages S input by the user according to the input operation.

[0065] S301b, the model training device determines the number n of coding layers of the initial model according to the number L of coding layers and the number S of stages.

[0066] For example, the model training device can determine the number n of coding layers of the initial model according to the number L of coding layers and the number S of stages in combination with the above formula (1) or formula (2).

[0067] After the initial model is built, the method further includes:

[0068] S303: The model training device trains the initial model.

[0069] For example, the initial model can be trained with the learning rate in the learning rate baseline.

[0070] S304: The model training device monitors the loss function value of the initial model.

[0071] S305. After determining that the loss function value of the initial model meets the preset conditions, the model training device amplifies the initial model to obtain an amplified model (hereinafter referred to as the "amplified model" for ease of description).

[0072] Specifically, in the embodiments of the present application, it is taken into consideration that when determining the timing of amplification, on the one hand, if amplification is performed before the loss function value of the current model converges, it will have a certain negative impact on the accuracy of subsequent models; on the other hand, if amplification is performed when the current model has a slight convergence trend, the acceleration effect of the entire model training will be limited due to the short training time of the current model; on the other hand, if the training time of the current model is too long, there may be problems of model overfitting and the entire model consuming too much training time.

[0073] Based on the above considerations, in one implementation, the above preset conditions may include: monitoring that the change amplitude of the loss function value of the initial model in a plurality of consecutive sliding windows is less than an amplitude threshold.

[0074] That is to say, in the embodiment of the present application, the loss function value of the initial model can be monitored by means of a sliding window. And when the change amplitude of the loss function value of the initial model monitored in multiple sliding windows is less than the amplitude threshold, the initial model is amplified.

[0075] The variation range of the loss function value can be specifically reflected as a parameter obtained by calculating the slope using the loss function values ​​collected at both ends of the sliding window.

[0076] It is understandable that in actual application, the size of the sliding window and the size of the amplitude threshold can be set according to actual needs. In the embodiment of the present application, there is no limitation on the size of the sliding window and the size of the amplitude threshold.

[0077] In this way, on the one hand, by judging whether the change amplitude of the loss function value of the initial model in the sliding window is less than the amplitude threshold, it can be determined whether the current model has reached an appropriate degree of convergence; on the other hand, by judging whether the change amplitude of the loss function value in multiple consecutive sliding windows is less than the amplitude threshold, the interference of unstable factors on the judgment result can be avoided.

[0078] For example, Figure 5 As shown, S305 may specifically include:

[0079] S3051. The model training device continuously obtains the loss function value of the initial model monitored in the sliding window.

[0080] S3052: The model training device determines whether the change amplitude of the loss function value of the initial model monitored in the current sliding window is less than the amplitude threshold.

[0081] If yes, execute S3053; if no, end the monitoring process and execute S3051 again in the next monitoring cycle.

[0082] S3053. Determine whether there are multiple consecutive sliding windows in the subsequent window in which the change amplitude of the loss function value of the initial model monitored is less than the amplitude threshold.

[0083] If so, it is determined that the loss function value of the initial model meets the preset conditions, and then the initial model is amplified to obtain the amplified model; if not, the monitoring process is terminated and S3051 is re-executed in the next monitoring cycle.

[0084] In another implementation, the above-mentioned preset condition may include: within p consecutive sliding windows, the number of sliding windows in which the change amplitude of the loss function value of the initial model is monitored to be less than the amplitude threshold exceeds the number threshold q.

[0085] For example, if p=10 and q is 8, then more than 8 sliding windows are monitored within 10 consecutive sliding windows, and the change amplitude of the loss function value of the initial model in the more than 8 sliding windows is less than the amplitude threshold, then the initial model is amplified.

[0086] In the above implementation, on the one hand, by judging whether the change amplitude of the loss function value of the initial model in the sliding window is less than the amplitude threshold, it can be determined whether the current model has reached an appropriate degree of convergence; on the other hand, by judging whether the number of sliding windows in which the change amplitude of the loss function value monitored within a period of time is less than the amplitude threshold is sufficient, the interference of unstable factors on the judgment result can be avoided.

[0087] S306: The model training device trains the augmented model.

[0088] Exemplarily, if the number of coding layers in the model is expanded to twice the original number of coding layers each time the model is expanded, the number of coding layers in the expanded model is twice the original number of coding layers n (i.e., the number of coding layers n of the initial model), then the number of coding layers of the expanded model is 2n.

[0089] The parameters of the 2n coding layers in the augmented model are obtained by copying and stacking the parameters of the n coding layers in the initial model twice. For example, assuming that the n coding layers of the initial model are coding layer 1, coding layer 2, ..., coding layer n. Then, among the 2n coding layers in the augmented model, the parameters of the 1st to nth coding layers are the same as the parameters of the above coding layer 1, coding layer 2, ..., coding layer n, and the parameters of the n+1th to 2nth coding layers are the same as the parameters of the above coding layer 1, coding layer 2, ..., coding layer n.

[0090] As another example, if the number of coding layers in the model is increased by m coding layers each time the model is augmented, the number of coding layers in the augmented model is twice the original number of coding layers n (i.e., the number of coding layers n of the initial model), then the number of coding layers of the augmented model is n+m.

[0091] Among them, the parameters of the 1st to nth coding layers among the n+m coding layers of the augmented model are the same as the parameters of the 1st to nth coding layers of the initial model, and the parameters of the n+1th to n+mth coding layers are the same as the parameters of the (n+1-m)th to nth coding layers of the initial model.

[0092] Specifically, in the embodiments of the present application, it is considered that when training an augmented model, if an initial learning rate that matches the current model can be adaptively determined and the augmented model is trained using the initial learning rate, the convergence speed of the model can be improved and the model training overhead can be saved.

[0093] Based on the above considerations, in one implementation, S306 specifically includes:

[0094] S3061. The model training device obtains a first loss function value of the amplification model when the amplification model is trained using a preset learning rate.

[0095] In a possible design, the preset learning rate may be specifically a learning rate determined according to a preset correspondence between a learning rate and a number of iterations included in a learning rate baseline. Therefore, the method may further include:

[0096] In the preset corresponding relationship between the learning rate and the number of iterations, a preset learning rate corresponding to the target number of iterations is determined.

[0097] The target number of iterations is the number of model iterations that have been performed by the time the first model is amplified to obtain the second model.

[0098] For example, when the learning rate baseline includes: the corresponding initial learning rate is 0.66 for the 1st to 30th iterations; the corresponding initial learning rate is 0.33 for the 31st to 60th iterations, and when the number of iterations of the initial model is 40, the above-mentioned preset learning rate can be 0.33. Then, the model training device uses 0.33 as the initial learning rate to train the augmented model, and after several iterations (the number of iterations can be determined according to actual needs), obtains the loss function value of the augmented model (i.e., the first loss function value).

[0099] S3062. The model training device determines a first learning rate corresponding to a first loss function value according to a correspondence between a learning rate and a loss function value in a process of training an initial model.

[0100] For example, the model training device can record the changes in the learning rate and loss function value of the model during the process of training the initial model through the content of S303. Then, when executing S3062, the model training device finds the first learning rate corresponding to the first loss function according to the corresponding relationship between the learning rate and the loss function value recorded above.

[0101] In a possible design, considering that during the change of the learning rate and the loss function value recorded by the model training device in the process of training the initial model, there may be no loss function value that is completely consistent with the first loss function. In this case, the two loss function values ​​closest to the first loss function value can be taken from the recorded loss function values, and then the learning rate corresponding to the first loss function value can be obtained using an interpolation calculation method or other fitting method as the first learning rate.

[0102] S3063. The model training device uses the first learning rate as the initial learning rate to train the augmented model.

[0103] Afterwards, the augmented model can be trained and augmented in the same manner as S304-S305, and the initial learning rate of the model after the augmentation can be determined in the same manner as S306. Next, the contents of S304-S306 can be repeatedly executed until a target model that meets the requirements is obtained.

[0104] Above Figure 4 The example shown mainly introduces the implementation process of the model training method provided by the embodiment of the present application in the scenario of depth amplification. In the actual application process, the embodiment of the present application can also be applied to the scenarios of width amplification and data amplification.

[0105] It should be noted that the so-called deep expansion in the embodiments of the present application can be understood as expanding the model by increasing the model depth in stages (for example, increasing the model encoding layer) during the model training process of gradually expanding the model in stages. Figure 1 or Figure 2 The example shown, and Figure 4 The model training method shown is to amplify the model in a deep amplification manner.

[0106] Width expansion can be understood as the process of gradually expanding the model in stages. First, the initial model with fewer channels (or smaller model weights) is constructed to reduce the number of parameters of the initial model. Then, the model is expanded by increasing the width of the model (i.e., the number of channels of the model, or the weight of the model) in stages. For example Figure 6 As shown, assuming that the weight shape (or weight scale) of the initial model is k / 2×k / 2, then by width expansion, the weight of the model can be expanded to k×k.

[0107] Data augmentation can be understood as the process of using low-resolution data to train the model first, and then increasing the resolution of the training data in stages to amplify the model parameters. Figure 7 As shown in the figure, assuming that the size of the data used by the initial model is w×factor in width and h×factor in height. Then after data augmentation, the size of the data used by the training model is w in width and h in height. Among them, factor is a value less than 1.

[0108] The following introduces the model training method provided by the embodiment of the present application in combination with examples and the model training process based on width expansion. Figure 8 As shown, the method may include:

[0109] S401: The model training device determines the weight of the initial model.

[0110] For example, the model training device may multiply the required weight of the target model by a factor less than 1 to obtain the weight of the initial model.

[0111] S402: The model training device constructs an initial model.

[0112] Specifically, after determining the weight of the initial model through S401, the model training device can construct a small model generated based on the width level of the target model according to the weight.

[0113] S403: The model training device trains the initial model.

[0114] For example, the initial model can be trained with the learning rate in the learning rate baseline.

[0115] S404: The model training device monitors the loss function value of the initial model.

[0116] The implementation process of S404 may refer to S304 above, and repeated contents will not be repeated here.

[0117] S405. After determining that the loss function value of the initial model meets the preset conditions, the model training device expands the width of the initial model to obtain an expanded model (hereinafter referred to as an "expanded model" for ease of description).

[0118] Among them, the specific process of the model training device in S405 determining that the loss function value of the initial model meets the preset conditions can be referred to S305 above, and the repeated content will not be repeated here.

[0119] S406: The model training device trains the augmented model.

[0120] The implementation process of S406 can refer to S306 above, and the repeated contents will not be repeated here. Figure 4 The implementation process shown is similar, and then the contents of S404-S406 can be repeatedly executed until a target model that meets the requirements is obtained.

[0121] The following introduces the model training method provided by the embodiment of the present application in combination with examples and the model training process based on data augmentation. Fig. 9 As shown, the method may include:

[0122] S501. The model training device performs low-resolution sampling on original training data.

[0123] For example, the model training device performs low-resolution sampling on the w×h training image in the original training data to obtain a (w×factor)×(h×factor) training image.

[0124] S502: The model training device constructs an initial model.

[0125] Among them, the process of constructing the initial model by the model training device can refer to the relevant technical content and will not be elaborated here.

[0126] S503: The model training device trains the initial model.

[0127] For example, the initial model can be trained with the learning rate in the learning rate baseline.

[0128] S504: The model training device monitors the loss function value of the initial model.

[0129] The implementation process of S504 may refer to S304 above, and repeated contents will not be repeated here.

[0130] S505. After determining that the loss function value of the initial model meets the preset conditions, the model training device performs data amplification on the initial model to obtain an amplified model (hereinafter referred to as the "amplified model" for ease of description).

[0131] Among them, the specific process of the model training device in S505 determining whether the loss function value of the initial model meets the preset conditions can be referred to S305 above, and the repeated content will not be repeated here.

[0132] S506: The model training device trains the augmented model.

[0133] The implementation process of S506 can refer to S306 above, and the repeated contents will not be repeated here. Figure 4 The implementation process shown is similar, and then the contents of S504-S506 can be repeatedly executed until a target model that meets the requirements is obtained.

[0134] Combined with the above Figure 4-Figure 9 , the model training method provided according to this embodiment is described in detail, and various devices corresponding to the model training method provided by this embodiment will be described below.

[0135] like Fig.10 As shown, it is a schematic diagram of the structure of a model training device provided in this embodiment. The model training device 60 can be a software / hardware device for model training. Specifically, the model training device 60 can include a desktop computer, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, and a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) device, and various artificial intelligence (artificial intelligence, AI) hardware accelerators and other devices that can perform model training, or the model training device 60 can also be software running on the above-mentioned devices.

[0136] The model training device 60 can be used to perform the above Figure 4-Figure 9 All or part of the steps performed by the model training device. Specifically, the model training device 60 includes:

[0137] The training unit 601 is used to train the first model.

[0138] A monitoring unit 602, configured to monitor the loss function value of the first model;

[0139] an amplification unit 603, configured to amplify the first model to obtain a second model after determining that the loss function value of the first model satisfies a preset condition; wherein the amplified parameters in the second model are obtained by copying the parameters in the first model;

[0140] The training unit 601 is also used to train the second model.

[0141] In one implementation, the training unit 601 is further used to train the second model, including:

[0142] A training unit 601 is specifically used to obtain a first loss function value of the second model when the second model is trained using a preset learning rate;

[0143] The training unit 601 is specifically configured to determine a first learning rate corresponding to the first loss function value according to a correspondence between a learning rate and a loss function value in a process of training the first model;

[0144] The training unit 601 is specifically configured to use the first learning rate as an initial learning rate to train the second model.

[0145] In one implementation, the training unit 601 is also used to determine the preset learning rate corresponding to the target number of iterations in the preset correspondence between the learning rate and the number of iterations; the target number of iterations is the number of model iterations that have been performed by the time the first model is amplified to obtain the second model.

[0146] In one implementation, the preset condition includes: in a plurality of consecutive sliding windows, the change amplitude of the loss function value of the first model is monitored to be less than the amplitude threshold. Alternatively, the preset condition includes: in p consecutive sliding windows, the number of sliding windows in which the change amplitude of the loss function value of the first model is monitored to be less than the amplitude threshold exceeds the number threshold q.

[0147] In one implementation, the first model includes n coding layers constructed in a stacking manner, where n is a positive integer.

[0148] The amplification unit 603 is used to amplify the first model after determining that the loss function value of the first model meets a preset condition, so as to obtain a second model including:

[0149] The amplification unit 603 is used to add m coding layers to the end of the n coding layers included in the first model close to the decoding end after determining that the loss function value of the first model meets the preset conditions, so as to obtain the second model; the parameters in the m coding layers are the parameters in the m coding layers close to the decoding end among the n coding layers, copied in a direction from shallow to deep, and m is a positive integer less than or equal to n.

[0150] In one implementation, the training unit 601 is further configured to obtain the number of coding layers L of the target model and the number of stages S in the model training process. The training unit is further configured to determine the number of coding layers n of the first model according to the number of coding layers L and the number of stages S.

[0151] In one implementation, the amplification unit 603 is used to amplify the first model to obtain the second model after determining that the loss function value of the first model meets a preset condition, including:

[0152] The expansion unit 603 is used to expand the width of the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition.

[0153] In one implementation, the amplification unit 603 is used to amplify the first model to obtain the second model after determining that the loss function value of the first model meets a preset condition, including:

[0154] The amplification unit 603 is used to perform data amplification on the first model to obtain a second model after determining that the loss function value of the first model meets a preset condition.

[0155] For a more detailed description of the training unit 601, the monitoring unit 602 and the amplification unit 603, please refer to Figure 4-Figure 9 The relevant description in the method shown will not be repeated here.

[0156] Fig.11 A schematic diagram of the structure of another model training device provided in this embodiment. The model training device 70 can be a chip or a system on a chip. Specifically, the model training device 70 can include all or part of the hardware in a device that can perform image processing, such as a desktop computer, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a super mobile personal computer, a netbook, a cellular phone, a personal digital assistant, an augmented reality\virtual reality device, and various artificial intelligence hardware accelerators.

[0157] The model training device 70 may include: a processor 701 , a communication line 702 , a memory 703 , and part or all of at least one communication interface 704 .

[0158] The processor 701 is used to execute the Figure 4-Figure 9 All or part of the steps performed by the model training device in the provided method.

[0159] Specifically, the processor 701 may include a general-purpose central processing unit (CPU), and the processor 701 may also include a microprocessor, a field programmable gate array (Field Programmable Gate Array, FPGA), a digital signal processor (digital signal processing, DSP) or an application-specific integrated circuit (application-specific integrated circuit, ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0160] In a specific implementation, as an embodiment, the processor 701 may include one or more CPUs, such as Fig.11 CPU0 and CPU1 in.

[0161] In a specific implementation, as an embodiment, the model training device 70 may include multiple processors, such as Fig.11 701 and processor 708 in the embodiment of the present invention. Each of these processors can be a single-CPU processor or a multi-CPU processor. The processor here can refer to one or more devices, circuits, and / or processing cores for processing, for example, computer data (computer program instructions).

[0162] In addition, the memory 703 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM). Memory 703 can be independent and connected to processor 701 via communication line 702. Memory 703 can also be integrated with processor 701.

[0163] The memory 703 stores computer instructions. The processor 701 can execute all or part of the steps in the model training method provided in this embodiment by executing the computer instructions stored in the memory 703. Fig.11 As shown, the computer instructions stored in the memory 703 may include software modules for implementing the functions of each functional unit in the above-mentioned model training device 60. The processor 701 can be used to execute the model training method provided in this embodiment by executing the computer instructions stored in the memory 703.

[0164] Optionally, the computer-executable instructions in this embodiment may also be referred to as application program codes, which is not specifically limited in this embodiment.

[0165] In addition, the communication interface 704 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0166] In addition, the communication line 702 is used to connect the various components in the model training device 70. Specifically, the communication line 702 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are marked as communication lines 702 in the figure.

[0167] In a specific implementation, as an embodiment, the data processing device 70 may further include an output device 707 and an input device 706. The output device 707 may communicate with the processor 701 and may display information in a variety of ways. For example, the output device 707 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 706 may communicate with the processor 701 and may receive user input in a variety of ways. For example, the input device 706 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0168] In addition, the model training device 70 may also include a storage medium 705. The storage medium 705 is used to store computer instructions and various data for implementing the technical solution of this embodiment. When the model training device 70 executes the above-mentioned model training method of this embodiment, the computer instructions and various data stored in the storage medium 705 are loaded into the memory 703, so that the processor 701 can execute the computer instructions stored in the memory 703 to execute the model training method provided by this embodiment.

[0169] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a model training device. Of course, the processor and the storage medium can also be present in the model training device as discrete components.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in this embodiment is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a communication device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as an SSD.

[0171] In this embodiment, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different implementations are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.

[0172] In this embodiment, "at least one" means one or more, "more than one" means two or more, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, for elements (element) in the singular form "a", "an" and "the", unless the context clearly stipulates otherwise, it does not mean "one or only one", but means "one or more than one". For example, "a device" means one or more such devices. Furthermore, at least one (at least one of)... "means one or any combination of subsequent associated objects, for example, "at least one of A, B and C" includes A, B, C, AB, AC, BC, or ABC. In the text description of this embodiment, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of this embodiment, the character " / " indicates that the previous and next associated objects are in a "division" relationship.

Claims

1. A model training method, characterized in that: include: Train the first model; Monitoring the loss function value of the first model; After determining that the loss function value of the first model meets a preset condition, the first model is amplified to obtain a second model; wherein the amplified parameters in the second model are obtained by copying the parameters in the first model; The second model is trained.

2. The method according to claim 1, characterized in that The training of the second model comprises: Obtaining a first loss function value of the second model when the second model is trained using a preset learning rate; Determining a first learning rate corresponding to the first loss function value according to a correspondence between a learning rate and a loss function value in a process of training the first model; The second model is trained by using the first learning rate as an initial learning rate.

3. The method according to claim 2, characterized in that The method further comprises: In the preset correspondence between the learning rate and the number of iterations, the preset learning rate corresponding to the target number of iterations is determined; the target number of iterations is the number of model iterations that have been performed by the time the first model is amplified to obtain the second model.

4. The method according to any one of claims 1 to 3, characterized in that: The preset condition includes: monitoring that the change amplitude of the loss function value of the first model in a plurality of consecutive sliding windows is less than an amplitude threshold; Alternatively, the preset condition includes: within p consecutive sliding windows, the number of sliding windows in which the change amplitude of the loss function value of the first model is monitored to be less than the amplitude threshold exceeds the number threshold q.

5. The method according to any one of claims 1 to 4, characterized in that: The first model includes n coding layers constructed in a stacking manner, where n is a positive integer; After determining that the loss function value of the first model meets a preset condition, amplifying the first model to obtain a second model includes: After determining that the loss function value of the first model meets the preset conditions, m coding layers are added to the end of the n coding layers included in the first model close to the decoding end to obtain the second model; the parameters in the m coding layers are the parameters in the m coding layers close to the decoding end among the n coding layers, and are copied in a direction from shallow to deep, and m is a positive integer less than or equal to n.

6. The method according to claim 5, characterized in that The method further comprises: Get the number of encoding layers L of the target model and the number of stages S in the model training process; According to the number of coding layers L and the number of stages S, the number of coding layers n of the first model is determined.

7. The method according to any one of claims 1 to 4, characterized in that: After determining that the loss function value of the first model meets a preset condition, amplifying the first model to obtain a second model includes: After determining that the loss function value of the first model meets a preset condition, the width of the first model is expanded to obtain a second model.

8. The method according to any one of claims 1 to 4, characterized in that: After determining that the loss function value of the first model meets a preset condition, amplifying the first model to obtain a second model includes: After determining that the loss function value of the first model meets a preset condition, data augmentation is performed on the first model to obtain a second model.

9. A model training device, characterized in that: include: A training unit, used for training a first model; A monitoring unit, used to monitor the loss function value of the first model; an amplification unit, configured to amplify the first model to obtain a second model after determining that the loss function value of the first model satisfies a preset condition; wherein the amplified parameters in the second model are obtained by copying the parameters in the first model; The training unit is also used to train the second model.

10. A model training device, characterized in that: include: The method comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises instructions for implementing the method according to any one of claims 1 to 8 when the instructions are executed on a processor.

Images