A Calculation Method of Adaptive Learning Rate

Abstract

The present invention relates to an adaptive learning rate calculation method, comprising the following steps: step S1: based on the initial batch number and the combination of initial optimal network parameters, obtain the number of secondary batches; step S2: based on the number of secondary batches, obtain The current optimal network parameter combination; Step S3: Based on the current optimal network parameter combination, the current loss function and the amount of training data, obtain the current batch number; Step S4: Based on the current batch number, obtain the current learning rate; step S5: the current batch number replaces the secondary batch number, and repeats steps S2-step S5 until the training model weights converge. Compared with the existing technology, it avoids generating low-quality training model weights, ensures that the generated training model weights are optimal, increases chess power when playing Go, and reduces training time for training model weights and superpower consumption of GPU and other hardware.

Description

technical field [0001] The invention relates to the field of deep learning, in particular to an adaptive learning rate calculation method. Background technique [0002] Artificial Intelligence (AI) has been vigorously developed along with many applications in human reality scenarios, and the progress of artificial intelligence in Go has also achieved great results. The weight of the training model is the key basis for the performance of the Go AI in the game, and it is the chess information generated by the Go AI through the residual network training. [0003] Learning rate is a classic hyperparameter of neural network, and it is also one of the problems that plague neural network training, because the parameters cannot be learned by conventional methods. The learning rate of training model weight training in the residual network of many Go AIs today is a fixed value, and the network training cannot be based on the environmental needs of Go AI (weight initialization, networ...

AI Technical Summary

Problems solved by technology

[0004] Generally, in the initial stage of network training, it is often better to set a larger learning rate, because the weight of the training model is far away from the optimal value, and a larger learning rate can quickly approach the optimal value; and in the later stage of training, because it is already close to the optimal value value, at this time, it is better to use a smaller learning rate, and a larger learning rate will easily lead to fluctuations around the real optimal value, that is, it cannot reach the optimal value

The inferior training model weights generated by the training data will cause Go AI to have many blind spots when playing the game, and its chess strength will be greatly reduced, and the non-adaptive learning rate will consume a lot of training time for the training model weights, and cause GPU and other hardware. super energy loss

Images