Unlock instant, AI-driven research and patent intelligence for your innovation.

DNN layer deep association learning rate dynamic learning method

A dynamic learning and learning rate technology, applied in neural learning methods, biological neural network models, etc., can solve problems such as different requirements for parameter convergence amplitudes

Inactive Publication Date: 2020-05-05
SHANDONG SYNTHESIS ELECTRONICS TECH
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a dynamic learning method of deep association learning rate of DNN, which overcomes the different requirements of different network layers for the convergence range of parameters, and the problem of self-adaptive dynamic adjustment of learning rate of each network layer

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • DNN layer deep association learning rate dynamic learning method
  • DNN layer deep association learning rate dynamic learning method
  • DNN layer deep association learning rate dynamic learning method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0022] like figure 1 As shown, the traditional learning parameter setting and the process of affecting network parameter adjustment are: 1. Directly obtain the global learning rate (LR) parameter value of this iteration according to the LR learning rate curve; 2. When adjusting parameters during the training process, According to the learning rate parameter value corresponding to the current round of training, weighted adjustments are made to the change amplitude of each network layer parameter.

[0023] It can be seen from the above that the traditional learning rate variable is globally unified in the network, and changes according to the defined curve related to the training cycle (considering that the more training times, the smaller the amplitude of each parameter needs to be adjusted, so the learning rate is generally Decrease step by step), such as LR(N)=b*exp(-a*N), where a, b are constants, and N is the number of training iterations. In addition to the global unified...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a DNN layer deep association learning rate dynamic learning method. According to the method, a learning parameter LLR for only adjusting the learning rate of the current layeris added to each of all network layers with parameters of the DNN, the initial value of the LLR is manually set, the LLR numerical value is dynamically adjusted in the training process based on an existing loss function, and the LLR numerical value of each layer follows a curve impact adjustment method in the adjustment process, that is, when learning parameters are adjusted, the adjustment curveof the learning parameters oscillates along with the number of training times, and the oscillation interval is gradually decreased along with the increase of the number of training times. According tothe invention, the problems of different convergence amplitude requirements of different network layers on parameters and self-adaptive dynamic adjustment of the learning rate of each network layer can be solved.

Description

technical field [0001] The invention relates to a dynamic learning method of deep association learning rate of DNN, which belongs to the technical field of image processing and artificial intelligence. Background technique [0002] In deep learning systems based on convolutional networks, the learning rate is a hyperparameter that guides how we adjust the weights of the network through the gradient of the loss function. The higher the learning rate, the faster the parameter change rate caused by the loss function, but as the training progresses, objectively, the range of parameters that need to be adjusted will gradually become smaller. A high learning rate will inevitably bring about parameter convergence oscillations, that is, it cannot The minimum value of the loss function. The lower the learning rate, the slower the loss function will cause the parameters to change. A low learning rate ensures that we don't miss any local minima, but it also means that we will take lon...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/08
CPCG06N3/08
Inventor 朱锦雷井焜孙涛张传锋
Owner SHANDONG SYNTHESIS ELECTRONICS TECH