An Adaptive Activation Function Parameter Adjustment Method for Deep Neural Networks

Abstract

A kind of adaptive activation function parameter adjustment method for deep neural network, described method comprises the following steps: step 1, at first carry out mathematical definition to adaptive activation function parameter adjustment method; Step 2, carry out adaptive activation function based on MNIST data set and other classic activation functions to compare and analyze the experimental results. The network used has three hidden layers, each hidden layer has 50 neurons, and the gradient descent algorithm is used to iterate for 100 cycles. The learning rate is set to 0.01, and the minimum batch The number is 100; step 3, after the optimal activation function version is obtained in step 2, it is applied to the detection of specific bladder cancer cells. In the process of continuous training of the network, the present invention finds the optimal activation function suitable for the network by continuously adjusting its own shape, improves the performance of the network, reduces the overall number of learnable parameters of the adaptive activation function in the network, and accelerates the learning rate of the network , to improve the generalization of the network.

Description

technical field [0001] The invention belongs to the field of adaptive activation functions, and designs an adaptive activation function parameter adjustment method oriented to a deep neural network. Specifically, the adaptive activation function controls its own shape by adding learnable parameters. At the same time, these learning parameters can be updated with the progress of network training through the back propagation algorithm, reducing the overall number of learnable parameters of the adaptive activation function in the network. . Background technique [0002] Nowadays, machine learning is widely used in social life, while traditional machine learning mostly uses shallow structures, such as Gaussian Mixture Model (GMM), Conditional Random Field (CRF), Support Vector Machine (SVM), etc. The expressive ability of the function is limited, and the extraction of the original input signal features is relatively elementary. Its generalization ability for complex classificat...

AI Technical Summary

Problems solved by technology

If the neurons in a neural network are only linear operations, then the network can only express a simple linear map, even if the depth and width of the network are increased, it is still a linear map, and it is difficult to effectively model the nonlinearly distributed data in the actual environment.

Images