A learnable activation function method based on orthogonal Legendre wavelet

Abstract

The invention discloses a learnable activation function method based on orthogonal Legendre wavelet, and relates to the field of artificial intelligence technology. The steps of the invention are as follows: S1: for specific task data, divide the data set into a training set and a test set; set the maximum calculation amount MaxCal; S2: select a network model, initialize network parameters, activation function structure parameters K, M, and numerical parameter C; S3: generate Legendre wavelet according to parameters K, M, and C, and replace the activation function in the network; S4: determine the structure parameter K; S5: use the training set to train the model, determine the structure parameter M; S6: use the training set to train the model again, determine the numerical parameter C, and obtain the optimal activation function and network weight W. The invention can realize the automatic determination of the structure and parameters of the activation function through the improvement of the method, thereby further avoiding the gradient disappearance and neuron death phenomenon, retaining the nonlinearity and differentiability of the function, and improving the network accuracy and convergence effect.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence technology, and in particular relates to a learnable activation function method based on orthogonal Legendre wavelets. Background Art

[0002] Artificial Neural Networks (ANNs), also referred to as Neural Networks (NNs) or Connection Model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system to adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. The activation function is a crucial component of the neural network. It passes the information of one layer of the network to the next layer through some transformation, providing the overall model with nonlinear mapping capabilities, enabling the neural network to complete complex tasks such as image classification, target detection, fault diagnosis, etc. Traditional activation functions include sigmoid, tanh, etc. The sigmoid activation function compresses the continuous real value of the input to the interval [0,1]. This activation function was frequently used in early neural networks. The tanh activation function compresses the continuous real value of the input to the interval [-1,1]. Since the output is 0 mean, its convergence speed is faster than sigmoid, which is more convenient for the learning of the next layer of the network.

[0003] However, with the widespread application of convolution operations and deep neural networks, the number of neural network layers continues to increase. The blindness in the design of activation functions will cause the gradient to vanish in deep networks, making the network convergence weak, thereby greatly reducing the final accuracy of the network. The reason for this phenomenon is that the differential values ​​of "S"-shaped activation functions such as sigmoid and tanh are accumulated, causing the gradient values ​​of a large number of neurons to fall near 0, and the network cannot be further updated. These functions are derived from mathematics and statistics, and the variability of artificial intelligence tasks cannot be taken into account when they are designed and used. The activation functions designed based on linear rectifier units such as ReLU, LeakyReLU, ELU, Swish, etc. generated to address this problem will input two segments of the classification, and each segment is usually fitted with a linear function. However, these functions only alleviate the gradient vanishing phenomenon to a certain extent, and the side effect is that they cause serious neuron death, causing nearly half of the neurons to be inactivated prematurely and no longer updated, thereby limiting the overall accuracy of the neural network.

[0004] In summary, existing activation technologies require an adaptive task design method to automatically determine the structure and parameters of the activation function, so as to further avoid gradient vanishing and neuron death, retain the nonlinearity and differentiability of the function, and improve the network accuracy and convergence effect. Summary of the invention

[0005] The object of the present invention is to provide a learnable activation function method based on orthogonal Legendre wavelet to solve the problems raised in the above background technology.

[0006] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] The present invention is a learnable activation function method based on orthogonal Legendre wavelet, and the steps are as follows:

[0008] S1: For specific task data, divide the data set into training set and test set; set the maximum calculation amount MaxCal;

[0009] S2: Select the network model, initialize the network parameters, activation function structure parameters K, M, and numerical parameter C;

[0010] S3: Generate Legendre wavelet according to parameters K, M, C, and replace the activation function in the network;

[0011] S4: Determine the structural parameter K;

[0012] S5: Use the training set to train the model and determine the structural parameter M;

[0013] S6: Use the training set to train the model again, determine the numerical parameter C, and obtain the optimal activation function and network weight W;

[0014] S7: Substitute K, M, and C into Legendre wavelet to obtain the optimal activation function;

[0015] S8: Substitute the optimal activation function into the network model and test it using the test set;

[0016] S9: Compare the test results with the activation functions designed by mainstream methods.

[0017] Furthermore, a data set is selected and divided into a training set and a test set according to a certain ratio; at the same time, a maximum computation amount MaxCal is set to limit the computation amount not to be exceeded during the network back propagation process.

[0018] Furthermore, a deep network model is selected, and the parameters of each layer of the network are randomly initialized without loading any pre-trained weights; the initialization parameters K=M=2 are all used as Legendre wavelet structure parameters.

[0019] Furthermore, the Legendre wavelet is generated as follows:

[0020]

[0021] in, m is the order of Legendre polynomial, which takes the value of m=2,3,…,M, and t is the activation function input; L m (t) represents the m-th order Legendre polynomial, which is obtained by the following recursive formula:

[0022]

[0023] Furthermore, the interval [0,1] is divided into 2 k-1 activation interval, expressed as:

[0024] The feature matrix t extracted by the feature extractor is subjected to maximum and minimum normalization processing to obtain a new matrix t'.

[0025] Furthermore, each activation interval corresponds to M Legendre wavelets and weights C, K and M are hyperparameters; C is 2 k-1 M learnable parameters are updated along with the network using gradient descent. Substitute the Legendre wavelet of the corresponding interval and take the weighted sum of the corresponding interval to get the final output of the activation function.

[0026] Furthermore, the K, M, and C values ​​determined in S4, S5, and S6 are used, and the activation function is constructed according to the Legendre wavelet generation method in S3.

[0027] Furthermore, the optimal activation function in S7 is used to replace the original activation function in the network, and the network is tested on the test set to observe the overall progress and convergence of the network.

[0028] Furthermore, the activation function in the established network is replaced with mainstream activation functions such as ReLU, tanh, sigmoid, swish, etc., the entire network is retrained, and then the same accuracy and convergence tests are performed on the test set. The results are compared with those obtained in S7 to determine the practicality of the Legendre wavelet activation function under this model and dataset.

[0029] The present invention has the following beneficial effects:

[0030] The present invention can automatically determine the structure and parameters of the activation function through the improvement of the method, thereby further avoiding the gradient disappearance and neuron death phenomena, retaining the nonlinearity and differentiability of the function, and improving the network accuracy and convergence effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0032] Figure 1 It is a schematic diagram of the activation function method based on orthogonal Legendre wavelet of the present invention;

[0033] Figure 2 It is a network topology diagram of the Legendre wavelet activation function of the present invention;

[0034] Figure 3 It is a schematic diagram of the Legendre wavelet activation function image of the present invention;

[0035] Figure 4 This is a comparison chart of the convergence of the ResNet network of the present invention in the test set;

[0036] Figure 5 This is a comparison chart of the convergence of the Swin-Transformer of the present invention on the test set. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1-Figure 5 As shown, the present invention is a learnable activation function method based on orthogonal Legendre wavelet, and the steps are as follows:

[0039] S1: For the CIFAR-100 data of a specific task, divide the data set into a training set and a test set (random division, the ratio is 4:1). Select a data set, such as the image classification CIFAR-100 data set, the CIFAR-10 data set, the target detection CoCo data set, the face recognition data set, etc.; divide the data set into a training set and a test set according to a certain ratio; set the maximum calculation amount MaxCal = 6Gmac, and limit the network back propagation process to not exceed this amount of calculation. This amount of calculation takes about 6 hours to train on the case experimental conditions RTX3080 (GPU utilization rate 90%).

[0040] S2: Select the network model ResNet50, select a deep network model, such as VGG, ResNet, Transformer, etc.; randomly initialize the parameters of each layer of the network, without loading any pre-trained weights; initialize the parameters K = M = 2, all as Legendre wavelet structure parameters. C matrix is ​​initialized to all 1, as the Legendre wavelet numerical parameter size is 6x1.

[0041] S3: Generate Legendre wavelet according to parameters K, M, C, and replace the activation function in the network. The generated Legendre wavelet is mathematically expressed as:

[0042]

[0043] Specifically: Legendre wavelet is generated in the following way:

[0044]

[0045] in m is the order of Legendre polynomial, which takes the value of m=2,3,…,M, and t is the activation function input. m (t) represents the m-th order Legendre polynomial, which is obtained by the following recursive formula:

[0046]

[0047] Since the Legendre polynomials are mutually orthogonal, by integration, we can find that the Legendre wavelets with different n and m values ​​are also orthogonal. The network topology diagram of the Legendre wavelet activation function is shown below ( Figure 2 ):

[0048] The processing flow is as follows:

[0049] First, divide the interval [0,1] into 2 equal parts k-1 activation interval, expressed as:

[0050]

[0051] Secondly, the feature matrix t extracted by the feature extractor is normalized to obtain a new matrix t', in which each element has a value in [0,1], and is copied 2 k-1 The same t' maintains a one-to-one correspondence with the activation interval.

[0052] Then, the t' corresponding to each activation interval is processed. If the element in the matrix t' is not in the corresponding interval, the element is set to 0, and if it is in the corresponding interval, the original value is retained unchanged. k-1 different matrices, denoted as They represent feature inputs at different scales.

[0053] Finally, each activation interval corresponds to M Legendre wavelets and weights C (wavelet activation function numerical parameters). K and M are hyperparameters and remain unchanged during training; C is 2 k-1 M learnable parameters are updated along with the network using gradient descent. Substitute the Legendre wavelet of the corresponding interval and take the weighted sum of the corresponding interval to get the final output of the activation function

[0054] S4: Determine the structural parameter K to make the overall computational complexity of the network model moderate. Figure 1 The process described in the previous experiment was used to obtain the maximum K value of 3, which does not exceed the calculation amount MaxCal. At this time, the calculation amount is 4.16Gmac;

[0055] Specifically, this parameter can be determined by the total back-propagation computation of the network, which determines the number of segments the input is divided into in the design of the Legendre wavelet activation function. When the structural parameter is K0, the activation function input is divided into 2K0-1 parts. Using the neural network computation toolkit, the maximum K value can be used when the total computation of the network is limited to not exceed the maximum computation limit MaxCal.

[0056] S5: Use the training set to train the model and determine the structural parameter M to maximize the overall accuracy of the model. Figure 1 The experiment of the overview process shows that the maximum M value that makes the network most accurate in the training set is 3. The computational complexity is 4.19 Gmac.

[0057] Specifically, the value of M is increased from the integer 2, and theoretically the accuracy of the training set increases accordingly. When the accuracy of the training set remains the same or begins to decrease, the function approximation limit is reached, and the increase is stopped and the M parameter value at this time is recorded. The structural parameter M determines which polynomials of which order participate in the activation function in the Legendre wavelet activation design. When the value of M is 1, only zero-order and first-order functions participate in the activation function; when the value of M is 2, zero-order, first-order, and quadratic functions will participate in the activation function, and so on. The larger the value of M, the stronger the approximation ability of the designed activation function will provide to the network, but the overall back-propagation calculation amount of the network will also increase dramatically.

[0058] S6: Use the training set to train the model again, determine the numerical parameter C, and obtain the optimal activation function and network weight W. The parameter C obtained at this time is:

[0059]

[0060] Specifically, when K and M are determined, S3 is used to obtain the activation function output, which is passed to the subsequent neurons to calculate the overall error of the network on the training set. The error is partial derivatived with respect to the numerical parameter matrix C and updated using the gradient descent method. In this way, the K, M, and C parameter values ​​under the data can be obtained;

[0061] S7: Substitute K, M, and C into the Legendre wavelet to obtain the optimal activation function. Substitute the parameters K, M, and C determined in S4, S5, and S6 into the Legendre wavelet to generate the optimal activation function image as shown below: Figure 3 ;

[0062] The analytical expression of the function in two-dimensional space is:

[0063]

[0064] S8: Substitute the optimal activation function into the network model and test it with the test set. Without using pre-trained weights, the overall image classification accuracy of the network is 49.32±0.40. The network converges at about 200 epochs. Use the optimal activation function in S7 to replace the original activation function in the network, and test the network on the test set to observe the overall progress and convergence of the network.

[0065] S9: Compare the test results with the activation functions designed by mainstream methods. In order to compare the changes in convergence and accuracy of the activation functions designed by the design methods. Swin-Transformer conducts the same experiment on CIFAR-100, replacing different activation functions in each experiment, but all need to do the same steps from S1 to S9, proving its practicality in the CNN and Transformer fields;

[0066] Only the activation function in the established network is replaced with mainstream activation functions such as ReLU, tanh, sigmoid, swish, etc., the entire network is retrained, and then the same accuracy and convergence tests are performed on the test set. The results are compared with the results obtained in S7 to determine the practicality of the Legendre wavelet activation function under this model and dataset.

[0067] (I) From the perspective of convergence:

[0068] ① Compare the loss of the designed Legendre wavelet activation function with sigmoid, tanh, ReLU, LeakyReLU, and Swish activation functions on CIFAR-100 (the model uses Resnet50), such as Figure 4 As shown;

[0069] It can be seen that the Sigmoid function converges the slowest, followed by tanh, Swish, ReLU, and LeakyReLU. Compared with them, Legendre23 and Legendre33 functions converge the fastest. Legendre23 represents the case where the Legendre wavelet activation function parameter K is 2 and the parameter M is 3; Legendre33 represents the case where the parameters K and M are both 3.

[0070] ② Compare the loss of the designed Legendre wavelet activation function with sigmoid, tanh, ReLU, LeakyReLU, and Swish activation functions on CIFAR-100 (the model uses Swin-Transformer), such as Figure 5 As shown;

[0071] From the convergence of the functions in the test set, Legendre23 and Legendre33 functions converged at about 200 epochs.

[0072] The convergence speed of other functions is the fastest at about 285 epochs, and the convergence speed is slower.

[0073] (II) In terms of accuracy and parameter quantity:

[0074] We performed five independent repeated experiments on each activation function, and took the average classification accuracy of the five experiments on the test set as the activation function performance indicator. At the same time, we calculated the total number of model parameters when using ResNet and Swin-Transformer. The case implementation results are shown in Table 1.

[0075] Table 1 Comparison of different activation functions

[0076]

[0077] From the perspective of classification accuracy, the Legendre wavelet activation function designed in this case can achieve the best performance in the CIFAR100 classification task. Without using pre-trained weights, the ResNet50 network combined with Legendre23 is 2.08 percentage points higher than the ResNet50 network combined with ReLU, and the ResNet50 network combined with Legendre33 is 2.40 percentage points higher than the ResNet50 network combined with ReLU. The Swin-Transformer combined with Legendre23 is 4.85 percentage points higher than the Swin-Transformer combined with ReLU, and the Swin-Transformer combined with Legendre33 is only 3.51 percentage points higher than the Swin-Transformer combined with ReLU. Such results show that the Legendre wavelet activation function is the closest to the optimal activation function.

[0078] From the perspective of parameter quantity, the models combining various activation functions have little difference in the overall parameter quantity. During the forward propagation process, each layer of Legendre23 will introduce 6 learnable parameters, and each layer of Legendre33 will introduce 12 learnable parameters, but these are less than the overall parameter quantity.

[0079] In terms of computational complexity, the Legendre wavelet activation function increases the computational complexity of the two types of models, which will affect the efficiency of back propagation. In ResNet50, the Legendre23 activation function will increase the computational complexity by 0.07Gmac, and the Legendre33 activation function will increase the computational complexity by 0.10Gmac. In Swin-Transformer, using the Legendre23 activation function will increase the computational complexity by 0.83Gmac, accounting for 19.08% of the original computational complexity; using the Legendre33 activation function will increase the computational complexity by 1.45Gmac, accounting for 33.3% of the original computational complexity.

[0080] In summary, this case shows that the learnable activation function designed based on orthogonal Legendre wavelets has the characteristics of high precision, fast convergence, and adaptability, and is more suitable for learning and predicting high-dimensional data. Compared with the existing methods for designing activation functions, the Legendre wavelet activation function slightly increases the amount of network calculations, but significantly improves the classification accuracy and approximation ability of the network. It avoids network non-convergence caused by gradient vanishing and dead neurons, and is a new method and tool for deep learning and neural network activation function design.

[0081] Implemented:

[0082] 1. Orthogonal Legendre wavelet generation method. Technical solution S3 describes how to generate Legendre polynomials and mutually orthogonal Legendre wavelets based on K and M values. The orthogonality of Legendre wavelets provides support and support for the network approximation capability.

[0083] 2. Activation function segmentation processing flow. Technical solution S3 explains how the activation function based on Legendre wavelet normalizes and segments the input and how the activation function performs special processing on a matrix. These operations are the prerequisites for the smooth operation of Legendre wavelet.

[0084] 3. Method for determining parameters K, M, and C in the activation function design technical solution. Technical solutions S4, S5, and S6 respectively describe how to determine the structural parameters K and M and the numerical parameter C in the Legendre wavelet activation function. Each parameter has a huge impact on the structure and value of the activation function. The determination of each parameter must be able to adapt to the characteristics of the data set in order to obtain high-precision results on the test set.

[0085] The following effects are achieved:

[0086] 1. The adaptability of the activation function structure design is achieved. In the ReLU activation function, the function divides the input into two positive and negative intervals, and uses 0-order and 1-order functions to connect the two intervals to form the function as a whole. In the LeakyReLU activation function, the function divides the input into two positive and negative intervals, and uses two 1-order functions to connect the two intervals to form the function as a whole. How many intervals the activation function input is divided into and what function to use for each interval was originally a controversial experimental issue. In the activation function design scheme based on Legendre wavelet, the function divides the input into 2K-1 intervals (as shown in step S3), and each interval is composed of a weighted combination of several polynomials of less than M degrees. Therefore, this design allows the activation function to rely on the network to determine the optimal activation function structure for the data set.

[0087] 2. The adaptive and piecewise learnable design of the numerical parameters of the activation function is realized. In the design of traditional activation functions such as sigmoid and tanh, the function is continuously differentiable within the domain interval, but there is a gradient vanishing phenomenon; in the design of activation functions based on rectified linear units such as ReLU, the function is continuous and nonlinear as a whole, and is not differentiable at the connection of the segmented intervals. At the same time, the dead neural network phenomenon is introduced on the negative semi-axis of the input. The learnable activation function design based on Legendre wavelet connects each interval, and the interval is piecewise differentiable, continuous and non-linear. After the input is segmented, each segment is represented by a different weighted sum of polynomials (according to the method provided in step S6), and the weights are regarded as network parameters and updated together with the network. Therefore, the activation function trained by this design method can learn the characteristics of inputs at different scales.

[0088] 3. Improve the convergence of the existing network. This is because in the Legendre wavelet, the Legendre wavelets in different intervals are orthogonal to each other. As the basis of the input-output mapping in this interval, the approximation ability of the entire network is improved, and the network can converge with fewer iterations (such as Figure 4 , Figure 5 As shown). This is because, in general, linearly independent vectors can be used as the basis of a finite-dimensional linear space, and orthogonal vectors are naturally linearly independent. The difference between the two is that an orthogonal basis can represent the maximum amount of information with the least data, while a non-orthogonal basis will have a certain "compressibility", which is a possible reason for limiting the slow convergence of the network and the failure to reduce network loss. The Legendre polynomials are obtained by Taylor expansion using Schmidt orthogonalization, and the Legendre wavelet is constructed using the wavelet function. Since the inner product between Legendre wavelets is 0, they are also orthogonal in the finite-dimensional linear space where the inner product is defined.

[0089] 4. Improved the accuracy of the existing network. The improvement in accuracy comes from the adaptability of the activation function structure on the one hand, and partly from the excellent approximation ability of the activation function on the other. In most cases, there is a high degree of similarity in the characteristics of the training set and the test set in the data set, which makes machine learning and deep learning theory practical. For specific tasks, such as CIFAR-100, Mnist, ImageNet image classification, CoCo object detection, GMM semantic segmentation, etc., since the feature distribution of the training set and the test set in certain domains (such as human skin color, the position of the subject in the picture) is similar, the feature extractor can be the same. The design method based on the Legendre wavelet learnable activation function can fit the characteristics of the data set well and significantly improve the network accuracy, as shown in Table 1.

[0090] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0091] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification, characterized in that: Here are the steps: S1: For a specific task, CIFAR-100 data is divided into a training set and a test set; S2: Select the deep network model, initialize the network parameters, activation function structure parameters K, M, and numerical parameter C; S3: Generate Legendre wavelet according to parameters K, M, C, and replace the activation function in the network; S4: Determine the structural parameter K; S5: Use the training set to train the model and determine the structural parameter M; S6: Use the training set to train the model again, determine the numerical parameter C, and obtain the optimal activation function and network weight W; S7: Substitute K, M, and C into Legendre wavelet to obtain the optimal activation function; S8: Substitute the optimal activation function into the network model and test it with the test set to improve the overall image classification accuracy of the network. Without using the pre-trained weights, the overall image classification accuracy of the network and the network convergence are improved. Use the optimal activation function in S7 to replace the original activation function in the network, and test the network on the test set to observe the overall progress and convergence of the network. S9: Compare the test results with the activation functions designed by mainstream methods.

2. The method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 1, characterized in that: Select a data set and divide it into a training set and a test set according to a certain ratio; at the same time, set the maximum computation amount MaxCal to limit the computation amount not to exceed during the network back propagation process.

3. The method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 1, characterized in that: Select a deep network model and randomly initialize the parameters of each layer of the network without loading any pre-trained weights.

4. The method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 1, characterized in that: The Legendre wavelet is generated as follows: ， In the formula, is the order of the Legendre polynomial, which is Input for activation function; express The Legendre polynomial of order is obtained by the recursive formula: 。 5. The method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 4, characterized in that: The interval The average is activation interval, expressed as: ； The feature matrix extracted by the feature extractor Perform maximum and minimum normalization processing to obtain a new matrix .

6. The method of applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 5, characterized in that: Each activation interval corresponds to Legendre wavelets and weights C, K, and M are all hyperparameters; C is learnable parameters, which are updated along with the network using gradient descent; Substitute the Legendre wavelet of the corresponding interval and take the weighted sum of the corresponding interval to get the final output of the activation function.

7. The method for applying a learnable activation function based on orthogonal Legendre wavelet to image classification according to claim 1, characterized in that: Use the K, M, and C values ​​determined in S4, S5, and S6, and construct the activation function according to the Legendre wavelet generation method in S3.

Images