An image classification model generation method and device based on an L2NU activation function

Abstract

The application relates to an image classification model generation method based on an L2NU activation function, which comprises the following steps: S101, replacing ReLU activation functions in original structures of all residual modules in a ResNet-50 image classification model with L2NU activation functions; S102, removing all BN layers in the ResNet-50 image classification model in step S101; S103, setting an activation function of an output layer of the ResNet-50 image classification model in step S102; S104, initializing each weight matrix of a neural network of the ResNet-50 image classification model in step S103 into a standard normal distribution, and setting each layer bias of the neural network into a 0 vector; S105, training the ResNet-50 image classification model in step S104 based on an image classification training data set, and obtaining an image classification model based on the L2NU activation function. Compared with a traditional activation function, the L2NU activation function of the application has the beneficial effect that the model classification precision function is improved when the L2NU activation function is applied to a deep learning model for image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and in particular to an image classification model generation method and device based on an L2NU activation function. BACKGROUND

[0002] Nowadays, the network structure of deep learning models is constantly expanding, and in particular, the depth of the network is constantly deepening, which brings many training problems. When the number of network model layers exceeds a certain number, the training process becomes increasingly slow and even difficult to obtain optimization. This is a problem that all mainstream deep learning models currently face. For example, compared with FCN, a network with more than 10 layers cannot be trained and optimized by gradient descent without introducing any training techniques. In a limited training time, the accuracy of the model may not be improved at all. The training problem mainly comes from the fact that the update gradient cannot be effectively backpropagated. SUMMARY

[0003] To solve the above technical problems, the present application provides an image classification model generation method and device based on an L2NU activation function.

[0004] In a first aspect, the present application provides an image classification model generation method based on an L2NU activation function, which comprises the following steps:

[0005] S101: replacing the ReLU activation function in the original structure of all residual modules in the ResNet-50 image classification model with an L2NU activation function;

[0006] The L2NU activation function is specifically:

[0007]

[0008] Wherein, n is the number of neurons in the network layer of the image classification model, X is the pre-activation mode of the network layer, and X=(x1, x2, …, xn), xi is the output of each neuron, and i∈{1, 2, …, n}; n i

[0009] S102: removing all BN layers in the ResNet-50 image classification model in step S101;

[0010] S103: setting the activation function of the output layer of the ResNet-50 image classification model in step S102 to

[0011] ​​S104: initialize each weight matrix of the neural network of the ResNet-50 image classification model in step S103 to a standard normal distribution, and set each layer bias of the neural network to a 0 vector;

[0012] S105: train the ResNet-50 image classification model in step S104 based on an image classification training data set to obtain an image classification model based on an L2NU activation function.

[0013] By adopting the above technical solution, compared with traditional activation functions such as ReLU, Tanh and Sigmoid, the L2NU activation function of the present application has the following effects when applied to a deep learning model for image classification: the model classification precision function is improved; the model parameter training time is shortened, and the model converges faster; the model training process is more stable, and the parameter value range and the gradient back propagation value range are relatively stable during the training process.

[0014] Preferably, S103 specifically includes: setting the activation function of the output layer of the ResNet-50 image classification model in step S102 to and performing cross-entropy loss calculation on the loss function of the ResNet-50 image classification model based on the L2NU p function; the loss function is specifically:

[0015]

[0016] wherein Y is the logits output of the network, and Y=(y1, y2, …, yn), yn is the unactivated output value of the i-th neuron of the network output layer, n is the label dimension, zn is the i-th element of the sample label Z=(z1, z2, …, zn) in one-hot form. n i i n

[0017] Preferably, S105 specifically includes: training the ResNet-50 image classification model in step S104 based on CIFAR-10 and CIFAR-100 image classification training data sets using the stochastic gradient descent method, setting the optimizer to the Momentum optimizer, setting the momentum coefficient to 0.9, setting the data batch size to 100, and using a learning rate of 0.1 for 100 rounds of training in the training process, and then using a learning rate of 0.01 for 10 rounds of parameter fine-tuning to obtain an image classification model based on the L2NU activation function.

[0018] Preferably, after step S105, the method further includes: S106: verifying the performance of the image classification model based on the L2NU activation function.​​​​

[0019] Preferably, the S106 specifically includes: training FCNs with different numbers of layers respectively using Sigmoid activation functions, Tanh activation functions, ReLU activation functions and L2NU activation functions, and recording the optimal accuracy of the FCN network on the CIFAR-10 image classification training data set after 3000 rounds of training with a learning rate of 0.01.

[0020] Preferably, the S106 further includes: learning CIFAR-100 image classification training data sets respectively using VGG-11, VGG-13, VGG-16 and ResNet convolutional neural network models, replacing the ReLU activation functions of the convolutional layers and fully connected layers in the original structure of each convolutional neural network with L2NU activation functions, to compare the performance differences between L2NU activation functions and ReLU activation functions, and replacing the Softmax functions of the output layers of each convolutional neural network with L2NU p functions, to compare the performance differences between L2NU p functions and Softmax functions.

[0021] In a second aspect, the present application further provides an image classification method, which comprises the following steps:

[0022] S201: obtaining an image classification sample to be classified;

[0023] S202: inputting the image classification sample into an image classification model based on L2NU activation functions, wherein the image classification model based on L2NU activation functions is obtained by pre-training according to the method of the first aspect;

[0024] S203: outputting the classification result of the image classification model based on L2NU activation functions.

[0025] In a third aspect, the present application further provides an image classification model generation device based on L2NU activation functions, which comprises:

[0026] a model modification module: configured to replace the ReLU activation functions in the original structure of all residual modules in the ResNet-50 image classification model with L2NU activation functions; further configured to remove all BN layers in the ResNet-50 image classification model; further configured to set the activation function of the output layer of the ResNet-50 image classification model to further configured to initialize each weight matrix of the neural network of the ResNet-50 image classification model in step S104 to a standard normal distribution, and set the bias of each layer of the neural network to a 0 vector;

[0027] The mathematical definition of the L2NU activation function is as follows:

[0028]

[0029] where n is the number of neurons in the network layer of the image classification model, X is the pre-activation mode of the network layer, and X = (x1, x2, …, xn), x n ), xi is the output of each neuron, and i e {1, 2, …, n}; i

[0030] The training module is configured to train the modified ResNet-50 image classification model based on an image classification training data set;

[0031] The determining module is configured to train the modified ResNet-50 image classification model into an image classification model based on the L2NU activation function by the method of the first aspect.

[0032] In a fourth aspect, the present application further provides an image classification device, which comprises:

[0033] The acquisition module is configured to acquire an image classification sample collected by an image acquisition device;

[0034] The classification module is configured to input the image classification sample into the image classification model based on the L2NU activation function, which is trained in advance based on the method of the first aspect, and output a classification result of the image classification model based on the L2NU activation function.

[0035] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the computing method of the first aspect.

[0036] In summary, the present application at least has the following beneficial technical effects:

[0037] 1. The present application proposes a brand-new deep learning activation function, which greatly expands the maximum trainable depth of the model when the gradient optimization algorithm is applied to the deep learning model. Unlike the traditional activation function mapping from numerical value to numerical value, the L2 normalization unit is a mapping from mode to mode. Due to its special mathematical structure, it can ensure that the gradient is effectively back-propagated during the gradient back-propagation process. This feature ensures that the application fields of deeper network models such as image classification and natural language processing can obtain more effective and stable optimization gradients when using deep models.

[0038] ​2. Compared with traditional activation functions such as ReLU, Tanh and Sigmoid, the L2NU activation function of the present application has the following effects when applied to a deep learning model for image classification: the model classification accuracy function is improved; the model parameter training time is shortened, and the model converges faster; the model training process is more stable, and the parameter value range and the gradient backpropagation value range remain relatively stable during the training process. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain principles of the present application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0040] Figure 1 is a flow chart of an image classification model generation method based on the L2NU activation function in an embodiment of the present application.

[0041] Figure 2 is a schematic diagram of the change in data distribution of L2NU acting on a two-dimensional plane.

[0042] Figure 3 is a schematic diagram of L2NU function images drawn based on different a values.

[0043] Figure 4 is a comparison diagram of traditional activation functions and L2 normalization units.

[0044] Figure 5 is a schematic diagram of replacing ReLU in the original structure of all residual modules in the ResNet-50 image classification model with L2NU.

[0045] Figure 6 is a schematic diagram of the function image of |Softmax(x)-Softmax(y)| and the gradient direction of maximizing the function.

[0046] Figure 7 is a schematic diagram of the function image of |L2NUp(x)-L2NUp(y)| and the gradient direction of maximizing the function.

[0047] Figure 8 is a schematic diagram of the |f(x)-f(y)| gradient image corresponding to Softmax and L2NUp.

[0048] Figure 9 is a schematic diagram of the classification accuracy of different FCNs on the CIFAR-10 dataset as the number of network layers changes.

[0049] Figure 10 is a flow chart of an image classification method in an embodiment of the present application.

[0050] Figure 11 is a module structure schematic diagram of an image classification model generation apparatus based on an L2NU activation function in an embodiment of the present application.

[0051] Figure 12 is a module structure schematic diagram of an image classification apparatus in an embodiment of the present application.

[0052] Figure 13 is a structure schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0054] In a deep learning model, the activation function is an important component indispensable to each layer of the network. The activation function has very rich roles in the network, including providing non-linear capabilities for the model, numerically compressing the intermediate output, converting the network output into a predicted probability with mathematical meaning, etc. Commonly used activation functions include Sigmoid function, tanh function, and other activation functions with gradients tending to zero at both ends; ReLU, ELU, SeLU, and other activation functions with linear at one end and gradients equal to or tending to zero at the other end; Swish function, GeLU, and other non-monotonic activation functions; MAXOUT activation functions that select multiple activations. For activation functions with gradients tending to zero at both ends, if the activation value is different from the target activation value, the gradient will tend to zero in the back propagation process, and it will take a huge calculation cost to update the activation value to the target value under the condition of the gradient tending to zero, which is a long process and may even not be completed within a limited time. This problem is called neuron saturation.

[0055] On the other hand, the gradient of the function at both ends is gradually decreasing, so during the training process, the gradient of back propagation will decrease layer by layer based on the chain rule, and the network parameters at the front will be increasingly difficult to train effectively. Due to these problems, ReLU and other one-end linear functions are widely used. Because of the characteristics of the linear one-end value compression, it can help the model to effectively back propagate the gradient and alleviate the gradient disappearance problem.

[0056] However, such activation functions cause the mean of the data distribution to shift towards the positive direction, rather than being centered at zero. Without normalization, this shift in distribution will inevitably cause the network parameters to shift, causing the numerical scale of the update target of some parameters to increase, which will result in more iterations being required for the fitting process during the early training of the network, and will not be conducive to the subsequent fine-tuning of the parameters.

[0057] More importantly, the gradient of the negative end of ReLU is equal to zero, which means that when the output of all samples at a certain neuron is less than or equal to zero, the neuron can no longer obtain the gradient, and the neuron is said to be dead. In the extreme case, when all neurons in a layer have this problem, the network is completely paralyzed and loses the ability to update the parameters of the layer and all previous layers.

[0058] Swish function compared with GeLU, ReLU introduces more stable optimization process of the network by smoother function and smoother gradient, however, the problem of neuron saturation and output distribution shift still exists.

[0059] MAXOUT activation function replaces a neuron with a group of neurons, and the output value at each time is the maximum output value of the group of neurons. This can not only introduce a nonlinear process, but also enable the gradient to be back propagated without additional nonlinear functions, thus improving the gradient optimization ability of the model. However, compared with traditional neural networks, the parameter demand of MAXOUT increases exponentially, which is not conducive to the construction of complex and large network structures.

[0060] In deep neural network structures, a new technique called Batch Normalization (BN) method is widely used.

[0061] The BN method normalizes a batch of input data, and then scales and shifts the normalized data distribution through trainable parameters to unify the output data distribution of the specified network layer.

[0062] Since BN method has trainable parameters and acts on the output of the entire network, it is also called BN layer. The purpose of BN layer is to keep the output of network layer within a stable numerical range through the normalization process, thereby reducing the possibility of gradient propagation problems, and to fine-tune the output data distribution through trainable parameters, making it easier to be fitted or divided by the subsequent network layer, which also ensures the smooth progress of gradient backpropagation.

[0063] Now, convolutional network models such as VGG, GoogLeNet or ResNet model all use BN layer to ensure the smooth progress of network training and play a role in stabilizing training.

[0064] However, due to the special operation mechanism of BN layer, it consumes a large amount of computing resources. And a small number of trainable parameters in BN layer have a great impact on model performance, so there are higher requirements for model hyperparameter settings, otherwise it is difficult to obtain reasonable model performance.

[0065] Numerous studies have shown that reasonable parameter initialization can effectively reduce the possibility of gradient propagation problems and optimize network performance. For example, the Xavier initialization method initializes the network weights to a uniform distribution with a certain variance, so that the data distribution in the forward propagation and backpropagation process of the network remains as consistent as possible, thereby ensuring stable parameter training.

[0066] However, the Xavier initialization method is based on strict assumptions, and most commonly used activation functions do not apply to this method. The He initialization method complements the initialization method for ReLU activation function. By scaling the target variance of Xavier initialization, it balances the variance change caused by ReLU, effectively improving the gradient optimization performance of the model.

[0067] However, initialization methods alone cannot guarantee stable gradient propagation of the model during long-term training, and additional mechanisms are needed to ensure stable parameter distribution during optimization.

[0068] Therefore, we propose a new activation function with the following characteristics:

[0069] (1) 0-centered rotational symmetry, so that the output data has a zero-centered distribution, which is convenient for model fitting;

[0070] (2) has a nonlinear compression ability, and the output value range is within a certain range;

[0071] (3) smooth curve, conducive to the stability of model parameter update;

[0072] (4) The gradient of the back propagation process is greater than a certain value, which has prevented the problems of neuron saturation and neuron death. However, the second point and the fourth point are in conflict in essence, and the numerical compression capability means a smaller gradient, which is not conducive to the gradient propagation in a multi-layer network. Therefore, the contradiction needs to be solved by proposing a new form of activation function.

[0073] A first aspect of the embodiments of the present application discloses a method for generating an image classification model based on an L2NU activation function, referring to Figure 1 The method specifically includes the following steps:

[0074] S101: Replacing ReLU activation functions in original structures of all residual modules in a ResNet-50 image classification model with L2NU activation functions;

[0075] The L2NU activation function is specifically:

[0076]

[0077] Wherein, n is the number of neurons in a network layer of the image classification model, X is the pre-activation mode of the network layer, and X=(x1, x2, …, xn), xi is the output of each neuron, and i∈{1, 2, …, n}; n i

[0078] S102: Removing all BN layers in the ResNet-50 image classification model in step S101;

[0079] S103: Setting the activation function of the output layer of the ResNet-50 image classification model in step S102 to

[0080] In specific embodiments, S103 specifically includes: setting the activation function of the output layer of the ResNet-50 image classification model in step S102 to and performing cross-entropy loss calculation on the loss function of the ResNet-50 image classification model based on the L2NU p function; the loss function is specifically:

[0081]

[0082] Wherein, Y is the logits output of the network, and Y=(y1, y2, …, yn), yi is the unactivated output value of the i-th neuron of the network output layer, n is the label dimension, zi is the i-th element of the sample label Z=(z1, z2, …, zn) in one-hot form. n i i n ​​​​​​

[0083] S104: initialize each weight matrix of the neural network of the ResNet-50 image classification model in step S103 to a standard normal distribution, and set each layer bias of the neural network to a 0 vector;

[0084] In specific embodiments, the Xavier initialization or He initialization technique is not used, and each weight matrix of the network is directly initialized to a standard normal distribution, and each layer bias of the network is set to a 0 vector.

[0085] S105: training the ResNet-50 image classification model in step S104 based on an image classification training data set, to obtain an image classification model based on an L2NU activation function.

[0086] In specific embodiments, S105 specifically includes: based on CIFAR-10 and CIFAR-100 image classification training data sets, training the ResNet-50 image classification model in step S104 using a stochastic gradient descent method, setting the optimizer to a Momentum optimizer, setting the momentum coefficient to 0.9, setting the data batch size to 100, and using a learning rate of 0.1 for 100 rounds of training, and then using a learning rate of 0.01 for 10 rounds of parameter fine-tuning, to obtain an image classification model based on an L2NU activation function.

[0087] S106: performance verification of the image classification model based on the L2NU activation function.

[0088] In specific embodiments, S106 specifically includes: training FCNs of different layers using Sigmoid activation functions, Tanh activation functions, ReLU activation functions, and L2NU activation functions, respectively, and recording the optimal accuracy of the FCN network after 3000 rounds of training on the CIFAR-10 image classification training data set using a learning rate of 0.01.

[0089] In further embodiments, S106 further includes: learning CIFAR-100 image classification training data sets using VGG-11, VGG-13, VGG-16, and ResNet convolutional neural network models, respectively, by replacing the ReLU activation functions of the convolutional layers and fully connected layers in the original structure of each convolutional neural network with L2NU activation functions, to compare the performance differences between L2NU activation functions and ReLU activation functions, and by replacing the Softmax functions of the output layers of each convolutional neural network with L2NU p functions, to compare the performance differences between L2NU p functions and Softmax functions.

[0090] According to an embodiment of the present invention, in step S101, a novel type of activation function, L2NU, is disclosed, which is called L2-Normalization Unit (L2NU). L2NU is cleverly constructed to satisfy the four characteristics required by the aforementioned activation function, introducing a process similar to BN layers into the network. Without requiring BN layers or parameter initialization settings, it possesses the excellent properties of a model that applies BN layers and parameter initialization methods, significantly increasing the trainable depth of the model.

[0091] In essence, L2NU compresses the distribution in n-dimensional space onto the surface of an n-dimensional unit hypersphere centered at the origin through numerical scaling, as shown below. Figure 2 As shown, Figure 2 The data distribution changes when L2NU is applied to a two-dimensional plane are shown.

[0092] The purpose of constructing L2NU is to maintain a constant variance in the intermediate outputs of each layer of the network. For any fully connected network using L2NU as the activation function, the input dimension of a certain layer of the network is n, the output dimension is m, and the layer input X = (x1, x2, ..., xn) is given. n The weights W are n×m dimensional matrices, the biases B are m dimensional vectors, and the unactivated output is Y = X·W + B. Assuming X, W, and B are independent distributions with a mean of 0, according to Var(Y) = n·Var(X)·Var(W), where Var(X) represents the variance of the distribution corresponding to X. Since X is the activated output of the previous layer, the sum of squares has been normalized to 1, therefore we have formula (1-2):

[0093]

[0094] That is, the variance of the inactive output distribution of any layer in the network is only related to the variance of the weight distribution of that layer. As long as the weights of all layers are set with the same initial distribution, the distribution of the network forward propagation can be guaranteed to be stable.

[0095] In addition, L2NU possesses many excellent properties that enable it to function as an activation function, including:

[0096] (1) The output data is distributed in a zero-centered distribution; (2) It has numerical compression capability, and the range of the function is between (-1, 1); (3) The function is smooth, the function value is continuous, and the function gradient value is also continuous; (4) It has adaptive gradient; (5) The output data is independent of the pre-activation modulus, and can effectively reduce the training cost of the model after replacing Softmax.

[0097] It is necessary to supplement that when L2NU acts on the activation function of the convolutional layer, only independent normalization operation is performed on each feature map, rather than all output bits of a single sample, which enhances the expression ability of the model. Moreover, the L2NU in the convolutional layer is amplified by multiplying a constant coefficient, and the coefficient is equal to the square root of the product of the length and width of the corresponding feature map, the purpose is to maintain the stability of the distribution of network forward propagation.

[0098] In specific embodiments, the characteristic of L2NU adaptive gradient is the key to ensure the smooth optimization of the network, and the present application discloses how L2NU adaptively changes the gradient of each neuron activation function, thereby helping the process of gradient backpropagation to proceed more smoothly.

[0099] In MAXOUT, only one neuron in a group of neurons plays a role in network forward propagation or gradient backpropagation at a time, and the propagation process of the remaining neurons is masked by the activation function and does not participate in calculation and update, thus causing a large amount of parameter redundancy in the network. For L2NU, the use of activation function is a group of neurons in network layer, and the values output by them jointly act to construct the activation function between neurons, thereby generating their respective activation values.

[0100] Suppose the output of a layer of neurons is (x1, x2, …, x n If only the change of the value x i output by a neuron in the layer of neural network is considered, and the activation values of the remaining neurons are set as constants, it is not difficult to let Then, the activation function of the neuron can be represented as formula (1-3):

[0101]

[0102] For different values of a, the function curve will also differ, Figure 3 shows the function images drawn based on a number of different values of a.

[0103] From the different function discoveries in Figure 3 , it can be found that different a values will directly cause the activation function form to change, more specifically, the value range of the gradient of the activation function will change. When a tends to 0, the function tends to the Sign function, that is, the gradient value at the origin of the coordinate axis is large and changes sharply, and the gradients at both ends decrease until they tend to 0. When a takes a large value, the function is linear in a certain interval near the origin, and the gradient gradually transitions to 0 towards both ends.

[0104] Such function characteristics bring the following characteristics:

[0105] (1) When the output values of the neurons in this layer have large differences, and when the outputs of other neurons are all small, while one neuron output is large, the gradient of back propagation of this neuron will tend to 0, while other neurons will have large gradients to balance the excessive value of individual neurons to drown out the contributions of other neurons. This feature ensures that the parameters of each neuron in the same layer are updated by reasonable gradients to ensure that they are in similar numerical scales;

[0106] (2) When the average numerical scale of the output values of the neurons in this layer is small, each neuron parameter will obtain a large update gradient, and vice versa. Generally, the scale of the intermediate output of the network is gradually increasing, so such characteristics make the network have larger update gradients in the early training and more stable update gradients in the later training, and the scale of the parameters is also controlled within a certain range by the activation function, reducing the possibility of gradient explosion problem;

[0107] (3) Based on the first two characteristics, the numerical scale of each neuron is gradually approaching during the training process, so it can be simply estimated that for a layer with n neurons, when the expected value of the weight W is 0 and the variance is 1, the gradient of each neuron activation function is around n -1 / 2 This is independent of the size of the output value before neuron activation, which effectively prevents the problem of neuron saturation and eliminates the concern of neuron death;

[0108] (4) L2NU has the ability to scale the parameters of each network layer while keeping the activation value unchanged, as shown in equation (1-4):

[0109] L2NU(X×W+B) = L2NU(X×k×W+k×B) (1-4)

[0110] Where X is the input pattern, W is the weight matrix of the network layer, B is the bias vector of the network layer, and k is a positive constant. This feature can prevent the parameter values from expanding too much by adding an L2 regularization term to the loss function during network training.

[0111] Based on the above characteristics brought by L2NU, the gradient values during the training process of the network are in a dynamic change process. Such dynamic changes prevent all neurons from having extreme values and being unable to back propagate, and also ensure that at least one neuron in each network layer has an effective gradient to propagate to the previous network layer. Therefore, the application of L2NU as an activation function in the network can greatly reduce the possibility of gradient problems in the gradient optimization method, thereby enabling the network structure design to be further deepened.

[0112] As Figure 4As shown in the figure, the traditional activation function such as Sigmoid, ReLU and L2 normalization unit are compared, the traditional activation function takes the output value of a single neuron as input and activates the mapping to a new value; the L2 normalization unit takes the output mode of the network layer as input and activates the mapping to a new mode.

[0113] As shown in the figure, the traditional activation function such as Sigmoid, ReLU and L2 normalization unit are compared, the traditional activation function takes the output value of a single neuron as input and activates the mapping to a new value; the L2 normalization unit takes the output mode of the network layer as input and activates the mapping to a new mode. Figure 5 As shown in the figure, the traditional activation function such as Sigmoid, ReLU and L2 normalization unit are compared, the traditional activation function takes the output value of a single neuron as input and activates the mapping to a new value; the L2 normalization unit takes the output mode of the network layer as input and activates the mapping to a new mode.

[0114] According to the embodiment of the present application, in the prior art in recent years, almost all classification neural networks will use Softmax function as the activation function of the output layer, because of its excellent statistical theory basis, the combination of Softmax function and cross-entropy loss function plays an excellent effect on the training of classification model. However, L2NU will make the optimization of network easier if it replaces the traditional Softmax function as the activation function of the output layer due to its special function structure.

[0115] In order to facilitate the use of cross-entropy loss function, when using L2NU as the activation function of the output layer, the translation and scaling are performed to make the value range of L2NU in [0, 1], the process is defined by the formula, and we call the transformed activation function L2NU p , the expression of L2NU p is:

[0116]

[0117] Suppose a classification problem with two categories, the sample labels of the two categories are (0, 1) and (1, 0) respectively, so for the prediction value (x, y) of the network output, the maximization of |f(x)–f(y)| value can improve the classification ability of the model, where f is the activation function. Figure 6 And Figure 7 the |f(x)–f(y)| function image of the network prediction value after Softmax activation function and the network prediction value after L2NUp activation are drawn respectively.

[0118] By observing Figure 6It's easy to see that only when the numerical scale of the network's predicted values ​​is large can the two activation values ​​Softmax(x) and Softmax(y) after passing through the Softmax function have a larger gap. Therefore, the gradient direction of network training will be based on increasing the numerical scale of these two values. This means that for the network to achieve good results, the numerical scale of the parameters first needs to reach or exceed a certain threshold. This places higher demands on the initialization settings of the network parameters and the setting of the learning rate. At the same time, this gradient direction will also cause the network parameters to continuously increase, and the increased parameters will bring additional training problems. For L2NU... p ,like Figure 7 As shown, compared to the Softmax function, it still has a range that maximizes the two activation values ​​L2NU(x) and L2NU(y) at all numerical scales, even near the origin of the coordinate axis. Therefore, the gradient for maximizing the distance between them converges towards x = -y. This characteristic brings the advantage that regardless of the numerical scale of the network parameters, the optimal goal of maximizing the distance between them can be achieved. Furthermore, the gradient optimization is the same at all numerical scales, so this process is faster than that of the Softmax function, and the optimal goal of L2NU is achievable, while for Softmax, this optimal goal can only be approached through continuous iteration and can never be reached. Figure 8 As shown, the above process can be more clearly understood by drawing a gradient comparison graph of the two functions.

[0119] The aforementioned advantages stem from the fourth characteristic of L2NU described above. Essentially, since the network's computation process is no longer affected by the magnitude of the activation patterns in each layer, but only by their orientation, the network's logits output during optimization no longer needs to fit infinitely large values, but rather the vector direction of the labels. Compared to training one vector to become another with a larger magnitude, the cost of using rotation to align the directions of two vectors is far less than the former; therefore, L2NU... p Applying it to the output layer can effectively accelerate model training.

[0120] According to an embodiment of the present invention, in step S106, the present application performs performance verification on an image classification model based on the L2NU activation function. In the performance verification experiment, the effects of activation functions such as ReLU, Sigmoid, Tanh and L2NU on the accuracy performance of fully connected networks and the maximum number of trainable layers of the model are compared. Furthermore, the changes in model training and accuracy performance after L2NU replaces the native activation functions of several existing excellent model structures of convolutional networks are demonstrated.

[0121] The CIFAR-10 and CIFAR-100 datasets are used in the performance verification experiment, which are used to verify the influence of the activation function on the model accuracy and the influence on the gradient backpropagation. The CIFAR-10 dataset is a color image dataset with three channels of RGB, 32x32 pixels in size, which contains ten categories of several animals and vehicles, and contains 50,000 training images and 10,000 test images. The CIFAR-100 dataset is similar to the CIFAR-10 dataset, with the same size of image data, and the same number of training samples and test samples. The difference is that the number of categories of the CIFAR-100 dataset is increased to 100, which requires the model to have stronger fitting and classification ability to perform this training task. All image data in the experiment is augmented by translation and flipping to expand the training sample size, where the translation of the image is in the up, down, left and right four directions, and the maximum translation is 4 pixels.

[0122] The performance verification experiment is divided into two parts. The first part uses different activation functions to train FCN of different layers, and records the optimal accuracy of the network on the CIFAR-10 dataset after 3000 rounds of training with a learning rate of 0.01. This ensures that the model is fully trained and eliminates the influence of overfitting. The training uses the Adam optimizer and the cross-entropy loss function, and 5 parallel experiments are performed. When the accuracy of the 5 experiments of a certain activation function corresponding to the network is less than or equal to 10%, it is considered that the network layer cannot be trained within the given time. It should be noted that in order to facilitate the construction and representation of network models of different depths, the output dimensions (number of neurons) of all network layers are arranged in an equal ratio sequence. Specifically, for an n-layer network, the parameters of each layer are defined by formula (1-6), where d i is the output dimension of the i-th layer, d in is the input dimension of the network, i.e. the dimension of the data, d out is the output dimension of the network, i.e. the dimension of the label, represents the ceiling function.

[0123]

[0124] The second part of the experiment uses three VGG convolutional neural network models with different structures and a ResNet model to learn the CIFAR-100 classification dataset. The three VGG networks are VGG-11, VGG-13 and VGG-16, which are structures A, B and D mentioned in the original work, and the default model does not use the batch normalization method. By replacing the activation function of the convolutional layer in the original network structure with the ReLU of the fully connected layer activation function with L2NU, the performance difference between L2NU and ReLU as the activation function is compared. On the other hand, by replacing the Softmax function of the output layer of the network with L2NU p , the performance difference between L2NU and Softmax function output layer activation function is compared. The VGG network uses the Adam optimizer, a learning rate of 0.001, a data batch size of 2000 samples, and 100 training rounds. Each model is run in parallel for 3 experiments. ResNet uses the Momentum optimizer with a momentum coefficient of 0.9, a data batch size of 100, and a learning rate of 0.1 for 100 rounds of training, followed by 10 rounds of parameter fine-tuning using a learning rate of 0.01. When measuring model running time, the following hardware is used to train the model: Intel 9960X + Nvidia RTX2080Ti.

[0125] In the performance verification experiment of training FCN with different activation functions and different numbers of layers, the experiment shows the training of FCN using Sigmoid function, Tanh function, ReLU and L2NU as the activation function of the hidden layer, where the models corresponding to Sigmoid function, Tanh function and ReLU use Softmax function as the output layer activation function, and the network using L2NU as the hidden activation unit uses Softmax function and L2NU p as the output layer activation function, respectively. Two structures are trained, and the FCN network structure used in the experiment is 2 to 16 layers, 20 layers and 50 layers, respectively, and the specific network layer dimensions are defined in equations (1-6).

[0126] The training accuracy of FCN with different numbers of layers in a given time is shown in Figure 9 . From Figure 9As can be seen, the FCN with ReLU as the activation function has the best classification accuracy among the three activation functions for 6 layers and less than 6 layers. However, the FCN with ReLU and the FCN with Sigmoid function and the FCN with Tanh function all reach the best performance when the number of layers is 3, and the performance of the model starts to decline when the number of layers is more than 3. When the number of layers is 8 or more, the FCN with ReLU cannot be trained and optimized normally in all parallel experiments, and the accuracy is only 10%, which means that the model performance is at the stage of blind guessing. Then, the FCN with Tanh function cannot be trained when the number of layers is 9 or more, and the FCN with Sigmoid function cannot be trained when the number of layers is 10 or more. In contrast, the FCN with L2NU as the activation function of the hidden layer and the FCN with L2NU p As the output layer activation function of the FCN, the training accuracy steadily increases with the increase of the number of network layers. When the depth of the network is 12, the accuracy is higher than that of the 3-layer ReLU network, and the network structure is still normal when the depth of the network is 50. Even in the same number of training iterations, the 50-layer FCN with L2NU reaches a classification accuracy of 66.72%, which is much higher than the classification accuracy of 61.15% of the three-layer FCN with ReLU. In addition, if the output layer of the network using L2NU is replaced by the Softmax function, the training results of the parallel experiments will fluctuate greatly, and the average classification accuracy will also decrease with the increase of the number of network layers.

[0127] This experiment shows that the FCN model with L2NU as the activation function of the hidden layer can be trained to a much greater depth, and even a 50-layer network can be trained with effective growth of accuracy. On the other hand, the experiment verifies the effectiveness of L2NU p as the activation function of the output layer in calculating the loss function, L2NU p Replacing the Softmax function can effectively improve the training stability and classification effect of the deep structure of the FCN model using L2NU.

[0128] In the performance verification experiment of replacing the activation function of the convolutional network with L2NU, it first shows the use of L2NU to replace the activation function of three different layer VGG models, in order to show the influence of L2NU on the classification accuracy of the model compared with ReLU applied to the fully connected layer and the convolutional layer. The classification accuracy verification indicators include Top-1 accuracy and Top-5 accuracy. Among them, the default VGG network does not use BN layer, and only in the VGG-16 model corresponding experiment, the BN layer is added. Then we show the accuracy performance of L2NU replacing ReLU on ResNet50 model, and compare the accuracy influence and training process calculation cost brought by enabling and disabling BN layer. The specific experimental data is as follows:

[0129]

[0130] Table 1 Model accuracy performance after replacing VGG-11 activation functions with L2NU

[0131]

[0132] Table 2 Model accuracy performance after replacing VGG-13 activation functions with L2NU

[0133]

[0134] Table 3 Model accuracy performance after replacing VGG-16 activation functions with L2NU

[0135] Model Activation function Use BN layer Accuracy (%) Total training time (h) ResNet-50 ReLU No 1.00 1.61 ResNet-50 L2NU No 68.97 3.01 ResNet-50 ReLU Yes 66.72 5.16 ResNet-50 L2NU Yes 67.27 6.46

[0136] Table 4 Model accuracy and training time comparison after replacing ResNet activation functions with L2NU

[0137] Table 1 shows that replacing all activation functions of VGG-11 with L2NU can effectively improve the Top-1 classification accuracy and Top-5 classification accuracy of the model. Similarly, the data in Table 2 and Table 3 shows that replacing all activation functions of VGG-13 and VGG-16 with L2NU can also effectively improve the Top-1 classification accuracy and Top-5 classification accuracy of the model. On the other hand, although the VGG-11 without BN can still be trained under the original activation function, in the VGG-13 with an increased number of layers, the model using ReLU as the activation function of all hidden layers cannot be effectively optimized in the given training rounds, with only 1% Top-1 accuracy and 5% Top-5 accuracy, i.e. the network fails to learn any knowledge from the dataset. Further, in the VGG-16 with a further increased number of layers, both models using ReLU as the activation function of the fully connected layer lose the ability to learn. Moreover, in the three experiments, replacing only the ReLU activation function of the fully connected layer with L2NU can improve the Top-1 accuracy and Top-5 accuracy. Therefore, the three experiments collectively demonstrate that when the number of network layers is deep, using L2NU as the activation function of the fully connected layer can obtain better model classification ability. Although the second model in Table 1 does not improve the Top-1 accuracy compared to the first model, the model still brings a small amount of improvement in Top-5 accuracy compared to the original VGG-11. In addition, Table 1, Table 2 and Table 3 collectively show that replacing the original ReLU structure with L2NU at any position can simultaneously improve the Top-1 accuracy and Top-5 accuracy.

[0138] Compared with the VGG-16 model in Table 3 using the BN layer, the VGG-16 model using the ReLU can be trained, but due to the use of the BN layer, the training cost is greatly increased, and the relative stable accuracy performance cannot be reached in 100 training rounds. On the other hand, the VGG-16 model using the L2NU can further improve the Top-1 accuracy and Top-5 accuracy in the same training rounds. This shows that the L2NU can also obtain accuracy benefits from the BN structure, and the process has a smaller training cost than the same structure network using the ReLU.

[0139] It should be noted that when the ReLU is used as the activation function of the hidden fully connected layer and the L2NUp is used as the activation function of the output layer, or the L2NU is used as the activation function of the hidden fully connected layer and the Softmax function is used as the activation function of the output layer, the network accuracy will decrease to different degrees. In combination with the previous experimental purposes, it can be considered that the performance of the last layer activation function is affected by the activation function of the previous one or several layers of the network, and the model using the L2NU p as the activation function of the output layer can achieve better results.

[0140] The data in Table 4 shows that under the premise of having the BN layer, both activation functions can ensure the stable training of the ResNet-50, and the network using the L2NU as the activation function can achieve higher classification accuracy. On the contrary, under the premise of not using the BN layer, the training stability of the ResNet model using the ReLU decreases significantly, and in the parallel experiment, the gradient problem occurs after 10 training rounds, which leads to the model being unable to be optimized. The ResNet model using the L2NU not only obtains normal training, but also achieves higher classification accuracy than the network using the BN layer in the same training rounds, which shows that the BN layer brings additional computational overhead and increases the difficulty of model training. More importantly, although the network using the L2NU increases the time consumption by 25% compared with the original ResNet, the L2NU can be trained without the BN layer, and the model can save 56% of the calculation time after removing the BN layer.

[0141] The present application proposes a new activation function, which is called L2 normalization unit, abbreviated as L2NU. The L2NU is different from the commonly used activation functions such as the ReLU or the Sigmoid function. The L2NU has an action range of a layer of network, while the action object of other functions is a single neuron. The L2NU takes the L2 normalization result of a layer of network output as the activation value, has certain nonlinear ability, and brings many additional advantages.

[0142] In the training of fully connected networks, after replacing the commonly used activation functions such as Sigmoid function, ReLU, Tanh function, etc. in fully connected networks with L2NU, the number of trainable layers of the model is greatly increased. Even without introducing any additional structure or additional training techniques, the training of a 50-layer fully connected network is completed, and the 50-layer model does not reach the upper limit of the ability of L2NU. On the other hand, in the experiment, the fully connected network using L2NU activation function surpasses the optimal accuracy obtained by other activation functions in all network structures after the network structure is deepened to 12 layers, and the accuracy is steadily improved with the gradual increase of the number of layers.

[0143] In the training of convolutional networks, after replacing the most commonly used ReLU in convolutional networks with L2NU as the activation function, the difficulty of model training is significantly reduced, and the model accuracy is improved. First, the experiment uses VGG convolutional networks with different structures as the basic model, and by replacing the activation functions of the convolutional layers and the activation functions of the fully connected layers respectively, it is verified that in a deep network structure model, using L2NU to replace any ReLU can effectively improve the Top-1 accuracy and Top-5 accuracy of the model. And the model using L2NU can normally and stably complete the training without performing batch normalization processing, while the VGG-13 (10 convolutional layers and 3 fully connected layers) and the deeper VGG-16 using ReLU cannot obtain optimization under the same conditions. Finally, the model using L2NU can still obtain better model accuracy by applying the batch normalization technology. Then, the experiment uses ResNet-50 as the basic model, replaces the original activation function with L2NU, and obtains effective accuracy improvement. Then, the experiment compares the network model without BN layer, and only the L2NU corresponding network is successfully trained. Combined with the previous experiment, this result reflects the excellent gradient propagation ability of L2NU, which can optimize independently of the BN layer and obtain similar or even better model performance, and at the same time, removing the BN layer saves a lot of computational cost for the model.

[0144] The new deep learning activation function proposed in this application greatly expands the maximum trainable depth of the deep learning model when applying the gradient optimization algorithm. Unlike traditional activation functions that map numerical values to numerical values, L2 normalization unit is a pattern-to-pattern mapping. Due to its special mathematical structure, it can ensure that there is always a gradient that is effectively backpropagated during gradient backpropagation. This feature ensures that image classification, natural language processing and other applications of deeper network models can obtain more effective and stable optimization gradients when using deep models.

[0145] Compared with traditional activation functions such as ReLU, Tanh and Sigmoid, the activation function has the following advantages when applied to a deep learning model for image classification: (1) the model classification accuracy function is improved; (2) the model parameter training time is shortened, and the model converges faster; (3) the model training process is more stable, and the parameter value range and the gradient back propagation value range are relatively stable during the training process.

[0146] Referring to Figure 10 In a second aspect, the present application further provides an image classification method, the method comprising the following steps:

[0147] S201: obtaining an image classification sample to be classified;

[0148] S202: inputting the image classification sample into an image classification model based on the L2NU activation function, wherein the image classification model based on the L2NU activation function is pre-trained based on the method disclosed in the above aspect;

[0149] S203: outputting the classification result of the image classification model based on the L2NU activation function.

[0150] Referring to Figure 11 In a third aspect, the present application further provides an image classification model generation device based on the L2NU activation function, the device comprising:

[0151] The model modification module 301 is configured to replace the ReLU activation function in the original structure of all residual modules in the ResNet-50 image classification model with the L2NU activation function, remove all BN layers in the ResNet-50 image classification model, and set the activation function of the output layer of the ResNet-50 image classification model to The model modification module 301 is further configured to initialize each weight matrix of the neural network of the ResNet-50 image classification model in step S104 to a standard normal distribution, and set each layer bias of the neural network to a 0 vector.

[0152] The mathematical definition of the L2NU activation function is as follows:

[0153]

[0154] wherein n is the number of neurons in the network layer of the image classification model, X is the pre-activation pattern of the network layer, and X=(x1, x2, …, xn), wherein xi is the output of each neuron, and i∈{1, 2, …, n}; n i

[0155] ​​Training module 302: Used to train the modified ResNet-50 image classification model based on the image classification training dataset;

[0156] Determine module 303: used to train the modified ResNet-50 image classification model into an image classification model based on the L2NU activation function through the method described in the first aspect.

[0157] Reference Figure 12 Fourthly, this application also proposes an image classification device, the device comprising:

[0158] Acquisition module 401: Used to acquire image classification samples acquired by the image acquisition device;

[0159] Classification module 402: Used to input image classification samples into an image classification model based on the L2NU activation function, wherein the image classification model based on the L2NU activation function is pre-trained based on the method described in the first aspect, and outputs the classification result of the image classification model based on the L2NU activation function.

[0160] Fifthly, refer to Figure 13 It shows a schematic diagram of the structure of a computer system 100 suitable for implementing electronic devices according to embodiments of the present application. Figure 13 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0161] like Figure 13 As shown, the computer system 100 includes a central processing unit (CPU) 101, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 102 or programs loaded from storage section 108 into random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the system 100. The CPU 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0162] The following components are connected to the I / O interface 105: an input part 106 including a keyboard, a mouse, etc.; an output part 107 including a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 108 including a hard disk, etc.; and a communication part 109 including a network interface card such as a LAN card, a modem, etc. The communication part 109 performs communication processing via a network such as the Internet. A drive 110 is also connected to the I / O interface 105 as necessary. A removable medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 110 as necessary, so that a computer program read therefrom is installed in the storage part 108 as necessary.

[0163] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 109, and / or installed from the removable medium 111. When the computer program is executed by the central processing unit (CPU) 101, the above-described functions defined in the methods of the present application are performed.

[0164] As another aspect, the present application also provides a computer-readable storage medium, which can be included in the electronic device described in the above embodiments; or can exist separately without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods as shown in the above embodiments. Figure 1

[0165] ​Note that the computer readable storage medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the present application, the computer readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave in a propagated data signal, in which the computer readable program code is embodied. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable storage medium that is not a computer readable storage medium that can be used to store program code, which can be used by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber, RF, etc., or any suitable combination thereof.

[0166] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0167] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. The body of computer readable program code can be any suitable set of instructions stored in a storage medium, which can be run on a system using an operating system. The system can be a mainframe computer, a desktop computer, a laptop computer, or other programmable computer components that can store program code and execute it. The system can also be a mobile device, such as a smart phone, a tablet, or other mobile device that can store program code and execute it. The system can also be a server, a cloud computing system, or other system that can store program code and execute it. The system can also be a combination of the above systems.

[0168] The specific implementations of the present application described above are not intended to limit the scope of the present application, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0169] In the description of the present application, it should be understood that the terms "upper", "lower", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be understood as a limitation on the present application. The word 'comprising' does not exclude the existence of elements or steps not listed in the claims. The word 'a' or 'an' in front of an element does not exclude the existence of multiple such elements. The simple fact that certain measures are recited in mutually different dependent claims does not indicate that the combination of these measures cannot be used to improve. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. A method for generating an image classification model based on an L2NU activation function, characterized in that: The method comprises the following steps: S101: replace the ReLU activation function in the original structure of all residual modules in the ResNet-50 image classification model with an L2NU activation function; The L2NU activation function is specifically: ； wherein n is the number of neurons in a network layer of the image classification model, X is the pre-activation pattern of the network layer, and X = (x1, x2, …, xn) n ), xi is the output of each neuron, and i ∈ {1, 2, …, n}; i ); S102: remove all BN layers in the ResNet-50 image classification model in step S101; S103: set the activation function of the output layer of the ResNet-50 image classification model in step S102 to ； Wherein, x represents the pre-activation mode of the network layer; S104: initialize each weight matrix of the neural network of the ResNet-50 image classification model in step S103 to a standard normal distribution, and set the bias of each layer of the neural network to a 0 vector; S105: train the ResNet-50 image classification model in step S104 based on an image classification training data set to obtain an image classification model based on the L2NU activation function.

2. The method of claim 1, wherein the method is based on an L2NU activation function. The S103 specifically comprises: setting an activation function of an output layer of the ResNet-50 image classification model in the step S102 to , and performing cross-entropy loss calculation on a loss function of the ResNet-50 image classification model based on an L2NUp function, the loss function specifically being: ; Where Y is the logits output of the network, and Y = (y1, y2, ..., y n ), y i z is the inactive output value of the i-th neuron in the network output layer, n is the number of neurons in the network layer of the image classification model, and z is the inactive output value. i The sample labels are in one-hot format: Z = (z1, z2, ..., z...). n The i-th element of ).

3. The method of claim 1, wherein the method further comprises: determining a plurality of L2NU activation functions; and determining a plurality of image classification models based on the plurality of L2NU activation functions. The S105 specifically comprises: training the ResNet-50 image classification model in step S104 based on the CIFAR-10 and CIFAR-100 image classification training data sets using the stochastic gradient descent method, setting the optimizer to the Momentum optimizer, setting the momentum coefficient to 0.9, setting the data batch size to 100, and using a learning rate of 0.1 for 100 rounds of training, and then using a learning rate of 0.01 for 10 rounds of parameter fine-tuning to obtain an image classification model based on the L2NU activation function.

4. The method of claim 1, wherein the method further comprises: determining a plurality of L2NU activation functions; and determining a plurality of image classification models based on the plurality of L2NU activation functions. After step S105, the method further comprises: S106: performance verification of the image classification model based on the L2NU activation function.

5. The method of claim 4, wherein the method further comprises: determining a plurality of L2NU activation functions; and determining a plurality of image classification models based on the plurality of L2NU activation functions. The S106 specifically comprises: training different layer FCNs using Sigmoid activation function, Tanh activation function, ReLU activation function and L2NU activation function respectively, and recording the optimal accuracy of the FCN network after 3000 rounds of training on the CIFAR-10 image classification training data set using a learning rate of 0.

01.

6. The method of claim 5, wherein the method further comprises: The S106 further comprises: learning the CIFAR-100 image classification training data set using VGG-11, VGG-13, VGG-16 and ResNet convolutional neural network models respectively, replacing the ReLU activation function of the convolutional layer and the fully connected layer in the original structure of each convolutional neural network with the L2NU activation function, to compare the performance difference between the L2NU activation function and the ReLU activation function, and replacing the Softmax function of the output layer of each convolutional neural network with the L2NUp function, to compare the performance difference between the L2NUp function and the Softmax function.

7. An image classification method characterized by: The method comprises the following steps: S201: obtaining an image classification sample to be classified; S202: inputting the image classification sample into the image classification model based on the L2NU activation function, which is trained in advance based on the method of any one of claims 1-6. S203: output the classification result of the image classification model based on the L2NU activation function.

8. An image classification model generation device based on an L2NU activation function, characterized by: The device comprises: a model modification module, configured to replace ReLU activation functions in original structures of all residual modules in a ResNet-50 image classification model with L2NU activation functions, remove all BN layers in the ResNet-50 image classification model, and set an activation function of an output layer of the ResNet-50 image classification model to ； wherein x represents an activation pre-mode of a network layer; the model modification module is further configured to initialize each weight matrix of a neural network of the ResNet-50 image classification model to a standard normal distribution, and set each layer bias of the neural network to a 0 vector; a mathematical definition of the L2NU activation function is as follows: ； wherein n is the number of neurons in a network layer of the image classification model, X is the pre-activation pattern of the network layer, and X = (x1, x2, …, xn) n ), xi is the output of each neuron, and i ∈ {1, 2, …, n}; i ); a training module, configured to train the modified ResNet-50 image classification model based on an image classification training data set; a determination module, configured to train the modified ResNet-50 image classification model into an image classification model based on the L2NU activation function by the method in any one of claims 1-6.

9. An image classification apparatus characterized by comprising: The device comprises: an acquisition module, configured to acquire an image classification sample collected by an image acquisition device; a classification module, configured to input the image classification sample into an image classification model based on the L2NU activation function, the image classification model based on the L2NU activation function being trained in advance based on the method in any one of claims 1-6, and output a classification result of the image classification model based on the L2NU activation function.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method in any one of claims 1-7.

Images