An activation learning method, system and picture classification method for a picture classification artificial neural network

Abstract

The application relates to an activation learning method, system and picture classification method of an artificial neural network for picture classification, which comprises the following steps: constructing an artificial neural network while taking data and category labels or multi-modalities as inputs; training the artificial neural network through an unsupervised local learning method, so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of input samples; inputting data to be identified into the trained artificial neural network and optimizing missing category labels so that the output activation intensity is maximum, thereby obtaining a data classification result; or inputting category labels into the trained artificial neural network with random noise and optimizing missing data parts so that the output activation intensity is maximum, thereby realizing data generation. The application can better learn the statistical probability distribution of input sample data, unifies the framework of supervised learning, unsupervised learning and a generation model, and has strong practicability.

Description

TECHNICAL FIELD

[0001] The present application relates to an activation learning method, system and picture classification method of an artificial neural network for picture classification, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] At present, artificial neural networks mainly rely on back propagation technology for network training and learning, and have achieved great success in the fields of vision, natural language processing, etc. The back propagation technology calculates the gradient of the loss function to each parameter (neuron connection weight) in the neural network in a reverse manner, modifies the parameters through an optimization method, and realizes the minimization of the loss function. However, the neural network based on back propagation training can usually only face specific learning tasks, and depends on a large amount of training data, and has certain security problems for adversarial attacks.

[0003] A series of problems of the artificial neural network based on the back propagation technology, including: (1) relying on a large amount of training sample data, the main reason is that the back propagation technology is to minimize the loss function through an optimization method for network training, and cannot obtain enough local features from very few samples; (2) the neural network trained by the back propagation method is usually for a specific task, and cannot support general artificial intelligence, because the loss function for training the network is usually determined according to a specific task; (3) the neural network or deep learning model based on the back propagation method lacks sufficient security for adversarial attacks, for example, when we add noise to a picture of a certain person according to a certain rule, the neural network will completely identify it as another person, which brings security risks to the application of deep learning. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides an activation learning method of an artificial neural network for picture classification;

[0005] The application provides a method for learning and reasoning of a neural network overturning back propagation, namely activation learning.

[0006] The activation learning method of the application is a practical method for learning and reasoning of a neural network overturning back propagation, can obtain learning effect better than that of a back propagation network on a small amount of sample data, has better robustness to an adversarial attack and external interference, and can be used as a universal model suitable for various learning tasks.

[0007] The application further provides a classification and recognition method and a data generation method based on activation learning of an artificial neural network.

[0008] In conclusion, the application relates to an activation learning method, system and application of an artificial neural network, comprising the following steps: constructing an artificial neural network, inputting data and category labels or multi-modalities into the artificial neural network; training the artificial neural network by using an unsupervised local learning method, so that the output activation strength of the last layer of neurons of the artificial neural network reflects the typicality of input samples; inputting data to be recognized into the trained artificial neural network and optimizing missing category labels so that the output activation strength is maximum, to obtain a data classification result; or inputting category labels into the trained artificial neural network with random noise and optimizing missing data so that the output activation strength is maximum, to realize data generation. The application can better learn the statistical probability distribution of input sample data, unify the framework of supervised learning, unsupervised learning and a generation model, and has strong practicability.

[0009] The technical scheme of the application is as follows:

[0010] An activation learning method of an artificial neural network for picture classification, comprising:

[0011] constructing an activation learning artificial neural network;

[0012] training the constructed artificial neural network by a local learning method, comprising: training the connection weights of each neuron in the artificial neural network, so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of the input sample, that is, the more typical input sample with higher statistical probability has stronger activation of the output of the whole artificial neural network;

[0013] performing inference on the trained artificial neural network, comprising: given an arbitrary known part of the input, calculating or optimizing the remaining unknown part of the input, so that the output activation intensity of the whole artificial neural network is the maximum, where the output activation intensity is the L2 norm of the output.

[0014] According to the application, preferably, the constructed activation learning artificial neural network is a multi-layer neural network, each layer containing a linear transformation function and a nonlinear activation function; the parameters w of each layer are represented by a matrix, describing the connection weights between neurons of two layers; given the input data x of a certain layer, the output of the layer is f(w T x), f is a nonlinear activation function.

[0015] According to the application, preferably, the nonlinear activation function f of each layer of the constructed activation learning artificial neural network adopts a nonlinear function that keeps the L2 norm of the input and output unchanged, that is, for any input y, ‖f(y)‖=‖y‖;

[0016] Further preferably, the activation function adopts an absolute value function.

[0017] According to the application, preferably, the input data of the artificial neural network includes unlabeled data, data and labels, and multi-modal data.

[0018] According to the application, preferably, the input data is normalized before being input into the artificial neural network, that is, it is scaled in proportion so that the Euclidean length is 1 or a constant; for multi-modal data, each modality is normalized separately.

[0019] According to the application, preferably, the connection weights of each neuron in the artificial neural network are trained by an unsupervised local learning method, and the training method is as shown in formula ( ) :

[0020] △w ij = ηy j (x i - ∑k y k w ik ) ( )

[0021] Mode( In ), w ij Δw refers to the connection weight between neuron i and neuron j in a higher layer of an artificial neural network. ij It is the change in the connection weights, η is the learning rate, and x is the learning rate. i It is the input from neuron i during training, y j = ∑ j x i w ij It is the total weighted input of neuron j, ∑ k y k w ik This refers to the decay term from the feedback, where k iterates through all neurons in the current layer. When training neurons in a certain layer, given a training sample, the network can calculate the input x and y = w of this layer through a feedforward approach. T x, and according to the formula ( Calculate the modification amount △w of the parameters in this layer. ij， Finally, w ij Replace with w ij +△w ij .

[0022] According to a preferred embodiment of the present invention, each layer of the artificial neural network is a fully connected layer, a locally connected layer, or a convolutional layer. During the training of the convolutional layer, the average value Δw of the modification amount of the connection weights at different locations is calculated. ij And update △w based on this average value. ij Update w to w+△w ij .

[0023] Mode( ) of y j It can also be represented as the output of neuron j, in which case y j = f(∑ j x i w ij ), where f(.) is a non-linear activation function, such as the ReLU function. In this case, the activation function of the neural network does not need to maintain the input and output l2 norms invariant.

[0024] According to a preferred embodiment of the present invention, the connection weights of each neuron in the artificial neural network are trained using a stochastic gradient descent optimization method, including: minimizing ||x-ww|| T x‖ 2where x is a vector representation of input data of a neuron in a certain layer, w is a matrix representation of connection weights of the neuron in the certain layer, w T is a transpose of w.

[0025] According to the application, preferably, the connection weights of each neuron in the artificial neural network are trained by using a stochastic gradient descent optimization method, including: minimizing the expectation of ‖x-g(w,w T x)‖ 2 where x is a vector representation of input data of a neuron in a certain layer, w is a matrix representation of connection weights of the neuron in the certain layer, w T is a transpose of w, g(w,w T x) is a function of w and w T x.

[0026] An activation learning system of an artificial neural network for image classification, for realizing classification recognition and data generation, including:

[0027] An artificial neural network construction module, configured to: construct an activation learning artificial neural network;

[0028] An artificial neural network training module, configured to: train the constructed artificial neural network by using a local learning method, including: training the connection weights of each neuron in the artificial neural network, so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of input samples, i.e. the more typical input samples with higher statistical probability have stronger activation of the output of the whole artificial neural network;

[0029] A classification recognition and data generation module, configured to: input data to be recognized into the trained artificial neural network and optimize the missing class label input to maximize the output activation intensity of the network, to obtain a data classification result; or input a class label into the trained artificial neural network with random noise and optimize the missing data input to maximize the output activation intensity of the network, to realize data generation.

[0030] An application of an activation learning method of an artificial neural network, for classification recognition, including:

[0031] Input data to be recognized into the trained artificial neural network and optimize the missing class label input to maximize the output activation intensity of the artificial neural network, to obtain a data classification result.

[0032] According to the application, preferably, in the process of training the constructed artificial neural network, the classes are encoded and input into the artificial neural network together with data for activation learning training;

[0033] Further preferably, the class coding method is to express the class i as a matrix, the i-th row is 1, and the other rows are 0.

[0034] According to the application, in the classification recognition process, the class of the data to be recognized is inferred according to the data to be recognized, so that the maximum output activation intensity of the artificial neural network is obtained; specifically, assuming that an input data is a, and the label coding is z, the classification recognition process is to find the best label z for the given data (i.e., the data to be recognized) a, so that the output activation intensity ‖y(a,z)‖ of the artificial neural network is the maximum. 2

[0035] According to the application, the classification recognition ability of the activation learning is improved through feedback information, including: adjusting the learning rate through the feedback signal of the classification recognition result, the learning rate being R(F)η, wherein R(F) is a feedback signal adjustment coefficient; if the input data is correctly classified, R(F) is set to 1, and if the input data is incorrectly classified, R(F) is set to a value greater than 1 to strengthen the intensity of this type of input data in learning.

[0036] According to the application, the classification recognition ability of the activation learning is improved through feedback-based counter-learning, including: taking the input data and its correct label as positive samples for learning, and taking the input data and the incorrect label that makes the output activation intensity the strongest as negative samples for counter-learning.

[0037] Further preferably, the output activation intensity g1 of the given positive sample and the modification amount △w1 of the connection weight are calculated, the output activation intensity g2 of the given negative sample and the modification amount △w2 of the connection weight are calculated, the final modification amount △w of the connection weight is △w1 + α(△w1-△w2), and the connection weight w is updated to w+△w. Here, α depends on g1 and g2, for example, if g1-g2<0.1, then α=1, otherwise α=0.

[0038] An application of an activation learning method of an artificial neural network, for generating data, including: generating data through a trained artificial neural network; specifically:

[0039] The type label coding of the data to be generated is expressed as a, the generated data is expressed as z, and the optimization method is used to solve z to maximize ‖y(a,z)‖ 2 -β|z|1, so that ‖z‖=1; ‖y(a,z)‖ 2 is the output activation intensity of the artificial neural network, 0≦β≦0.1, |z|1 is the L1 norm of the generated data, and β|z|1 is an adjustment term for improving the quality of the generated data.

[0040] ​Further preferably, v is optimized to maximize ‖y(a,z)‖ 2 -β|z|1, where z=v / ‖v‖.

[0041] Further preferably, a v is randomly initialized, the gradient g of the optimization objective with respect to v is calculated at each step, a step size γ is determined, and v is iterated as v+γg, and through continuous iteration, until ‖g‖ is sufficiently small, and finally v converges to a local optimum.

[0042] According to the present application, preferably, an iterative method is used to solve the generated data z, specifically, a z is randomly initialized first, and then iteration is performed, and at each step, ‖y(a,z)‖ 2 -β|z|1 is calculated, and the partial derivative of ‖y(a,z)‖

[0043] According to the present application, preferably, random noise is added to the trained artificial neural network at each time of generating data, including randomly adjusting the output of part or all neurons, specifically, multiplying the neuron output by a coefficient of 1+δ, and δ is a small noise independent of each neuron.

[0044] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the activation learning method of the artificial neural network for picture classification when executing the computer program.

[0045] A computer readable storage medium having a computer program stored thereon, the computer program implementing the steps of the activation learning method of the artificial neural network for picture classification when executed by a processor.

[0046] The present application has the following beneficial effects:

[0047] 1. The activation learning technology proposed in the present application overturns the back propagation technology of neural network training, and can better learn the statistical probability distribution of input sample data.

[0048] 2. The activation learning proposed in the present application can significantly outperform the model trained based on back propagation in the learning task of classification and recognition in the small sample scenario. Figure 4If each digit only has 10 training pictures, experiments show that activation learning can also have an error recognition rate of about 15%.

[0049] 3、The activation learning proposed in the application has better security against adversarial attacks and stronger robustness against external interference or random covering. Taking the MNIST dataset as an example, the activation can still obtain a classification error rate of not more than 8% under the condition of covering 1 / 3. If some straight lines are randomly added on the picture to interfere, the corresponding digit can also be recognized with relatively high accuracy. It shows that the activation learning has good generalization ability and can tolerate interference that has not been learned.

[0050] 4、The activation learning proposed in the application can be used as a generation model to generate high-quality pictures. Compared with most existing deep learning generation models, the pictures generated by the activation learning can be more consistent with the true probability distribution of the sample and have higher representativeness. Another advantage is that the same trained network can be used for both classification and data generation tasks, which is not possessed by the back propagation deep learning model.

[0051] 5、The activation learning local connection layer proposed in the application can effectively improve the stability of model training and the prediction accuracy of the model. Taking the CIFAR-10 dataset as an example, the dataset contains 60,000 color pictures of 10 categories of objects. If a neural network with two fully connected layers, each containing 32x32x3 neurons, is trained by activation learning, a classification error rate of 43.59% can be obtained; if the first fully connected layer is replaced by a local connection layer with a maximum distance of 5, the classification error rate can be reduced to 41.63% without any data enhancement. The network structure is as shown in Figure 6 At the same time, the performance of the neural network is improved, the parameters of the entire network can be reduced, and the inference calculation time can also be reduced. In the biologically inspired network learning method, the best method so far is the method proposed by Krotov and Hopfield in 2019, which has a classification error rate of 49.25% on CIFAR-10.

[0052] 6、The activation learning proposed in the application can better support general task learning. The back propagation technology in deep learning currently modifies network parameters by optimizing the loss function of the network, and the determination of the loss function is related to the specific task, so the back propagation learning is usually only applicable to specific learning tasks. Building a learning model for general tasks is an important direction of artificial intelligence. The training process of activation learning is mainly to learn the statistical probability distribution of the training sample, which is independent of the specific learning task, so it can better support the establishment of a general task learning model.

[0053] 7. The activation learning proposed in this invention can better support multimodal learning and establish connections between different content. For example, an activation learning network can be built to support three modal inputs: images, text, and audio. Figure 6 As shown, during training, any subset of modalities can be provided for learning. By providing examples of image and text modalities, a connection between images and text can be established; by providing examples of images and sound, a connection between images and sound can be established. For a well-trained neural network, even if it has never simultaneously learned text and sound modalities, it can still infer text from sound modalities or sound from text modalities. This ability to associate content plays a crucial role in human intelligence.

[0054] 8. The activation learning proposed in this invention can use fewer network nodes compared to deep learning models based on backpropagation, thereby reducing the computational load of the model during training and inference. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of a pre-constructed artificial neural network trained using an activation learning method.

[0056] Figure 2 A schematic diagram illustrating the training of an artificial neural network using an unsupervised localized learning method;

[0057] Figure 3 A diagram illustrating image data classification and recognition;

[0058] Figure 4 This is a schematic diagram of the structure of the artificial neural network used for the MNIST dataset in an embodiment of the present invention;

[0059] Figure 5 The image is a handwritten character generated by the activation learning neural network in this invention example;

[0060] Figure 6 This is a schematic diagram of the structure of the artificial neural network used for the CIFAR-10 dataset in an embodiment of the present invention;

[0061] Figure 7 A schematic diagram of an artificial neural network trained using activation learning methods when inputting images, text, and sound modalities. Detailed Implementation

[0062] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0063] Example 1

[0064] An activation learning method for artificial neural networks used for image classification includes:

[0065] Construct an artificial neural network for activation learning;

[0066] Train the constructed artificial neural network by a local learning method, including: training the connection weights of each neuron in the artificial neural network, so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of the input sample, that is, the more typical input sample with higher statistical probability has stronger activation of the output of the whole artificial neural network;

[0067] Conduct inference on the trained artificial neural network, including: given an arbitrary known part of the input, calculate or optimize the remaining unknown part of the input, so that the output activation intensity of the whole artificial neural network is the maximum, where the output activation intensity is the L2 norm of the output.

[0068] The constructed artificial neural network for activation learning is a multi-layer neural network, each layer containing a linear transformation function and a nonlinear activation function; The parameters w of each layer are represented by a matrix, which describes the connection weights between two layers of neurons; Given the input data x of a certain layer, the output of this layer is f(w T x), f is a nonlinear activation function. The input of the network can be various types of data, not limited to unlabeled data, or data label pairs, or multi-modal data. Through activation learning, information correlation between each sub-block of input can be established in an unsupervised manner.

[0069] The nonlinear activation function f of each layer of the constructed artificial neural network for activation learning adopts a nonlinear function that keeps the l2 norm of the input and output unchanged, that is, for any input y, ‖f(y)‖=‖y‖; The activation function can adopt but is not limited to the absolute value function.

[0070] The input data of the artificial neural network includes but is not limited to unlabeled data, data and labels, and multi-modal data. Figure 7 A schematic diagram of training the constructed artificial neural network by activation learning method for picture, text and sound three modal inputs.

[0071] Normalization processing is performed before inputting the input data into the artificial neural network, that is: scaling proportionally to make the Euclidean length (L2 norm) 1 or a constant; For multi-modal data, normalization processing is performed separately for each modality.

[0072] The well-constructed artificial neural network is trained by a local learning method, including: each layer of neurons is trained by a forward unsupervised local learning method, and each layer of unsupervised features is reconstructed from data in the previous layer by the learning method; the activation strength of each layer of neurons tends to decrease with the increase of the number of network layers; and the output activation strength (l2 norm of output) of the last layer of neurons in the artificial neural network approximately estimates the relative probability size or typicality of input data, as shown in Figure 1 The trained activation learning neural network can well learn and estimate the probability distribution of training samples.

[0073] The connection weight of each layer of neurons in the artificial neural network is trained by an unsupervised local learning method, and the training method is as shown in formula ( ).

[0074] △w ij = ηy j (x i - ∑ k y k w ik ) ( )

[0075] In formula ( ), w ij is the connection weight between neuron i and neuron j in the higher layer in the artificial neural network, △w ij is the change amount of the connection weight, η is a learning rate, x i is an input from neuron i during training, y j = ∑ j x i w ij is the total weighted input of neuron j, ∑ k y k w ik is a decay term from feedback, and k traverses all neurons in the layer; as shown in Figure 2 During training of a layer of neurons, a training sample is given, the network can calculate the input x and y = w T x of the layer by a forward feeding manner, and calculate the modification amount △w ij， of the parameters in the layer according to formula ( ), and finally replace w ij with w ij + △w ij . The training process can be performed in a small batch manner, in each training step, a small batch of samples are randomly selected, and the average value of △w ij is calculated to update w ijThe whole training process ends after a given number of steps are completed, and this training method has strong anti-overfitting ability, and can not use the early stopping method commonly used in deep learning to reduce overfitting.

[0076] The core of the above unsupervised localization learning method is to introduce a decay term -∑kykwik from feedback. This decay term can realize the competition of different synaptic connections in the same neuron and the competition of the activation of neurons in the same layer, so as to achieve the purpose of neural network training and learning. It can be proved that through this method, the data of each layer can be decomposed into non-orthogonal principal component features, and the features of each layer can reconstruct the data of the previous layer. Moreover, the more typical the input data is, the higher the average activation strength of the output is, which can well support activation learning. In addition, it also has certain advantages in small sample feature extraction and security of adversarial attack compared with the back propagation training method.

[0077] Each layer of the artificial neural network can be but is not limited to a fully connected layer, a local connection layer or a convolutional layer. In the local connection layer and the convolutional layer, only the neurons of adjacent layers with close distances are connected. The difference between the local connection layer and the convolutional layer is that the connection weights of the neurons at different positions in the local connection layer are different, and there is no parameter sharing, while the convolutional layer is based on parameter sharing, that is, the parameters at different positions are the same. Based on the unsupervised local learning method, the training of the local connection layer is the same as that of the fully connected layer, except that the connection of the neurons is less. In the training of the convolutional layer, the average value of the connection weight modification amount of different positions is calculated as △w ij , and the average value is used to update △w ij , and w is updated to w+△w ij .

[0078] The formula ( ) y j can also be expressed as the output of neuron j, and at this time y j = f(∑ j x i w ij ), wherein f(.) is a nonlinear activation function, such as a RELU function. In this case, the activation function of the neural network does not need to keep the input and output l2 norms unchanged.

[0079] Embodiment 2

[0080] An activation learning method of an artificial neural network for image classification according to Embodiment 1, the difference is that:

[0081] An optimization method such as stochastic gradient descent is used to train the connection weights of each layer of neurons in the artificial neural network, including: minimizing‖x-ww T x‖ 2expectation, x is a vector representation of input data of a certain layer of neurons, w is a matrix representation of connection weights of the layer of neurons, w T is the transpose of w. The stochastic gradient descent method is to first select an initial randomized w value, then calculate the gradient of the optimization objective with respect to w according to a randomly input sample, and adjust w in the direction of the gradient descent according to a certain step size, and finally w converges to a local optimum.

[0082] The connection weights of each layer of neurons in the artificial neural network are trained by using an optimization method such as stochastic gradient descent, including: minimizing the expectation of ‖x-g(w,w T x)‖ 2 expectation, x is a vector representation of input data of a certain layer of neurons, w is a matrix representation of connection weights of the layer of neurons, w T is the transpose of w, g(w,w T x) is a function of w and w T x.

[0083] The inference process of activation learning is to find the remaining unknown part of the input so that the output activation intensity of the entire artificial neural network is the largest, given the known part of the input. One advantage of such an architecture is that it can unify supervised learning, unsupervised learning, semi-supervised learning and generative models. For example, for supervised learning, the data can be submitted to the network together with the class label for training, and then in the inference process if only the data is provided, the corresponding class label can be found to make the output activation intensity of the entire network the largest, thus completing the classification task. For training data without labels, it can also be provided to the network to learn unsupervised features (freeze other parts of the network), and finally unsupervised learning and supervised learning can be combined to realize semi-supervised learning, that is, the network can use a large amount of unlabeled data and a small amount of labeled data for learning and training.

[0084] Embodiment 3

[0085] An artificial neural network activation learning system for realizing classification recognition and data generation, comprising:

[0086] An artificial neural network construction module configured to construct an artificial neural network for activation learning;

[0087] An artificial neural network training module configured to train the constructed artificial neural network by a local learning method, including: training the connection weights of each layer of neurons in the artificial neural network so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of the input sample, that is, the more typical input sample with higher statistical probability has stronger activation of the output of the entire artificial neural network;

[0088] The classification and data generation module is configured to: input the data to be identified into a trained artificial neural network and optimize the missing category label input to maximize the network's output activation strength, thereby obtaining the data classification result; or, input the category label into a trained artificial neural network with added random noise and optimize the missing data input to maximize the network's output activation strength, thereby generating the data.

[0089] The activation learning described above, based on the same pre-trained neural network, can be used for classification, unsupervised feature extraction, multimodal learning, and can also serve as a generative model to create data. In contrast, in backpropagation-based deep learning, the classification model and the generative model are completely independent neural networks.

[0090] Example 4

[0091] The application of the activation learning method of the artificial neural network described in Example 1 or 2 for classification and recognition includes:

[0092] The data to be identified is input into a trained artificial neural network, and the missing category labels are optimized to maximize the activation intensity of the output of the artificial neural network, thus obtaining the data classification result.

[0093] During the training of the constructed artificial neural network, the categories are encoded and input along with the data into the artificial neural network for activation learning training;

[0094] The method of category encoding is to represent category i as a matrix, with the i-th row being 1 and all other rows being 0.

[0095] In the classification and recognition process, the category of the data to be identified is inferred based on the data to be identified, so as to maximize the activation intensity of the entire artificial neural network output. Specifically, assuming an input data is 'a' and the label encoding is 'z', the classification and recognition process is to find the best label z given the data to be identified, i.e., the data 'a', so as to maximize the activation intensity of the entire artificial neural network output, ‖y(a,z)‖. 2 It is the largest, such as Figure 3 As shown, y(a,z) is the output of the entire artificial neural network.

[0096] Calculating the class label causes the output activation intensity ‖y(a,z)‖ of the entire artificial neural network. 2 It is the largest, and the specific implementation methods include:

[0097] Enumerate all possibilities of z and find the output activation intensity ‖y(a,z)‖ 2 The largest corresponding z;

[0098] Alternatively, gradient descent can be used to maximize the output activation intensity ‖y(a,z)‖2 , find the local optimal z;

[0099] Or, using an iterative way, each step using the output activation strength ‖y(a,z)‖ 2 The partial derivative of z is updated to z, through multiple iterations until convergence to the local optimal z. The advantage of using activation learning for classification tasks is that it can achieve significantly better accuracy than backpropagation training models with fewer training samples, and this method can still maintain high classification accuracy in the presence of external interference or coverage.

[0100] Feedback information is used to improve the classification ability of activation learning, including: adjusting the learning rate through the feedback signal of the classification result, the learning rate is R(F)η, where R(F) is the feedback signal adjustment coefficient; If the input data is correctly classified, set R(F)=1, if the input data is incorrectly classified, set R(F) to a value greater than 1, usually R(F) is between 1 and 20, for example R(F)=10, to strengthen the intensity of such input data in learning. By this method and choosing the appropriate R(F), the accuracy of activation learning in various classification tasks can be significantly improved.

[0101] Feedback-based anti-learning is used to improve the classification ability of activation learning, including: taking the input data and its correct label as positive samples for learning (the learning rate of the local learning method is positive), and taking the input data and the error label that makes the output activation strength the strongest as negative samples for anti-learning (the learning rate of the local learning method is negative). Through this forward and backward learning, data with correct classification labels can obtain higher output activation strength than data with incorrect classification labels, improving the accuracy of classification tasks, and achieving a recognition error rate comparable to humans (2% to 2.5%) in handwritten digit recognition.

[0102] Given the positive sample (data and correct label) to calculate its output activation strength g1 and the modification amount △w1 of the connection weight, given the negative sample (data and error label) to calculate its output activation strength g2 and the modification amount △w2 of the connection weight, and the final modification amount △w of the connection weight is △w1 + α(△w1-△w2), update the connection weight w to w+△w. Here α depends on g1 and g2, for example, if g1-g2<0.1, then α=1, otherwise α=0.

[0103] Experiments are conducted using MNIST data, which contains 60,000 handwritten digit images, each image is 28x28 pixels in black and white. Training Figure 4The artificial neural network contains two layers, each layer has 1064 neurons, and uses the absolute value function as the activation function. The input of the artificial neural network includes normalized pictures and normalized category label encoding, each category is encoded into a 10x28 matrix, and the i-th row of the matrix corresponding to category i is 1 and the other rows are 0. Based on 60,000 labeled pictures, through active learning without using feedback, the artificial neural network can achieve a recognition error rate of 3.37%; if the learning rate is adjusted using feedback R(F) = 10, through active learning this network can achieve a recognition error rate of 2.72%; if the negative sample is calculated through feedback signal, through active learning this network can achieve a recognition error rate of 2.28%, which shows that through feedback information, the learning ability of the network can be improved, and the recognition ability of the human on MNIST (error rate between 2% and 2.5%) can be achieved. Continue to reduce the number of labeled pictures, based on 600 labeled pictures, through active learning without using feedback, the artificial neural network achieves a recognition error rate of 9.74%, while the back propagation algorithm can only achieve an error recognition rate of 25.5%, which shows that active learning can achieve significantly better learning ability than back propagation algorithm in the case of small sample. If there are only 10 training pictures for each number, the experiment shows that active learning also has an error recognition rate of about 15%.

[0104] Continue to take CIFAR-10 data as an example for experiment, this data set contains 60,000 color pictures, belonging to 10 categories of objects, and the size of each picture is 32x32x3. The artificial neural network structure is as shown in Figure 6 The artificial neural network contains two layers of neurons. The first layer of neurons uses local connection layer, each neuron can only connect to the input layer of the picture with a horizontal or vertical distance of not more than 5, and each neuron can connect to all the input layers of label encoding. The number of first layer neurons is 32x32x3. The second layer of neurons uses full connection layer, and the number of neurons is 32x32x3. The input of the artificial neural network includes 32x32x3 pictures and 10x10 category encoding. After normalization, the L2 norm of the picture is 2, and the L2 norm of the category encoding is 1. The classification error rate of active learning is 41.63% without any data enhancement. If data enhancement is used and the number of neurons in the first layer is increased to 32x32x16, the error recognition rate of active learning on CIFAR-10 can be further reduced to 37.52%. Among the biologically inspired network learning methods, the best method so far is the method proposed by Krotov and Hopfield in 2019, which has a classification error rate of 49.25% on CIFAR-10.

[0105] Example 5

[0106] The application of the activation learning method of the artificial neural network described in embodiment 1 or 2 is used to generate data, including: generating data by the trained artificial neural network; in particular:

[0107] The type label code of the data to be generated is represented as a, the generated data is represented as z, and z is solved by an optimization method to maximize ‖y(a,z)‖ 2 -β|z|1, so that ‖z‖=1 (or a given constant); ‖y(a,z)‖ 2 is the output activation strength of the artificial neural network, 0≦β≦0.1 is a small constant, and |z|1 is the L1 norm of the generated data. β|z|1 is an adjustment term for improving the quality of the generated data.

[0108] z can be replaced by v / ‖v‖, and the above non-convex constrained optimization problem becomes an unconstrained optimization problem, that is, v is optimized to maximize ‖y(a,z)‖ 2 -β|z|1, where z=v / ‖v‖. In order to ensure the convergence of v, a penalty term about ‖v‖ can be added to the optimization objective, which can use but is not limited to (‖v‖-1) 2 .

[0109] The gradient descent method is used to solve the above unconstrained optimization problem, including: randomly initializing a v, for example, v

[0110] It can be initialized as a uniformly distributed data; the gradient g of the optimization objective with respect to v is calculated at each step, a step size γ is determined, and v is iterated as v+γg. Through continuous iteration, when ‖g‖ is small enough, v converges to a local optimal point.

[0111] An iterative method is used to solve the generated data z, in particular: first, a z is randomly initialized, then iterated, and at each step, the partial derivative of ‖y(a,z)‖ 2 -β|z|1 with respect to z is calculated and normalized to update z. After multiple iterations, z converges to a local optimal z.

[0112] In order to improve the diversity of the generated data, random noise is added to the trained artificial neural network during each generation of data, including but not limited to randomly adjusting the output of part or all neurons, in particular: multiplying the neuron output by a coefficient of 1+δ, where δ is a small noise independent of each neuron. For example, Gaussian noise, uniform noise, etc.

[0113] MNIST data is used as an example for data generation experiments. Here, the trained artificial neural network in embodiment 5 is used for image classification and recognition, and the number of training samples is 60000. No feedback information is used in the training process.Figure 5 The handwritten pictures randomly generated by the method of the present application using gradient descent are demonstrated based on this trained artificial neural network.

[0114] Embodiment 6

[0115] A computer device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the activation learning method of the artificial neural network for picture classification according to any one of embodiments 1 or 2 when executing the computer program.

[0116] Embodiment 7

[0117] A computer readable storage medium, having stored thereon a computer program, the computer program implementing the steps of the activation learning method of the artificial neural network for picture classification according to any one of embodiments 1 or 2 when executed by a processor.

Claims

1. An activation learning method for artificial neural networks used for image classification and recognition, characterized in that, include: Construct artificial neural networks for activation learning; The constructed artificial neural network is trained using local learning methods, including: training the connection weights of each neuron in the artificial neural network so that the output activation intensity of the last neuron in the artificial neural network reflects the typicality of the input sample, that is, the more typical the input sample is and the higher the statistical probability, the stronger the activation of the entire artificial neural network output. Inference is performed on a trained artificial neural network, including: given any known part of the input, calculating or optimizing the remaining unknown part of the input so that the output activation strength of the entire artificial neural network is maximized, where the output activation strength is the L2 norm of the output. The data to be identified is input into a trained artificial neural network, and the missing category label input is optimized to maximize the activation intensity of the output of the artificial neural network, thus obtaining the data classification result. The data to be identified includes images; The constructed activation learning artificial neural network is a multi-layered neural network, with each layer containing a linear transformation function and a non-linear activation function; the parameters w of each layer are represented by a matrix, describing the connection weights between neurons in two layers; given the input data x of a certain layer, the output of this layer is f(w). T x), f is a nonlinear activation function, w T It is the transpose of w; The nonlinear activation function f of each layer of the constructed activation learning artificial neural network adopts a nonlinear function that keeps the input and output l2 norm unchanged, that is, for any input y, it satisfies ‖f(y)‖=‖y‖; The activation function is an absolute value function; The connection weights of each neuron in the artificial neural network are trained using an unsupervised local learning method, as shown in the equation (). As shown in the image: △w ij = ηy j (x i - ∑ k y k w ik ) ( ) Mode( In ), w ij Δw refers to the connection weight between neuron i and neuron j in a higher layer of an artificial neural network. ij It is the change in the connection weights, η is the learning rate, and x is the learning rate. i It is the input from neuron i during training, y j = ∑ j x i w ij It is the total weighted input of neuron j or y j = f(∑ j x i w ij ) is the output of neuron j, ∑ k y k w ik This refers to the decay term derived from feedback; when training a neuron in a certain layer, given a training sample, the network calculates the input x and y = w of this layer through a feedforward method. T x or y = f(w) T x), and according to the formula ( Calculate the modification amount △w of the parameters in this layer. ij Finally, w ij Replace with w ij +△w ij .

2. The activation learning method for an artificial neural network for image classification according to claim 1, characterized in that, The input data of the artificial neural network includes unlabeled data, labeled data, and multimodal data; Before inputting the data into the artificial neural network, normalization is performed, namely: scaling proportionally to make the Euclidean length a constant; for multimodal data, normalization is performed separately for each modality.

3. The activation learning method for an artificial neural network for image classification according to claim 1, characterized in that, The connection weights of each neuron in the artificial neural network are trained using the stochastic gradient descent optimization method, including minimizing ||x - ww||. T x‖ 2 Expectations; Alternatively, the connection weights of each neuron in the artificial neural network can be trained using a stochastic gradient descent optimization method, including: minimizing ||x – g(w, w)|| T x)‖ 2 The expectation, g(w,w) T x) is w and w T A function of x.

4. An activation learning system for an artificial neural network for image classification, used to implement the activation learning method for an artificial neural network for image classification as described in any one of claims 1-3, for achieving image classification and recognition, characterized in that, include: The artificial neural network building module is configured to: build artificial neural networks for activation learning; The artificial neural network training module is configured to train the constructed artificial neural network through a local learning method, including training the connection weights of each layer of neurons in the artificial neural network so that the output activation intensity of the last layer of neurons in the artificial neural network reflects the typicality of the input sample, that is, the more typical the input sample with a higher statistical probability, the stronger the activation of the entire artificial neural network output. The classification and data generation module is configured to: input the data to be identified into a trained artificial neural network and optimize the missing category label input to maximize the network's output activation strength, thereby obtaining the data classification result; or, input the category label into a trained artificial neural network with added random noise and optimize the missing data input to maximize the network's output activation strength, thereby generating the data. The data to be identified includes images.

5. A method for image classification using the activation learning method of an artificial neural network for image classification as described in any one of claims 1-3, for image classification and recognition, characterized in that, include: The data to be identified is input into a trained artificial neural network, and the missing category label input is optimized to maximize the activation intensity of the output of the artificial neural network, thus obtaining the data classification result. During the training of the constructed artificial neural network, the categories are encoded and input along with the data into the artificial neural network for activation learning training. The method of category encoding is as follows: the category i is represented as a matrix, with the i-th row being 1 and all other rows being 0; In the classification and recognition process, the category of the data to be identified is inferred based on the data to be identified, so as to maximize the activation intensity of the entire artificial neural network output. Specifically, assuming an input data is 'a' and the label encoding is 'z', the classification and recognition process is to find the best label z given the data to be identified, i.e., the data 'a', so as to maximize the activation intensity of the entire artificial neural network output, ‖y(a,z)‖. 2 It is the largest, and y(a,z) is the output of the entire artificial neural network; The activation learning classification and recognition capabilities are enhanced by using feedback information, including: adjusting the learning rate based on the feedback signal of the classification and recognition results, where the learning rate is R(F)η, and R(F) is the coefficient for adjusting the feedback signal; if the input data is correctly classified, R(F) = 1 is set, and if the input data is misclassified, R(F) is set to a value larger than 1 to strengthen the learning of this type of input data. Improving activation learning classification and recognition capabilities through feedback-based reverse learning includes: learning input data and its correct label as positive samples, and learning input data and the incorrect label that makes the output activation strength the strongest as negative samples. Given a positive sample, calculate its output activation intensity g1 and the modification amount Δw1 of the connection weights. Given a negative sample, calculate its output activation intensity g2 and the modification amount Δw2 of the connection weights. The final modification amount Δw of the connection weights is Δw1 + α(Δw1-Δw2), and the connection weights w are updated to w+Δw.

Images