Application of an unsupervised local learning method based on artificial neural networks

Abstract

The application relates to an application of an unsupervised local learning method based on an artificial neural network, the artificial neural network being a multilayer neural network, wherein parameters of each layer are trained by the unsupervised local learning method. The neural network learning method provided by the application is unsupervised, does not need manual label marking, and can efficiently extract features from unlabeled data. The features extracted by the learning method are universal, can be used as inputs of various learning tasks, and can effectively improve the accuracy of various learning tasks. The neural network trained by the learning method can be used as a pre-training model of various tasks to improve the accuracy of various tasks. The learning method is different from a widely used back propagation method, is easier to realize on-chip training of the artificial neural network based on hardware, and supports the development of technologies such as optical AI chips.

Description

Technical Field

[0001] This invention relates to an application of an unsupervised local learning method based on artificial neural networks, belonging to the field of artificial intelligence technology. Background Technology

[0002] Currently, artificial neural networks primarily rely on backpropagation for training and learning, and have achieved significant success in fields such as vision and natural language processing. Backpropagation calculates the gradient of the loss function with respect to each parameter (neuron connection weights) in a reverse manner, and then modifies these parameters using optimization methods such as stochastic gradient descent to minimize the loss function. However, backpropagation differs significantly from the human brain's working mechanism. It struggles to obtain sufficient features from limited sample data, resulting in a high demand for training data in various image, speech, text, and time-series recognition, detection, segmentation, and prediction tasks. Its accuracy is low with small sample sizes, limiting the application of artificial intelligence in many scenarios, such as industrial applications. Furthermore, neural network systems trained using backpropagation face security issues. For example, adding adversarial noise to a given face image can easily lead to the machine misidentifying it as another person. Exploring practical neural network learning techniques and systems different from backpropagation is one of the core issues in the field of artificial intelligence. Summary of the Invention

[0003] This invention addresses a series of limitations of backpropagation technology by proposing a brain-inspired, practical artificial neural network learning method and system. It establishes a simple learning rule that can adjust and learn the parameters of the neural network in an unsupervised manner based on input data. It can be widely applied to various artificial intelligence application scenarios, including but not limited to image, speech, text, and time series classification, recognition, generation, detection, segmentation, and prediction tasks, thereby improving the accuracy of learning tasks.

[0004] This invention's technical solution can extract unsupervised features from large amounts of unlabeled data or construct general unsupervised pre-trained models, improving the accuracy of various artificial intelligence learning tasks under current backpropagation techniques. This invention's technical solution can also simultaneously use data and labels as input to a neural network for joint learning, extracting the correlation features between data and labels in an unsupervised manner, and establishing a new training and inference model for artificial neural networks. Compared to current mainstream backpropagation techniques, this invention's technical solution can learn various features and correlations in data from a small amount of sample data, exhibiting better robustness against adversarial attacks and external interference, opening a new window for the development of general-purpose task artificial intelligence.

[0005] This invention proposes a neural network training method completely different from backpropagation. It is forward-trained, fully local, and unsupervised, capable of obtaining a large amount of localized feature information from a small number of samples, and exhibits better security compared to the backpropagation method. Based on this unsupervised learning method, features can be extracted from large amounts of unlabeled data or general unsupervised pre-trained models can be built, improving the accuracy of various learning tasks.

[0006] The present invention also provides an application of the above-mentioned unsupervised local learning method based on artificial neural networks and a classification and recognition system based on the unsupervised local learning method based on artificial neural networks.

[0007] The technical solution of this invention is as follows:

[0008] A generalized unsupervised local learning method for artificial neural networks is proposed. The artificial neural network is a multi-layered network, comprising: the parameters of each layer are trained using an unsupervised local learning method, and the parameters w of each layer are represented by a matrix describing the connection weights between neurons in two layers; given the input data x of a certain layer, the output of that layer is f(w). T x), f is a non-linear activation function, and the training method for each layer of neurons is as shown in equation ( As shown in the image:

[0009] △w ij = ηy j (x i - ∑ k y k w ik ) ( )

[0010] Mode( In ), w ij Δw refers to the connection weight between neuron i in the next layer and neuron j in the current layer of an artificial neural network. ij It is the change in the connection weights, η is the learning rate, and x is the learning rate. i It is the input from neuron i during training, y j = ∑ j x i w ij It is the total weighted input of neuron j, ∑ k y k w ik This refers to the decay term from the feedback, where k traverses all neurons in this layer;

[0011] When training a layer of neurons, given a training sample, the input x and y = w of this layer are calculated using a feedforward method. T x, and according to the formula ( Calculate the modification amount △w of the parameters in this layer. ij， Finally, w ij Replace with w ij +△w ij .

[0012] Mode( ) of y j It can also be represented as the output of neuron j, in which case y j = f(∑ j x i w ij ), where f(.) is a nonlinear activation function, such as the ReLU function. In this case, equation ( The local learning method described can still effectively extract unsupervised features.

[0013] According to a preferred embodiment of the present invention, the formula ( ) can be represented in matrix and vector form, as shown in equation ( As shown in the image:

[0014] △w = ηy(x - wy) T ( )

[0015] Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x is the input vector of this layer, and y = w T x is the total weighted input, (x - wy) T It is the transpose of x-wy.

[0016] According to a preferred embodiment of the present invention, the formula ( ) can be extended to a more general form, such as equation ( As shown in the image:

[0017] △w = ηf(y) (x – g(w,y)) T ( )

[0018] Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x represents the input vector of this layer, and y = w. T x is the total weighted input, f(y) is a function of y, and g(w,y) is a function of w and y.

[0019] According to a preferred embodiment of the present invention, each layer in the artificial neural network may be, but is not limited to, a fully connected layer, a locally connected layer, or a convolutional layer. During the training of the convolutional layer, Δw is calculated at different positions. ij The average value, and update w based on that average value. ij .

[0020] According to a preferred embodiment of the present invention, an unsupervised local learning method trains a neural network of any number of layers from the bottom up; each layer extracts the main features from the data of the previous layer and transforms the data through a non-activation function; the trained artificial neural network realizes unsupervised feature extraction, and the higher the feature layer, the more complex the features.

[0021] According to a preferred embodiment of the present invention, the non-activation function in the artificial neural network is, but is not limited to, a square function, and the output features of each layer can be subjected to layer normalization processing.

[0022] According to a preferred embodiment of the present invention, the artificial neural network can be trained layer by layer or all layers can be trained together. If each layer is trained layer by layer, it starts from the bottom layer and trains layer by layer upwards. If all layers are trained together, the parameters of all layers are learned and adjusted simultaneously during the training process.

[0023] Applications of an unsupervised local learning method based on artificial neural networks include classification and recognition, industrial inspection, medical image diagnosis, image segmentation, temporal prediction, text translation, and speech recognition.

[0024] A classification and recognition system based on an unsupervised local learning method using artificial neural networks, comprising:

[0025] The unsupervised pre-training module is configured to train and construct an artificial neural network based on unlabeled data using an unsupervised local learning method for artificial neural networks.

[0026] The classification and recognition module is configured to take the output of the unsupervised pre-training module as input, train an artificial neural network for classification and recognition tasks through backpropagation, and output the classification and recognition results.

[0027] The fine-tuning module is configured to fine-tune the parameters of each layer of the artificial neural network for classification and recognition tasks through backpropagation, thereby improving classification and recognition performance.

[0028] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of a generalized unsupervised local learning method for artificial neural networks.

[0029] A computer-readable storage medium having a computer program stored thereon, the steps of which, when executed by a processor, implement a generalized unsupervised local learning method for artificial neural networks.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. The neural network learning method proposed in this invention is unsupervised, requiring no manual labeling, and can efficiently extract features from unlabeled data.

[0032] 2. The features extracted by the learning method of this invention are universal and can be used as input for various learning tasks, effectively improving the accuracy of various learning tasks.

[0033] 3. The neural network trained by the learning method of this invention can be used as a pre-trained model for various tasks to improve the accuracy of various tasks.

[0034] 4. The learning method of this invention differs from the widely used backpropagation method, making it easier to achieve on-chip training of hardware-based artificial neural networks and supporting the development of technologies such as optical AI chips. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating how each layer in a multilayer neural network is trained and its connection weights are adjusted using local learning rules.

[0036] Figure 2 A schematic diagram illustrating the training of an artificial neural network using an unsupervised local learning method;

[0037] Figure 3 This is a schematic diagram of a classification and recognition system based on an unsupervised pre-trained model.

[0038] Figure 4 This is a schematic diagram of the network structure of a classification and recognition system based on unsupervised feature input in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the network structure of a classification and recognition system based on an unsupervised pre-trained model in an embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0041] Example 1

[0042] A generalized unsupervised local learning method for artificial neural networks, wherein the artificial neural network is a multi-layer neural network, and the parameters of each layer are trained through an unsupervised local learning method, such as... Figure 1 As shown. The parameters w of each layer are represented by a matrix that describes the connection weights between neurons in two layers; given the input data x of a certain layer, the output of that layer is f(w). T x), f is a non-linear activation function, and the training method for each layer of neurons is as shown in equation ( As shown in the image:

[0043] △w ij = ηy j (x i - ∑ k yk w ik ) ( )

[0044] Mode( In ), w ij Δw refers to the connection weight between neuron i in the next layer and neuron j in the current layer of an artificial neural network. ij It is the change in the connection weights, η is the learning rate, and x is the learning rate. i It is the input from neuron i during training, y j = ∑ j x i w ij It is the total weighted input of neuron j, ∑ k y k w ik This refers to the decay term derived from feedback, where k iterates through all neurons in this layer; for example... Figure 2 As shown.

[0045] When training a layer of neurons, given a training sample, the input x and y = w of this layer are calculated using a feedforward method. T x, and according to the formula ( Calculate the modification amount △w of the parameters in this layer. ij， Finally, w ij Replace with w ij +△w ij This training process can be performed in mini-batch mode. In each training step, a mini-batch of samples is randomly selected, and Δw is calculated. ij Update w using the average value ij The entire training process ends after a given number of steps. This training method has strong resistance to overfitting and does not require the early stopping method commonly used in deep learning to reduce overfitting.

[0046] Mode( The learning rule is a further development of the well-known Heblin and Oja rules. The original Heblin rule causes the training of connection weights to fail to converge, while the Oja rule is only used for training single neurons. The core here is the introduction of a decay term derived from the local output feedback -∑ k y k w ik This decay term enables competition among different synaptic connections within the same neuron and among neurons in the same layer for activation, thereby achieving the goal of neural network training and learning. It can be proven that this method can decompose the data of each layer into non-orthogonal principal component features, and the features of each layer can reconstruct the data of the previous layer.

[0047] Mode( ) of y j It can also be represented as the output of neuron j, in which case y j = f(∑ j x i w ij ), where f(.) is a nonlinear activation function, such as the ReLU function. In this case, equation ( The local learning method described can still effectively extract unsupervised features.

[0048] Mode( ) can be represented in matrix and vector form, as shown in equation ( As shown in the image:

[0049] △w = ηy(x - wy) T ( )

[0050] Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x is the input vector of this layer, and y = w T x is the total weighted input, (x - wy) T It is the transpose of x-wy.

[0051] Mode( ) can be extended to a more general form, such as equation ( As shown in the image:

[0052] △w = ηf(y) (x – g(w,y)) T ( )

[0053] Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x represents the input vector of this layer, and y = w. T x is the total weighted input, f(y) is a function of y, and g(w,y) is a function of w and y.

[0054] In an artificial neural network, each layer can be, but is not limited to, a fully connected layer, a locally connected layer, or a convolutional layer. In locally connected layers and convolutional layers, only neurons in adjacent layers with close proximity are connected. The difference between locally connected layers and convolutional layers is that the connection weights of neurons at different positions in a locally connected layer are different, with no parameter sharing, while convolutional layers rely on parameter sharing. Based on unsupervised local learning methods, the training of locally connected layers is the same as that of fully connected layers, except that there are fewer connections between neurons. In the training of convolutional layers, Δw is calculated at different positions. ij The average value, and update w based on that average value. ij .

[0055] Unsupervised local learning methods train neural networks of arbitrarily multiple layers from the bottom up; such as... Figure 1 As shown, each layer in an artificial neural network learns through a local learning rule, namely equation ( The connection weights are adjusted during training. Each layer extracts the main features from the data of the previous layer and transforms the data using a non-activation function. The trained artificial neural network achieves unsupervised feature extraction, with higher feature layers representing more complex features. Compared to backpropagation, this method can better extract richer features from training samples. It can be used for input data preprocessing or to build unsupervised pre-trained models. For various supervised learning tasks, accuracy can be improved through data preprocessing or fine-tuning of the pre-trained model.

[0056] The non-activation functions in artificial neural networks are, but are not limited to, squared functions, and the output features of each layer can be normalized to improve the stability of artificial neural network training.

[0057] Artificial neural networks can be trained layer by layer or all layers can be trained together. If trained layer by layer, training starts from the bottom layer and proceeds upwards. If all layers are trained together, the parameters of all layers are learned and adjusted simultaneously during the training process.

[0058] Example 2

[0059] A classification and recognition system based on an unsupervised local learning method using artificial neural networks, such as... Figure 3 As shown, it includes:

[0060] The unsupervised pre-training module is configured to train and construct an artificial neural network based on unlabeled data using an unsupervised local learning method for artificial neural networks.

[0061] The classification and recognition module is configured to take the output of the unsupervised pre-training module as input, train an artificial neural network for classification and recognition tasks through backpropagation, and output the classification and recognition results.

[0062] The fine-tuning module is configured to fine-tune the parameters of each layer of the artificial neural network for classification tasks by using backpropagation, based on the constructed unsupervised pre-training module and classification module, thereby improving the classification performance.

[0063] Compared to a single classification and recognition system based on backpropagation, the classification and recognition system based on an unsupervised pre-trained model proposed in this invention can significantly improve the accuracy of various classification and recognition tasks.

[0064] The unsupervised pre-training module is a general-purpose module whose training process is independent of specific learning tasks. This unsupervised pre-trained model can be combined with and improve the performance of various learning task modules, including but not limited to the classification and recognition modules mentioned above. It can be used for industrial inspection, medical image diagnosis, image segmentation, temporal prediction, text translation, speech recognition, etc.

[0065] Example 3

[0066] The application of the unsupervised local learning method based on artificial neural networks described in Example 1 for image classification and recognition includes:

[0067] An artificial neural network is constructed by training on unlabeled data using an unsupervised local learning method; such as... Figure 4 As shown, the constructed artificial neural network is a feature extraction network, including a 28*28 neuron layer, an activation function, and a normalization layer. The square function is used as the activation function, and the features are normalized afterwards. Unlabeled data refers to a number of unlabeled images, specifically the MNIST dataset, which includes 60,000 images of handwritten digits.

[0068] An artificial neural network for image classification and recognition is trained using backpropagation. This network is a classification network consisting of a single layer with 10 neurons and a softmax function, which outputs the predicted image category probability. Backpropagation is then used to fine-tune the parameters of each layer of the artificial neural network to improve its classification performance.

[0069] The image to be predicted is sequentially input into the trained feature extraction network and classification network, and the predicted image category probability is output to obtain the image classification and recognition result.

[0070] Using the MNIST dataset as an example, which contains 60,000 images of handwritten digits, a single-layer neural network performing backpropagation for classification yields an error rate of 7.23%. However, replacing the input images with the features extracted by this method reduces the error rate to 2.56%, demonstrating the effectiveness of the extracted features.

[0071] Continuing with the MNIST dataset as an example, if we use 60,000 labeled images... Figure 5The entire network trained via backpropagation achieves a false recognition rate of 1.62%. However, if the first layer is pre-trained using the method of this invention, followed by backpropagation training of other layers, and finally fine-tuning of the entire network, a false recognition rate of 1.36% can be obtained. This demonstrates that even with the same network and data, pre-training using the method of this invention can effectively improve the accuracy of the learning task, surpassing the effect of simply using backpropagation. Furthermore, considering only 600 labeled images, backpropagation alone achieves a false recognition rate of 24.7%; however, by using the pre-trained model of the first layer, the false recognition rate can be reduced to 20.7%. This indicates that pre-trained models based on a large amount of unlabeled data can effectively aid in supervised learning tasks.

[0072] Example 4

[0073] The application of the unsupervised local learning method based on artificial neural networks described in Example 1 is used for industrial inspection, medical image diagnosis, image segmentation, time series prediction, text translation, or speech recognition, etc. The specific implementation steps include:

[0074] First, a large amount of unlabeled data is collected for a specific learning task, and a neural network is trained as an unsupervised pre-trained model based on a local learning method.

[0075] Secondly, select an existing network model for the specific learning task. For example, U-Net network model can be used for industrial inspection and medical image diagnosis, and Mask R-CNN network model can be used for image segmentation. The output of the pre-trained model is used as the input of the network model.

[0076] Next, collect labeled data and train the network model using backpropagation. You can first freeze the parameters of the unsupervised pre-trained model, train a task-specific network model, and then fine-tune all the parameters of the entire network.

[0077] Finally, the trained task-specific network model is used to achieve industrial inspection, medical image diagnosis, image segmentation, time series prediction, text translation or speech recognition, etc.

[0078] In this way, unsupervised features can be automatically extracted using a large amount of unlabeled data, thereby improving the accuracy of various supervised learning tasks.

[0079] Example 5

[0080] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement steps of a generalized unsupervised local learning method for artificial neural networks.

[0081] Example 6

[0082] A computer-readable storage medium having a computer program stored thereon, the steps of which, when executed by a processor, implement a generalized unsupervised local learning method for artificial neural networks.

Claims

1. An application of an unsupervised local learning method based on artificial neural networks, characterized in that, For image classification and recognition, an artificial neural network is constructed by training on unlabeled data using an unsupervised local learning method based on artificial neural networks. The constructed artificial neural network is a feature extraction network, including a 28*28 neuron layer, an activation function, and a normalization layer. The square function is used as the activation function, and the features are normalized afterward. The unlabeled data refers to several unlabeled images. An artificial neural network for image classification and recognition is trained using backpropagation. This network is a classification network, consisting of a single layer with 10 neurons and a softmax function. The softmax function outputs the predicted probability of the image category. The parameters of each layer of the artificial neural network are then fine-tuned using backpropagation. The image to be predicted is sequentially input into the trained feature extraction network and classification network, and the predicted image category probability is output to obtain the image classification and recognition result. The method is based on unsupervised local learning of artificial neural networks, including: the artificial neural network is a multi-layer neural network, and the parameters of each layer are trained by unsupervised local learning. The parameters w of each layer are represented by a matrix, which describes the connection weights between neurons in two layers; given the input data x of a certain layer, the output of that layer is f(w). T x), f is a non-linear activation function, and the training method for each layer of neurons is as shown in equation ( As shown in the image: △w ij = ηy j (x i - ∑ k y k w ik ) ( ) Mode( In ), w ij Δw refers to the connection weight between neuron i in the next layer and neuron j in the current layer of an artificial neural network. ij It is the change in the connection weights, η is the learning rate, and x is the learning rate. i It is the input from neuron i during training, y j = ∑ j x i w ij It is the total weighted input of neuron j or y j = f(∑ j x i w ij ) is the output of neuron j, ∑ k y k w ik This refers to the decay term from the feedback, where k traverses all neurons in this layer; When training a layer of neurons, given a training sample, the input x and y = w of this layer are calculated using a feedforward method. T x, and according to the formula ( Calculate the modification amount △w of the parameters in this layer. ij Finally, w ij Replace with w ij +△w ij .

2. The application of the unsupervised local learning method based on artificial neural networks according to claim 1, characterized in that, The formula ( ) can be represented in matrix and vector form, as shown in equation ( As shown in the image: △w = ηy(x - wy) T ( ) Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x is the input vector of this layer, and y = w T x is the total weighted input, (x - wy) T It is the transpose of x-wy; Or, the formula ( Expanded into the form ( As shown in the image: △w = ηf(y) (x – g(w,y)) T ( ) Mode( In this context, w represents the parameters of this layer, i.e., the connection weights, x represents the input vector of this layer, and y = w. T x is the total weighted input, f(y) is a function of y, and g(w,y) is a function of w and y.

3. The application of the unsupervised local learning method based on artificial neural networks according to claim 1, characterized in that, Each layer in the artificial neural network is a fully connected layer, a locally connected layer, or a convolutional layer; during the training of the convolutional layer, Δw is calculated at different positions. ij The average value, and update w based on that average value. ij .

4. The application of the unsupervised local learning method based on artificial neural networks according to claim 1, characterized in that, Unsupervised local learning methods train neural networks of any number of layers from the bottom up; each layer extracts the main features from the data of the previous layer, and the trained artificial neural network achieves unsupervised feature extraction. The higher the feature layer, the more complex the features.

5. The application of the unsupervised local learning method based on artificial neural networks according to claim 1, characterized in that, Artificial neural networks can be trained layer by layer or all layers can be trained together. If each layer is trained layer by layer, it starts from the bottom layer and trains layer by layer upwards. If all layers are trained together, the parameters of all layers are learned and adjusted simultaneously during the training process.

Images