Method for updating neural network, method for classifying and electronic device

Abstract

The present disclosure relates to the field of artificial intelligence. The present disclosure provides a neural network updating method, a classification method and an electronic device. In the neural network updating method, the neural network comprises a plurality of network layers, at least one of the plurality of network layers comprises a plurality of neurons, and the method comprises: obtaining input data of the neural network; and determining an activation state of at least one neuron in the neural network based on the input data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a continuous learning scenario in the field of artificial intelligence. More specifically, the present disclosure relates to a neural network updating method, a classification method, and an electronic device. BACKGROUND

[0002] Traditional machine learning is performed for a fixed task, that is, a data set used to train a neural network (may also be referred to as a neural network model, or a learning model) contains training data with a fixed distribution. When a new data set (i.e., a data set containing training data with a new distribution different from the fixed distribution) is input, the neural network generally needs to be retrained. After retraining, the neural network can only respond to the new data set, but cannot respond to the original data set (i.e., a data set containing the fixed category of data). This problem is referred to as "Catastrophic Forgetting" in machine learning. In fact, the "Catastrophic Forgetting" is a result of the "Stability-Plasticity Dilemma" faced by machine learning, where stability refers to the ability to maintain original knowledge while learning new knowledge, and plasticity refers to the ability to learn new knowledge.

[0003] Continual Learning is training a neural network on a continuous sequence composed of multiple different tasks. Continual Learning aims to solve the above-mentioned "Catastrophic Forgetting" problem, and more specifically, it maintains the performance of the neural network on historical tasks while training the neural network based on new input data to adapt to new tasks. Continual Learning is the key to adapting a neural network to rapid changes in learning tasks, and thus is crucial for realizing the application of artificial intelligence in real-world scenarios. How to make the neural network better maintain the performance on historical tasks while adapting to new tasks is a problem to be optimized. SUMMARY

[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a neural network updating method, a classification method, and an electronic device.

[0005] According to an aspect of the present disclosure, a neural network updating method is provided, wherein the neural network includes a plurality of network layers, at least one of the plurality of network layers contains a plurality of neurons, and the method includes: obtaining input data of the neural network; and determining an activation state of at least one neuron in the neural network based on the input data.

[0006] According to an aspect of the present disclosure, there is provided a classification method, comprising: obtaining data to be classified; using an updated neural network to classify the data to be classified, and outputting classified data, wherein the updated neural network is composed of activated neurons.

[0007] According to an aspect of the present disclosure, there is provided an electronic device, comprising: a processor; a memory for storing computer program instructions; wherein when the computer program instructions are loaded and run by the processor, the processor performs the updating method of the neural network and / or the classification method as described above.

[0008] According to an aspect of the present disclosure, there is provided a training / or updating method of a neural network, wherein the neural network comprises a plurality of network layers including an input layer, a plurality of hidden layers, and an output layer, and at least one of the plurality of network layers comprises a plurality of channels, the method comprising: determining a loss function comprising: determining an expectation term to maximize a probability that the neural network outputs a true classification label; and determining a regularization term to minimize a number of activated channels in a training / or updating phase of a current task, the regularization term being an accumulation of a channel regularization term for each channel in the neural network, wherein when the channel regularization term is greater than or equal to a threshold value, it is determined that the channel is activated in the training / or updating phase of the current task; and when the channel regularization term is less than the threshold value, it is determined that the channel is not activated in the training / or updating phase of the current task.

[0009] Further, according to an aspect of the present disclosure, there is provided a training / or updating method of a neural network, wherein in a training / or updating phase for a current task, an activated channel in a first layer of hidden layers is configured to: receive information indicating channel variables of all channels of an input layer from all channels of the input layer; generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel in the first layer of hidden layers and all channels of the input layer, wherein the updated connection parameter remains unchanged in a training / or updating phase of a subsequent task.

[0010] Further, according to an aspect of the present disclosure, there is provided a training / or updating method of a neural network, wherein in a training / or updating phase for a current task, an activated channel in a first layer of hidden layers is configured to: receive information indicating channel variables of all channels of an input layer from all channels of the input layer; generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel in the first layer of hidden layers and all channels of the input layer, wherein the updated connection parameter remains unchanged in a training / or updating phase of a subsequent task.

[0011] Further, according to an aspect of the present disclosure, there is provided a method for training / or updating a neural network, wherein, in a training / or updating phase for a current task, an activated channel in an i+1-th hidden layer is configured to: receive, from an activated channel of an i-th hidden layer, information indicative of a channel variable of the activated channel of the i-th hidden layer, generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel of the i+1-th hidden layer and the activated channel of the i-th hidden layer, wherein the updated connection parameter remains unchanged in a training / or updating phase for a subsequent task, wherein i is an integer greater than or equal to 2.

[0012] Further, according to an aspect of the present disclosure, there is provided a method for training / or updating a neural network, wherein, in a training / or updating phase for a current task, an activated channel in an i+1-th hidden layer is configured to: receive, from an activated channel of an i-th hidden layer, information indicative of a channel variable of the activated channel of the i-th hidden layer, generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel of the i+1-th hidden layer and the activated channel of the i-th hidden layer, wherein the updated connection parameter remains unchanged in a training / or updating phase for a subsequent task, wherein i is an integer greater than or equal to 2.

[0013] Further, according to an aspect of the present disclosure, there is provided a method for training / or updating a neural network, wherein, in a training / or updating phase for a current task, an activated channel in an i+1-th hidden layer is configured to: receive, from an activated channel of an i-th hidden layer, information indicative of a channel variable of the activated channel of the i-th hidden layer, generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel of the i+1-th hidden layer and the activated channel of the i-th hidden layer, wherein the updated connection parameter remains unchanged in a training / or updating phase for a subsequent task, wherein i is an integer greater than or equal to 2.

[0014] Further, according to an aspect of the present disclosure, there is provided a method for training / or updating a neural network, wherein, in a training / or updating phase for a current task, an activated channel in an i+1-th hidden layer is configured to: receive, from an activated channel of an i-th hidden layer, information indicative of a channel variable of the activated channel of the i-th hidden layer, generate an updated connection parameter based on the information and a current connection parameter for a connection between the activated channel of the i+1-th hidden layer and the activated channel of the i-th hidden layer, wherein the updated connection parameter remains unchanged in a training / or updating phase for a subsequent task, wherein i is an integer greater than or equal to 2.

[0015] According to another aspect of the present disclosure, there is provided a training / updating apparatus of a neural network, wherein the apparatus performs the training / updating method as described above.

[0016] According to another aspect of the present disclosure, there is provided an electronic device comprising: a processor; a memory for storing computer program instructions; wherein the processor performs the training / updating method as described above when the computer program instructions are loaded and run by the processor. As will be described in detail below, the updating method and apparatus of a neural network, the classification method and the electronic device according to the present disclosure can better avoid the influence of the update for a new task on the performance of the neural network in completing a previous task.

[0017] It is to be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0019] Figure 1 A flowchart showing the operation for determining channel activation / deactivation is shown;

[0020] Figure 2 A design of input layer to first layer hidden layer in multi-head learning and single-head learning is shown;

[0021] Figure 3 A design between two hidden layers in multi-head learning and single-head learning is shown;

[0022] Figure 4A A design between last layer hidden layer and output layer in multi-head learning scenario is shown;

[0023] Figure 4B A design between last layer hidden layer and output layer in single-head learning scenario is shown;

[0024] Figure 5 is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure;

[0025] Figure 6 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure;

[0026] Figure 7is a schematic diagram showing a search for the best split point;

[0027] Figure 8A and 8B is a schematic diagram showing extending network capacity according to an embodiment of the present disclosure;

[0028] Figure 9 is a schematic diagram showing a training process according to an embodiment of the present disclosure;

[0029] Figure 10 is a schematic diagram showing network states in a training process according to an embodiment of the present disclosure;

[0030] Figure 11 is a flowchart showing a neural network updating method according to an embodiment of the present disclosure; and

[0031] Figure 12 is a flowchart showing an information bottleneck according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present disclosure more obvious, the following will describe example embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0033] A disadvantage of deep learning is that it cannot learn new information continuously as the human brain does. When a neural network is updated (also referred to as trained) for a new task, the information learned may be forgotten by the neural network. For example, assume that a neural network can classify 10 different data categories, if the neural network is required to classify 11 different data categories, that is, a new category is added, the neural network needs to be updated on a data set containing the 11th category, and there is little (or even no) data from the first 10 categories. After the update phase, the neural network may forget the performance of the first 10 categories, and as more and more categories are added, the neural network may completely forget the earlier updated categories.

[0034] Continual learning is to train a neural network on a continuous sequence composed of multiple different tasks. Continual learning aims to solve the problem of "catastrophic forgetting" described above, and this technology has a large demand in practical applications. For example, in an image-related application, a user can be provided with an image classification function, in addition to predefined categories, the user can also be allowed to customize categories. Therefore, the neural network needs to be updated using some images selected by the user and a small amount of stored images.

[0035] Continuous learning includes single-headed learning and multi-headed learning. Among them, (i) for single-headed learning, all tasks use one output layer structure, for example, class-incremental learning (Class-IL) belongs to single-headed learning; and (ii) for multi-headed learning, each task has an independent output layer, for example, task-incremental learning (Task-IL) belongs to multi-headed learning.

[0036] There are two main solutions to the problem of "catastrophic forgetting" in continuous learning:

[0037] The first method is a weight regularization method. For a neural network that has been trained for previous tasks and has achieved good training results, the weight regularization method estimates the importance of each parameter in the neural network for the previous tasks, and generates a weight regularization term for each parameter based on the importance, and adds the weight regularization term to the loss function. When training for a new task, the neural network using the weight regularization method will allow relatively unimportant parameters to change more to learn new tasks, and will keep the degree of change of relatively important parameters as small as possible.

[0038] The second method is a replay-based method. This type of method uses a small storage space to store input data samples belonging to previous tasks, and trains on the input data samples while learning for new tasks. When training for a new task, the neural network using the replay-based method will keep the features extracted for the input data samples as similar as possible to the features expressed without training for new tasks, so as to promote the model after training for new tasks to perform as close as possible to the performance before training for new tasks on previous tasks.

[0039] By using the above two methods, the problem of "catastrophic forgetting" in continuous learning can be alleviated, but the problem of "representation drift" is still not solved. Specifically, for training a neural network for a new task, the model parameters obtained according to the input data distribution of the previous task will continuously drift with the addition of the new task and will eventually be completely forgotten, so the performance of the neural network on the previous task will be affected.

[0040] For the learning model using the weight regularization method, 1) although the change degree of the relatively important parameters is kept as small as possible, there is still a slight change; 2) in addition, for each weight, the important coefficients and parameters need to be stored, so a large storage space is required; 3) if there are too many new tasks, the network capacity needs to be expanded, and the neural network cannot adaptively expand the network capacity. If the network capacity is automatically expanded every time there is a new task, there will also be a problem that the neural network cannot be fully utilized.

[0041] For the neural network using the representation regularization method, 1) although the model after training for a new task will try to be as close as possible to the performance of the previous model in completing the previous task, it cannot be exactly the same; 2) if there are too many new tasks, the network capacity needs to be expanded, and the neural network cannot adaptively expand the network capacity. If the network capacity is automatically expanded every time there is a new task, there will also be a problem that the neural network cannot be fully utilized.

[0042] Therefore, the inability to better solve the problem of "representation drift" is one of the limitations of continuous learning. In addition, how to reduce the storage space of the neural network and how to adaptively expand the capacity of the neural network are also aspects that need to be optimized for continuous learning. The size of the network will linearly grow with the number of new tasks.

[0043] The scheme provided by the embodiments of the present application relates to neural network updating in the field of artificial intelligence, wherein the updated neural network can be used for classification, segmentation, detection, etc. The specific implementation is described as follows. It should be noted that although the following embodiments are described in the context of classification tasks in image recognition, the application scenarios of the present application are not limited thereto, and can also be applied to any appropriate scenarios such as speech recognition, segmentation (e.g., semantic segmentation), detection (e.g., object detection), etc. In addition, the input data of the neural network of the present application includes at least one of the following: images (such as handwritten images or human images, etc.), audio / speech, and text.

[0044] In the embodiments of the present application, the neural network can be referred to as a neural network model, and can also be a learning model; the update can also be referred to as training (such as updating the neural network can also be referred to as training the neural network); the neurons in the neural network can also be referred to as channels.

[0045] The key differences between the embodiments of the present application and the prior art include at least one of the following aspects:

[0046] • The initial neural network is trained in a sparse-encouraging manner, which means that the network is not only required to achieve good performance on the task, but is also encouraged to use as few neurons as possible. Thus, only a minimal part of the network capacity is used for the current task, while the rest of the network capacity is reserved for future tasks;

[0047] • The activated part of the network is fixed, which is essential for old tasks, and can avoid the problem of expression drift, which means that the expression on the activated neurons will not drift no matter how many new tasks are incorporated;

[0048] • (Possibly) re-activate the non-activated part of the network, which can learn new information useful for new tasks;

[0049] • If a certain layer reaches its upper limit of capacity (also known as upper limit of ability), the layer can be expanded to increase the network capacity.

[0050] And the embodiments of the present application can achieve at least one of the following advantages:

[0051] • There is no problem of expression drift;

[0052] • Almost no additional storage space is needed to save the entire model (neural network);

[0053] • If necessary, the network can grow adaptively (optimize the model size);

[0054] • Even if the network needs to be expanded, the network size will not grow linearly with respect to the number of new tasks;

[0055] • Better utilization of network capacity can be achieved.

[0056] As an example, as shown in Figure 11 , a flowchart of a neural network updating method proposed by the embodiments of the present application is shown. The initial neural network is trained in a sparse-encouraging manner according to the input data, and the activated part and the non-activated part of the neural network are determined (i.e. the activation state of each neuron, which includes activation and non-activation). When there is a new classification or a new task, the neural network needs to be continuously updated, at which time the neural network can be trained according to the input data, and the activated part and the non-activated part of the neural network are updated (i.e. the activation state of each neuron is updated). When training the neural network, the activated part of the network can be fixed to avoid the problem of expression drift, or the non-activated part of the network can be re-activated (such as the neurons determined to be non-activated in the previous training result are re-activated, which can be trained). The network can learn new information useful for new tasks; when a certain layer reaches its upper limit of capacity (also known as upper limit of ability), the layer can also be expanded adaptively to increase the network capacity.

[0057] The embodiments of the present application can better avoid the influence of the update for the new task on the performance of the neural network in completing the previous task. When the input data distribution of the new task is greatly different from the input data distribution of the previous task, the problem of "representation drift" is more prominent, and therefore, the technical solutions proposed in the embodiments of the present application are particularly beneficial.

[0058] I. Traditional neural network model

[0059] The first embodiment is about a neural network model encouraging sparsity.

[0060] It is assumed that x represents an input image sample of a neural network model (the present example takes an image sample as input data for example, but the solutions of the embodiments of the present application are not limited thereto), and y represents a true image classification output by the input image sample x.

[0061] Loss function of the neural network model encouraging sparsity As shown in expression 1:

[0062]

[0063] wherein h l represents a hidden variable of an lth layer of the neural network model, wherein l = 1, 2, …, L, and L is the total number of hidden layers of the neural network model, h L represents a hidden variable of the last layer of the hidden layers; y represents a true image classification output by the input image sample x; I(h L ,y) represents mutual information between h L and y, the greater the mutual information value is, the more information h L and y contain to each other, the smaller the mutual information is, and the less information h L and y contain to each other.

[0064] The training target of the neural network model encouraging sparsity is to minimize the loss function , in particular, the minimization of the loss function includes:

[0065] (1) making I(h L ,y) as large as possible, that is, making the mutual information I(h L ,y) between the hidden variable h L of the last layer of the hidden layers of the network and the true classification label y of the input image data as large as possible, so as to make the probability of the neural network outputting the true classification label y as high as possible, and the classification accuracy of the neural network as high as possible.

[0066] II. (a) Improved network model encouraging sparsity

[0067] The second embodiment is about an improved network model encouraging sparsity.

[0068] Suppose x represents an input image sample of a neural network model, and y represents a true image classification output by the input image sample x. The loss function of the improved neural network model encouraging sparsity is As shown in expression 2:

[0069]

[0070] wherein I (h L , y) is an expected item, h l represents the hidden variable of the lth layer of the neural network model, wherein l = 1, 2, …, L, L is the total number of hidden layers of the neural network model, h L represents the hidden variable of the last layer of the hidden layer; y represents the true image classification output by the input image sample x, and the expected item represents the mutual information between h L and y, the greater the mutual information value is, the more information the two variables h L and y contain each other, the smaller the mutual information is, the less information the two variables h L and y contain each other.

[0071] wherein I (h l , h l-1 ) is a regular item for encouraging sparsity, which represents the mutual information between the hidden variable h l of the lth layer of the hidden layer and the hidden variable h l-1 of the (l-1)th layer of the hidden layer, the greater the mutual information value is, the more information the two variables h l of the lth layer of the hidden layer and h l-1 of the (l-1)th layer of the hidden layer contain each other, the smaller the mutual information is, the less information the two variables h l of the lth layer of the hidden layer and h l-1 of the (l-1)th layer of the hidden layer contain each other; γ is an adjustable hyperparameter, which can control the balance between the regular item for encouraging sparsity and the loss function .

[0072] The training target of the improved neural network model encouraging sparsity is to minimize the loss function , in particular, the minimization of the loss function includes:

[0073] (1) minimizing I (h L, y) is as large as possible, that is, the hidden variable h L of the last hidden layer of the network L , y) is as large as possible, so that the probability of the neural network outputting the real classification label y is as high as possible, and the classification accuracy of the neural network is as high as possible;

[0074] (2) Regularization term for encouraging sparsity is as small as possible, that is, the hidden variable h l of the l-th hidden layer of the network l-1 contains as little information as possible from the hidden variable h L of the f-1-th hidden layer, so that each hidden layer of the neural network model extracts as little information as possible from the previous layer, which can be understood as extracting only the most useful information.

[0075] Expression 3 shows a specific expression of the loss function of the improved neural network model for encouraging sparsity :

[0076]

[0077] , the first term is an expectation term, and q(y|h L ) represents the probability of the neural network model outputting the real classification label y for a given hidden variable h L of the last hidden layer, and the expectation term aims to make the probability of the neural network outputting the real classification label y as high as possible, thereby improving the classification accuracy of the network. Specifically, the subscript p gt of the expectation term represents the training set, and {x, y} ~ p gt represents that {x, y} is randomly sampled from the training set p gt ; the subscript p(h|x) of the expectation term represents the hidden variable distribution of all hidden layers after a given input image sample x, and the hidden variable distribution can be further decomposed as p(h1|x)p(h2|h1)…p(h l |h l-1), the subscript h~p(h|x) of the desired term represents that h is randomly sampled from the hidden variable distribution, where each term in the hidden variable distribution represents a distribution to which the hidden variable of a hidden layer should be subjected after a previous layer of the hidden layer is given, and each term in the hidden variable distribution is respectively configured by a corresponding hidden layer, for example, p(h1|x) represents a distribution to which the hidden variable h1 of the first layer of the hidden layer should be subjected after the input image sample x is given, p(h2|h1) represents a distribution to which the hidden variable h2 of the second layer of the hidden layer should be subjected after the first layer of the hidden layer h1 is given, and other terms are similar. In actual operation, if there are N samples and K classifications, the one-hot encoding of the nth sample is denoted as y (n) , and the kth element thereof is , where is 0 or 1. And the result output by the network is denoted as Then the first term can be further written as

[0078] , where the second term is a regular term for encouraging sparsity, and each summation term in the regular term for encouraging sparsity, that is, , corresponds to the information about the previous layer encoded in the cth channel of the lth layer of the hidden layer, L is the total number of hidden layers of the neural network model, C l represents the number of channels of the lth layer of the hidden layer, and the information bottleneck of the cth channel of the lth layer of the hidden layer is That is, and are parameters of the information bottleneck of the cth channel of the lth layer of the hidden layer, and are trainable parameters, and are optimized using back propagation, through which all trainable parameters (including weight connections, μ, σ) can receive the gradient of the loss function, and these parameters will be optimized to the best value. As understood by those skilled in the art, and also represent parameters of the information bottleneck of the cth channel of the lth layer of the hidden layer, and have the same meaning as and .

[0079] is the amount of information corresponding to the information bottleneck of the cth channel of the lth layer of the hidden layer, and is calculated by It is known that the loss function for training a neural network includes the sum of the information content corresponding to the information bottlenecks of each neuron in the hidden layer. In the embodiments of this application, a channel can also be referred to as a neuron; that is, the terms "channel" and "neuron" are used interchangeably. If If the value is 0, it means that the channel does not encode any information about the previous layer. This channel is useless for the training phase of the current task and therefore does not need to be activated in the current task; it can be reserved for subsequent tasks. If... If the channel is large, then it encodes more information about the previous layer. This channel is helpful for the training phase of the current task and therefore needs to be activated during the training phase of the current task. For example, the loss function when training a neural network includes the accumulation of the amount of information corresponding to the information bottleneck of each neuron in the hidden layer.

[0080] in, and These are two optimizable parameters used to define the latent variable distribution p(h1|x)p(h2|h1)…p(h l |h l-1 Each item in ) for example, p(h l |h l-1 The specific expression for ) is shown in expression 4:

[0081]

[0082] Among them, the hidden variable h of the l-th hidden layer l It is dimension H l ×W l ×C l The tensor of H, where H l W represents the height of the l-th hidden layer. l C represents the width of the l-th hidden layer. l This represents the number of channels in the l-th hidden layer. The superscripts i, j, and c represent the hidden variables h of the l-th hidden layer, respectively. l The three dimensions mentioned above.

[0083] in, Let f represent a Gaussian distribution. l l，j，c (·) is a learnable function, the function f l i，j，c (·) Receives the hidden variable h from the (l-1)th hidden layer. l-1 It also provides a fixed scalar value. Typically, the function f... l l，j，c (·) Each neural network layer is configured separately.

[0084] In practice, instead of directly sampling from the expression 4, we can first sample a random variable from a standard Gaussian distribution This random variable can also be regarded as a random perturbation that introduces random noise, and then we can obtain

[0085]

[0086] In fact, this operation is equivalent to making f l l，j，c (h l-1 ) pass through an information bottleneck defined by and .

[0087] In the process of optimizing and , the parameters in f l l，j，c (·) are optimized together with . Therefore, after the optimization operation, not only the optimized f l l，j，c (·) can be obtained, but also whether the cth channel of the lth hidden layer participates in the encoding of the information of the previous layer and to what extent it participates can be automatically learned by comparing with the threshold value.

[0088] The improved network model that encourages sparsity has a training target of minimizing the loss function , and specifically, the minimization of the loss function includes:

[0089] (1) making the expected item as large as possible, that is, making the mutual information I(h L , y) between the hidden variable h L of the last hidden layer of the network and the true classification label y of the input image data as large as possible, so that the probability of the neural network outputting the true classification label y is as high as possible, and the classification accuracy of the neural network is as high as possible;

[0090] (2) making the regular item as small as possible, so that the channel regular item ​As close to 0 as possible, that is, as many channels as possible are not involved in encoding information of the previous layer in the training phase of the current task, so that as few channels as possible are activated in the training phase of the current task, and as many channels as possible can be left for use in the training phase of the subsequent task.

[0091] The regular term encouraging sparsity is the accumulation of the channel regular term of each channel in the neural network model. By judging the channel regular term of each channel, it can be determined whether the channel is activated in the training phase of the current task. In this embodiment, the information amount corresponding to the information bottleneck can be determined by the regular term. For example, the information amount corresponding to the information bottleneck is the information amount of the previous network layer encoded by the neuron.

[0092] As shown in FIG. 1, the information bottleneck of the first neuron of the first hidden layer is shown in FIG. 2, and the parameters of the information bottleneck of the first neuron of the first hidden layer are shown in FIG. 3. Figure 12 Theoretically, if , the corresponding neuron encodes more information, and the neuron can be activated. μ and σ represent how much information the corresponding neuron encodes. μ can be regarded as a signal, and σ can be regarded as noise. If the signal is dominant, the neuron encodes useful information, and the neuron can be activated at this time. If the noise is dominant, the neuron encodes useless information, and the neuron can not be activated at this time.

[0093] Figure 1 A flowchart for determining the operation of channel activation / inactivation is shown, and the first channel of the first hidden layer channel is taken as an example for illustration. In the present application, a channel can also be referred to as a neuron, that is, the terms "channel" and "neuron" are used interchangeably.

[0094] In step S101, the channel regular term of the channel is compared with a threshold value.

[0095] In step S102, when the channel regular term is greater than or equal to the threshold value, it is determined that the channel is activated in the training phase of the current task.

[0096] In step S103, when the channel regular term is less than the threshold value, it is determined that the channel is not activated in the training phase of the current task. ​​​​​

[0097] In practice, if the channel regularization term A value of 0 indicates that the channel does not encode any information about the previous layer and does not need to be activated during the training phase for the current task. The channel regularization term... It may not be mathematically equal to zero, and the very small number of channels involved in encoding information in the previous layer are irrelevant to the training phase of the current task. Therefore, in practice, we can selectively activate channels during the training phase of the current task. The passage, and considered The channel encoding information is too limited, so it is not activated during the training phase of the current task and is reserved for use in the training phase of subsequent tasks. It should be noted that the condition for choosing whether to activate a channel can be any threshold; here, it is simply set to... The threshold is defined as 1 for illustrative purposes only.

[0098] Using the improved, sparsity-encouraging network model described above allows the network to achieve the same classification accuracy with a minimal number of activated neurons. In learning subsequent tasks, on the one hand, the "expression drift" problem can be solved by fixing the already activated neurons so that their expressions no longer change; on the other hand, neurons that were not activated in previous tasks can be reactivated to learn additional features for the new task, thus achieving good accuracy on the new task as well. If, at the end of a training phase, it is found that all neurons in certain layers are activated, leaving no additional neurons to participate in learning potential new features, then we can expand the network at that layer by adding some additional unactivated neurons.

[0099] The improved, sparsity-enhancing network model will be discussed in two scenarios: single-head learning and multi-head learning. In both cases, a complete training process is as follows: Figure 9 As shown. First, we obtained the first batch of data, and then used this batch of data to train a sparse network. The network state during the training process is as follows. Figure 10 As shown in (a), each channel corresponds to an information bottleneck. The information bottleneck participates in training and its activation is determined by its parameters after training. A typical training completion state is as follows: Figure 10 As shown in (b) of the diagram. In this embodiment, for at least one inactive neuron in a hidden layer, the corresponding information bottleneck is set to a preset value. For example, for the information bottleneck (μ, σ), setting the corresponding information bottleneck to the preset value could be setting μ to 1 and σ to 0. Based on the first batch of input data, the neural network is trained; according to the information bottleneck of the inactive neuron after training, the activation state of the inactive neuron is determined, and the neuron is activated. The channel is not activated. channel. After new data is acquired, the training on this part of data needs to be re-performed, and the training phase at this time is as shown in (c) of FIG. 1: Figure 10 (c) of FIG. 1: the activated channels remain activated and do not change, while the previously non-activated channels participate in the training again and decide whether to be activated after the training is completed. For example, if the information amount corresponding to the information bottleneck of the non-activated neuron after the training is greater than a preset threshold, it is determined that the activation state of the non-activated neuron is activated; if the information amount corresponding to the information bottleneck of the non-activated neuron after the training is not greater than the preset threshold, it is determined that the activation state of the non-activated neuron is not activated. A typical training completion state is as shown in (d) of FIG. 1. Figure 10 (d) of FIG. 1. After the training in this phase is completed, if necessary, the expansion of part of the layers of the network can be selected to open up new space. In this way, new features can be learned without forgetting old features. In an embodiment of the present application, the network is trained using the information bottleneck, so that the network becomes sparse after the training.

[0100] In the algorithm of the present application, there is an information bottleneck in the training process. The information bottleneck has a random variable ∈, which introduces random noise. Adding random noise during training, random disturbance can be regarded as a kind of data enhancement to alleviate the problem of overfitting.

[0101] On the contrary, for example, in the prior art adaptive network, there is no information bottleneck involved. Therefore, it is more troubled by the problem of overfitting, which leads to relatively poor performance.

[0102] II. (b) Expansion of the network model encouraging sparsity

[0103] In the present application, a channel can also be referred to as a neuron, that is, the terms "channel" and "neuron" are used interchangeably. In the present embodiment, whether to increase neurons in a hidden layer of the neural network can be determined based on the activation state of the neurons in the hidden layer. In some cases, all the neurons of a certain layer are activated, and at this time there is no additional non-activated neuron that can encode new information potentially for subsequent tasks, as shown in the second layer network layer of Figure 8A When this situation is encountered, additional new neurons can be added to the layer to expand its upper limit of capacity. For example, when all the neurons of a hidden layer in the neural network are in the activated state, a preset number of non-activated neurons are added for the hidden layer. As shown in Figure 8B , 2 neurons are added to the second layer network layer, where the above-mentioned preset number can be a fixed value or a variable number.

[0104] II. (c) Application of the improved network model encouraging sparsity

[0105] II.(c.1) Application of improved network model encouraging sparsity in multi-head learning

[0106] In the scenario of multi-head learning, we discuss how the above-mentioned improved network model encouraging sparsity solves the "expression drift" problem in three cases.

[0107] Firstly, the design of the input layer to the first hidden layer is as shown in Figure 2 .

[0108] Suppose that in the training phase for the current task, the first two channels in the first hidden layer are activated channels, and the last two channels are unactivated channels.

[0109] - For the activated channels in the first hidden layer, they are configured to receive information indicating the channel variables of all channels of the input layer from all channels of the input layer. Based on the information and the current connection parameters for the connections between the activated channels in the first hidden layer and all channels of the input layer, updated connection parameters are generated to complete the training phase for the current task. And in the training phase of the subsequent task, the updated connection parameters remain unchanged; in the embodiments of the present application, the connections between the activated neurons in the first hidden layer and the neurons of the previous network layer remain unchanged during the training process.

[0110] - For the unactivated channels in the first hidden layer, they are configured to receive information indicating the channel variables of all channels of the input layer from all channels of the input layer. Based on the information and the current connection parameters for the connections between the unactivated channels in the first hidden layer and all channels of the input layer, updated connection parameters are generated. And in the training phase of the subsequent task, the updated connection parameters are initialized so as to continue to be updated in the training phase of the subsequent task; in the embodiments of the present application, the connections between the unactivated neurons in the first hidden layer and the neurons of the previous network layer are trainable during the training process.

[0111] For ease of understanding, in Figure 2 , the channel parameters that remain unchanged in the subsequent training phase are shown in solid lines; the channel parameters that will be initialized in the subsequent training phase are shown in dashed lines.

[0112] Further, the design between the two hidden layers is as shown in Figure 3 .

[0113] Suppose that in the training phase for the current task, the first two channels in the i-th hidden layer are activated channels, and the last two channels are unactivated channels; the first two channels in the i+1-th hidden layer are activated channels, and the last two channels are unactivated channels.

[0114] For the activated channel of the (i+1)th hidden layer, it is configured to receive information indicating the channel variable of the activated channel of the ith hidden layer from the activated channel of the ith hidden layer. Based on the information and the current connection parameter of the connection between the activated channel of the (i+1)th hidden layer and the activated channel of the ith hidden layer, an updated connection parameter is generated to complete the training phase of the current task. In the training phase of the subsequent task, the updated connection parameter remains unchanged. And the connection parameter of the connection between the activated channel of the (i+1)th hidden layer and the non-activated channel of the ith hidden layer is set to 0 and fixed to 0 in the training phase of the subsequent task; in an embodiment of the present application, the connection between the activated neuron in the hidden layer other than the first layer and the activated neuron in the previous network layer remains unchanged during the training process; the connection between the activated neuron in the hidden layer other than the first layer and the non-activated neuron in the previous network layer is 0 and remains unchanged during the training process;

[0115] For the non-activated channel of the (i+1)th hidden layer, it is configured to receive information indicating the channel variable of all channels of the ith hidden layer from all channels of the ith hidden layer. Based on the information and the current connection parameter of the connection between the non-activated channel of the (i+1)th hidden layer and all channels of the ith hidden layer, an updated connection parameter is generated. And in the training phase of the subsequent task, the connection parameter will be initialized so as to continue to be updated in the training phase of the subsequent task. In an embodiment of the present application, the connection between the non-activated neuron in the hidden layer other than the first layer and each neuron in the previous network layer is trainable during the training process.

[0116] For the convenience of understanding, in Figure 3 , the channel parameter remaining unchanged in the subsequent training phase is shown in solid line; the channel parameter to be initialized in the subsequent training phase is shown in dashed line.

[0117] Further, the design between the Lth hidden layer and the output layer is as shown in Figure 4A , wherein it is assumed that L is the total number of hidden layers of the neural network model, that is, the Lth hidden layer is the last hidden layer. In multi-head learning, each task corresponds to a specific output layer channel, which is circled in the form of a block in Figure 4A for the output layer channel of the training phase of the current task.

[0118] For the activated channels in the output layer for the training phase of the current task, they are configured to receive information from the activated channels of the Lth hidden layer indicating the channel variables of the activated channels of the Lth hidden layer. Based on the information and the current connection parameters for the connections between the activated channels of the Lth hidden layer and the channels in the output layer for the current task, updated connection parameters are generated to complete the training phase of the current task. In the training phase of the subsequent task, the updated connection parameters remain unchanged. And the connection parameters for the connections between the activated channels of the Lth hidden layer and the unactivated channels in the output layer for the current task are set to 0 and fixed to 0 in the training phase of the subsequent task. In an embodiment of the present application, if the output layer is a multi-head output layer, the connections between the activated neurons in the output layer and the activated neurons in the previous network layer remain unchanged during the training process; the connections between the activated neurons in the output layer and the unactivated neurons in the previous network layer are 0 and remain unchanged during the training process; the connections between the unactivated neurons in the output layer and each neuron in the previous network layer are trainable during the training process.

[0119] For the convenience of understanding, in Figure 4A , the parameters that remain unchanged in the subsequent training phase are shown in solid lines; the channel parameters that will be initialized in the subsequent training phase are shown in dashed lines.

[0120] It should be noted that which network layers are activated channels in the above description is only exemplary and not limiting.

[0121] II.(c.2) Application of the improved network model encouraging sparsity in single-head learning

[0122] In the scenario of single-head learning, we discuss how the above-mentioned improved network model encouraging sparsity solves the "expression drift" problem in three cases.

[0123] First, the design of the input layer to the first hidden layer is the same as the design in the scenario of single-head learning as shown in Figure 2 , which will not be repeated here.

[0124] Further, the design between the two hidden layers is the same as the design in the scenario of single-head learning as shown in Figure 3 , which will not be repeated here.

[0125] Further, the design between the Lth hidden layer and the output layer is as shown in Figure 4B . Assume that L is the total number of hidden layers of the neural network model, that is, the Lth hidden layer is the last hidden layer. In single-head learning, all tasks use the same output layer structure, that is, all channels of the output layer are used.

[0126] For all channels in the output layer, it is configured to receive information indicating channel variables of all channels of the Lth hidden layer from all channels of the Lth hidden layer. Based on the information and connections between all channels in the output layer and all channels of the Lth hidden layer, updated connection parameters are generated. In the training phase of the subsequent task, the updated connection parameters will be initialized to continue to be updated in the training phase of the subsequent task. In an embodiment of the present application, if the output layer is a single-head output layer, the connections between the neurons of the output layer and the neurons of the previous layer can be trained during the training process.

[0127] As those skilled in the art will understand, the term "trainable" used in the present application means that the connection parameters can be updated during the training process.

[0128] In single-head learning, although the channel variables of the channels of the output layer can change with the change of the task, the channel variables of the activated channels of the Lth hidden layer are constant for a certain task. Therefore, in single-head learning, the test phase does not directly use the output layer, but uses the Lth hidden layer. Specifically, after the training of the task is completed, we calculate and store the feature centers of each class in the Lth hidden layer. In the test phase of the task, we calculate the feature values of the test input test samples in the Lth hidden layer, and calculate the distances between the feature values and the corresponding feature centers of each class, and select the class with the closest distance to the feature center as the predicted classification output. In single-head learning, there may be a softmax layer after the output layer to calculate the probability, so the neurons in the output layer interact with each other, and even if the connection is fixed, the output representation will be different.

[0129] It should be noted that which layers in the network are activated in the above description is only exemplary and not limiting.

[0130] II.(d) Improved network model encouraging sparsity to improve classification accuracy

[0131] In single-head learning and multi-head learning, the improved network model encouraging sparsity achieves higher classification accuracy than existing algorithms.

[0132] In the following, the improvement effect of the improved network model encouraging sparsity on classification accuracy is described by comparison with existing algorithms.

[0133] First, several existing algorithms are briefly introduced as follows.

[0134] (1) No forgetting learning (LwF) model: this algorithm is mainly for multi-head learning;

[0135] (2) Elastic Weight Consolidation (EWC) model: This algorithm can be used for both multi-head and single-head outputs;

[0136] (3) Progressive Neural Tangent Kernel (P&C) model: This algorithm is mainly used for multi-head outputs;

[0137] (4) Adaptive Network model: This algorithm is mainly used for multi-head outputs;

[0138] (5) Incremental Classifier Expression Learning (iCaRL) model: This algorithm is mainly used for single-head outputs;

[0139] (6) SupportNet model: This algorithm is mainly used for single-head outputs.

[0140] II. (d.1) Multi-head learning scenario

[0141] To show the classification accuracy advantage of the improved network model that encourages sparsity in the multi-head learning scenario, the following will use public datasets, which are compared with the no forgetting learning model, the elastic weight consolidation model, and the progressive neural tangent kernel model.

[0142] Table 1-1 shows the comparison of the average classification accuracy of the algorithm proposed by the present application, the no forgetting learning model, the elastic weight consolidation model, and the progressive neural tangent kernel model on the public dataset OMNIGLOT for all 50 characters in OMNIGLOT:

[0143]

[0144] It can be seen that in the multi-head learning scenario, the improved network model that encourages sparsity can provide higher classification accuracy compared with existing algorithms.

[0145] Specifically, the OMNIGLOT dataset contains 50 different character tables, each containing 20 to 50 characters, and each character contains multiple handwritten pictures completed by multiple people.

[0146] First, the improved network model that encourages sparsity obtains all the data of the first character table and trains the first character table as an input image sample. After the first training stage is completed, not only a model capable of classifying the first character table is obtained, but also each layer of the hidden layer of the model has only part of the channels activated, and the other unactivated channels are left for subsequent character table training. The channel parameters of the activated channels in the first character table training remain unchanged in subsequent training.

[0147] Then, the improved network model encouraging sparsity acquires all data of the second character set, and takes the second character set as input image samples to train. After the completion of the second training phase, our model can well classify on the second character set, and the classification accuracy on the first character set is not changed. Considering that the classification task of the second character set requires the model to learn some new features, therefore, compared with only completing the training for the first character set, the number of activation channels in each hidden layer of the model will increase.

[0148] Repeating such a process, a trained model can finally be obtained, which has relatively high classification accuracy on all 50 character sets.

[0149] The improved network model encouraging sparsity enables the model to maintain its classification accuracy on the previously trained character set during the model training process of the subsequent character set, thereby providing higher classification accuracy compared with existing algorithms.

[0150] In addition, other algorithms except the present application and the growable network do not have a mechanism to automatically increase network nodes. Compared with the present application and the growable network, the present application only needs to increase network nodes in 25% of the cases, while the growable network needs to increase network nodes in all cases, so the present application can obtain higher classification accuracy with fewer neurons.

[0151] Table 1-2 shows the comparison of the average classification accuracy of the algorithm proposed in the present application, the elastic weight consolidation model, the activation regularization model, and the growable network on all 5 tasks in the split-MNIST public data set:

[0152]

[0153] Table 1-2

[0154] It should be noted that the algorithm proposed in the present application has better effect than the growable network, which is mainly because the information bottleneck introduced in the present application actually plays a kind of regularization role similar to random inactivation, thereby improving the generalization ability of the model.

[0155] II.(d.2) Single-head learning scenario

[0156] In order to show the classification accuracy advantage of the improved network model encouraging sparsity in the single-head learning scenario, the following will use a public data set to compare it with the elastic weight consolidation model, the incremental classifier expression learning model, and the support network model.

[0157] Table 2 shows the comparison of the final classification accuracy of the proposed algorithm, the elastic weight consolidation model, the incremental classifier expression learning model, and the support network model on the public dataset MNIST after all the training tasks are completed (in this test, each algorithm uses the same size of storage space, i.e., the storage space can store a total of 200 pictures):

[0158]

[0159] Table 2

[0160] For the handwriting number recognition application, 95% of the classification accuracy is the accuracy requirement that a good handwriting number recognition algorithm should achieve. Comparing the proposed algorithm and the existing algorithms, only the proposed algorithm achieves the 95% classification accuracy, and the classification accuracy of the other algorithms is still some distance from the requirement.

[0161] Specifically, the MNIST dataset contains handwritten number pictures completed by multiple people, and each picture is black and white, corresponding to one number from 0 to 9. We divide these numbers into 5 different batches, i.e., 0 and 1 are the first batch, 2 and 3 are the second batch, and so on.

[0162] First, the improved network model that encourages sparsity obtains all the data of the first batch,

[0163] i.e., all the pictures corresponding to 0 and 1, and trains the first batch as the input image sample. After the training is completed, for any test picture, the model can judge whether it is 0 or 1. Moreover, each layer of the hidden layer of the model has only part of the channels activated, and the other unactivated channels are left for subsequent batch training. After the first batch of training is completed, a part of the 0 and 1 picture samples in the first batch are randomly selected and stored in the storage space.

[0164] Then, the improved network model that encourages sparsity obtains all the data of the second batch,

[0165] i.e., all the pictures corresponding to 2 and 3, and trains the second batch together with the previously stored part of the 0 and 1 picture samples in the first batch in the storage space as the input image sample, aiming to enable the model to classify four types of pictures, 0, 1, 2, and 3. After this round of training is completed, compared with only completing the training for the first batch, the number of activated channels in each layer of the hidden layer of the model will increase.

[0166] Then, some of the stored 0 and 1 picture samples in the storage space are randomly emptied, aiming to free up storage space to store some of the 2 and 3 picture samples in the second batch. Such a process is repeated, and finally a trained model can be obtained, which can classify pictures corresponding to 0 to 9.

[0167] The improved network model encouraging sparsity enables the model to maintain its classification accuracy on the previously trained batches during the subsequent batch training process, thereby providing higher classification accuracy compared to existing algorithms.

[0168] Further, Table 3-1 shows the comparison of the final classification accuracy obtained after completing all task training of the algorithm proposed in the present application, the elastic weight consolidation model, the incremental classifier expression learning model, and the support network model on the public data set Cifar10:

[0169]

[0170] Table 3-1

[0171] Specifically, the training set of Cifar10 contains 50,000 color pictures, and the test set includes 10,000 color pictures, which include 10 different categories such as trucks and frogs, and the number of pictures of each category is the same. Similarly, we divide the 10 categories into 5 batches (Batch), i.e., the first two categories are the first batch, the third and fourth categories are the second batch, and so on. The compared models are sequentially trained on the classification task of the five batches (in this test, the same storage space size is used for each algorithm, i.e., a storage space that can store a total of 200 pictures).

[0172] First, the improved network model encouraging sparsity obtains all data of the first batch,

[0173] That is, all pictures corresponding to the first two categories (airplanes and cars) are trained as input image samples using the first batch. After training is completed, for any test picture, the model can determine whether it is an airplane or a car. Moreover, each hidden layer of the model has only some channels activated, and the other unactivated channels are left for subsequent batch training. After the first batch is trained, a part of the airplane and car picture samples in the first batch are randomly selected and stored in the storage space.

[0174] Then, the improved network model encouraging sparsity obtains all data of the second batch,

[0175] That is, all the pictures corresponding to the third and fourth categories (bird and cat) are trained together with the second batch and a part of the pictures of the first batch (airplane and car) previously stored in the storage space as input image samples, aiming to enable the model to classify the four categories of pictures (airplane, car, bird and cat). After the completion of this round of training, the number of activation channels in each hidden layer of the model will increase compared to when only the training for the first batch is completed.

[0176] Then, some of the pictures of airplanes and cars stored in the storage space are randomly emptied, aiming to free up storage space to store a part of the pictures of birds and cats in the second batch. Repeat this process and eventually get a trained model that can classify all ten categories of pictures.

[0177] The improved network model that encourages sparsity enables the model to maintain its classification accuracy on previously trained batches during subsequent batch training, thereby providing higher classification accuracy compared to existing algorithms.

[0178] For this task, the algorithm proposed by the present application can achieve an accuracy of more than 70%, which is a significant improvement in classification accuracy compared to existing algorithms.

[0179] In addition, Table 3-2 shows the comparison of the final classification accuracy obtained after training all tasks by the algorithm proposed by the present application, the elastic weight consolidation model, the incremental classifier expression learning model, and the support network model on the public data set Cifar100:

[0180]

[0181] Table 3-2

[0182] Specifically, the training set of Cifar100 contains 50,000 color pictures, and the test set includes 10,000 color pictures, which include 100 different categories, and the number of pictures in each category is the same. In this experiment, we divide the 100 categories into 10 batches (Batch), i.e. the first ten categories are the first batch, the eleventh to the twentieth categories are the second batch, and so on. The compared models are trained on the ten batches in turn for classification tasks (in this test, the same storage space size is used for each algorithm, i.e. a storage space that can store a total of 200 pictures).

[0183] III. (a) Improved network model that encourages sparsity

[0184] In the improved network model that encourages sparsity described in II, the loss function as shown in expression 3 The global hyper-parameter, γ, is used. Therefore, the sparsity ratio cannot be set in advance when using the model, but needs to be manually adjusted multiple times through experiments during use.

[0185] To more accurately control the sparsity ratio, in the third embodiment, another improved sparsity-encouraging neural network model is proposed.

[0186] Expression 6 shows a loss function of the improved sparsity-encouraging neural network model The specific expression is as follows:

[0187]

[0188] Among them, the second term is a regular term for encouraging sparsity, and each summation term in the regular term for encouraging sparsity, i.e., the channel regular term Corresponding to the information about the previous layer encoded in the cth channel of the lth hidden layer.

[0189] In expression 6, the global hyper-parameter γ that needs to be manually adjusted is no longer used, but there is a corresponding local hyper-parameter for each neuron / feature map channel, The local hyper-parameter, is adaptively adjusted according to the specific situation in actual optimization.

[0190] The adaptive adjustment method of the local hyper-parameter, is as follows:

[0191] Before the training starts, the initialization value of the local hyper-parameter, is set to a value close to 0, for example, 1×10 -10 That is, in the initial stage of training, the training target of the loss function focuses on improving the classification accuracy, rather than compressing the model.

[0192] In the optimization process of the loss function as shown in expression 6 (for example, using gradient descent method for optimization), the local hyper-parameter, is periodically updated (for example, the local hyper-parameter, is updated once every 10 iterations). The specific updating method is, for example:

[0193] As shown in expression 7, Figure 7 all the local hyper-parameters, are sorted from small to large, and then the best split point is searched from left to right. The best split point is determined according to the target compression ratio, that is, in order to make the compression ratio equal to the ideal compression ratio, the neurons / feature map networks on the left side of the best split point should be deleted. Assuming that the ideal mask value This indicates whether a particular neuron / feature map channel should be deleted to achieve the desired compression ratio. Specifically, when the current... If the value is less than the threshold corresponding to the optimal segmentation point, then the ideal mask will be applied. Set to 0 to indicate that the neuron / feature map channel is to the left of the optimal segmentation point and should therefore be deleted to achieve the ideal compression ratio; when the current If the value is greater than the threshold corresponding to the optimal segmentation point, then the ideal mask will be applied. Set to 1 to indicate that the neuron / feature map channel is to the right of the optimal segmentation point and therefore should not be deleted.

[0194] Furthermore, assuming the actual mask is used This indicates whether a particular neuron / feature map channel has actually been deleted. Specifically, when the current... If the value is less than a preset pruning threshold (e.g., a pruning threshold of 1), then the actual mask will be applied. Set to 0 to indicate that the neuron / feature map channel has actually been deleted; when the current If the value exceeds the preset pruning threshold, the actual mask will be adjusted. Set to 1 to indicate that the neuron / feature map channel was not actually deleted.

[0195] Based on the above classification criteria, according to and The values ​​of can be categorized into four types across all neurons / feature map channels:

[0196]

[0197] Below, for each of the four types of neurons / feature map channels mentioned above, we will describe the hyperparameters corresponding to that neuron / feature map channel. Adjustment method:

[0198] when and Time: No corresponding hyperparameter Make any adjustments;

[0199] when and Time: In order to limit the channel encoding information of this neuron / feature map, the corresponding hyperparameters are... Increase the corresponding channel regularization term in the loss function. In practice, for example, we can adjust the corresponding hyperparameter... This is achieved by multiplying by a coefficient greater than 1 (e.g., 1.1).

[0200] when and Time: The corresponding hyperparameter The corresponding channel regularizer in the loss function is weakened. In practice, we can reset the corresponding hyper-parameter to the initial value (e.g., 1 x 10 -10 ) for example. Further, to accelerate the process of the neuron / feature map channel re-engaging in the training, we reset the corresponding hyper-parameter to the initial value, e.g., 0.01.

[0201] and To encourage the neuron / feature map channel to encode more information, we adjust the corresponding hyper-parameter to weaken the corresponding channel regularizer in the objective function. In practice, we can achieve this by multiplying the corresponding hyper-parameter by a coefficient less than 1 (e.g., 0.9).

[0202] Further, to ensure the adjustment speed, we set the value range for each , for example, the upper and lower limits are 1 x 10 -3 and 1 x 10 -10 , respectively. That is, when is multiplied by 1.1 and is greater than 1 x 10 -3 , we reset to 1 x 10 -3 ; when is multiplied by 0.9 and is less than 1 x 10 -10 , it should be reset to 1 x 10 -10 . The improved network model that encourages sparsity makes the results of the training better meet the preset sparsity ratio.

[0203] III.(b) Performance of the improved network model that encourages sparsity in achieving the preset sparsity ratio

[0204] The following illustrates the effect of the sparse network with controllable compression ratio using the Cifar10 dataset as an example. The Cifar10 dataset is a dataset containing 50,000 training images and 10,000 test images, each of which is a color image with a side length of 32 pixels, and these images come from 10 classes. The basic network model selected is VGG16, which has 16 layers, including 13 convolutional layers and 3 fully connected layers. For Cifar10 classification, the model has a lot of redundancy that can be compressed.

[0205] First, the classification task of Cifar10 is trained on VGG16. Then, the model is compressed by using the traditional information bottleneck network and the compression ratio controllable information bottleneck network proposed by the present application respectively. For the traditional information bottleneck network, different global hyperparameters γ are applied. For the compression ratio controllable information bottleneck network proposed by the present application, channel-specific local hyperparameters based on adaptive adjustment of different preset compression ratios are applied Then, the two models are compared from the two dimensions of classification accuracy and compression ratio.

[0206] Table 4 shows the classification accuracy and compression ratio of the traditional information bottleneck network under different γ, and Table 5 shows the classification accuracy and compression ratio of the compression ratio controllable information bottleneck network of the present application under different preset compression ratios.

[0207] γ test accuracy compression ratio 2 x 10 -5 ]] 91.69％ 4.60％ 5 x 10 -5 ]] 91.40％ 3.54％ 1 x 10 -4 ]] 89.92％ 2.78％

[0208] Table 4

[0209] preset compression ratio test accuracy compression ratio 4％ 91.15％ 4％ 3％ 91.04％ 3％ 2％ 90.43％ 2％

[0210] Table 5

[0211] It can be seen that the traditional information bottleneck network cannot accurately control the compression ratio, while using the compression ratio controllable information bottleneck network proposed by the present application, the actual compression ratio is completely consistent with the preset compression ratio, and at the same time, the classification accuracy is also comparable to that of the traditional information bottleneck network.

[0212] According to the above disclosed method, a training method of a neural network model for image recognition can obtain output data for recognizing images or features in images by using image data as input data of an artificial intelligence model. The artificial intelligence model can be obtained by training. Here, "obtained by training" means that a basic artificial intelligence model is trained by a training algorithm using multiple pieces of training data, and a predetermined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training the basic artificial intelligence model by the training algorithm. The artificial intelligence model can include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network calculation by performing calculation between a calculation result of a previous layer and the plurality of weight values.

[0213] Visual understanding is a technology that recognizes and processes things like human vision, including, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, three-dimensional reconstruction / positioning, or image enhancement.

[0214] Figure 5is a hardware block diagram illustrating an electronic device 500 according to an embodiment of the disclosure. The electronic device according to an embodiment of the disclosure includes at least a processor; and a memory for storing computer program instructions. When the computer program instructions are loaded and executed by the processor, the processor performs the training method of the neural network model and the image processing method as described above.

[0215] Figure 5 The electronic device 500 shown specifically includes a central processing unit (CPU) 501, a graphics processing unit (GPU) 502, and a main memory 503. These units are connected to each other through a bus 504. The central processing unit (CPU) 501 and / or the graphics processing unit (GPU) 502 can be used as the above-mentioned processor, and the main memory 503 can be used as the above-mentioned memory for storing computer program instructions. In addition, the electronic device 500 can further include a communication unit 505, a storage unit 506, an output unit 507, an input unit 508, and an external device 509, which are also connected to the bus 504.

[0216] At least one of the plurality of units can be implemented by an AI model. An AI-related function can be performed by a non-volatile memory, a volatile memory, and a processor.

[0217] The processor can include one or more processors. At this time, the one or more processors can be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a pure graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an artificial intelligence dedicated processor such as a neural processing unit (NPU).

[0218] The one or more processors control the processing of input data according to a pre-defined operation rule or an artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The pre-defined operation rule or the artificial intelligence model is provided through training or learning.

[0219] Here, the provision through learning means that a pre-defined operation rule or an artificial intelligence model of a desired feature is made by applying a learning algorithm to a plurality of learning data. The learning can be performed on a device itself in which an AI model according to an embodiment is executed, and can be implemented through a separate server / system.

[0220] The AI model can include a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation by computing an operation of a previous layer and the plurality of weights. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q-network.

[0221] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using a plurality of learning data to cause, allow, or control the target device to make a judgment or a prediction. Examples of a learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, semi-supervised learning, or reinforcement learning.

[0222] Figure 6 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the disclosure. As shown in Figure 6 The computer-readable storage medium according to the embodiment of the disclosure 600 has computer program instructions 601 stored thereon. When the computer program instructions 601 are run by a processor, the training method of a neural network model and the image recognition method according to the embodiment of the disclosure described with reference to the above figures are performed. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, an optical disc, a magnetic disc, etc.

[0223] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0224] The basic principles of the disclosure are described above in connection with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the disclosure are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the various embodiments of the disclosure must have. In addition, the above specific details of the disclosure are only for the purpose of example and for the purpose of understanding, and the above details do not limit the disclosure to the above specific details.

[0225] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0226] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0227] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0228] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0229] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An execution method by an electronic device, the method comprising: obtaining input data of a neural network and a label corresponding to the input data, the input data comprising at least one of image data, audio / speech data and text data, and the neural network comprising a plurality of network layers; determining an activation state of at least one neuron in at least one of the plurality of network layers in the neural network based on the input data and the label, wherein the neural network is configured to output a predicted label for input data without a label, wherein the activation state of a neuron comprises: activated and inactivated; and maintaining an information bottleneck of at least one activated neuron, and updating the neural network based on the input data by training the neural network.

2. The method of claim 1, wherein, There is a connection relationship between the activated neurons in the neural network and the activated neurons in the adjacent network layer.

3. The method of any one of claims 1-2, wherein, Determining the activation state of at least one neuron in at least one of the plurality of network layers in the neural network comprises: training the neural network based on a first batch of input data, identifying activated neurons and inactivated neurons in at least one of the plurality of network layers based on the information bottleneck of each of the at least one neuron.

4. The method of claim 3, further comprising: retraining the neural network based on a second batch of input data by updating the information bottleneck of the inactivated neurons while maintaining the information bottleneck of the activated neurons.

5. The method of claim 1, wherein the plurality of network layers comprises an input layer, at least one hidden layer, and an output layer.

6. The method of claim 5, further comprising: setting the corresponding information bottleneck of the inactivated neurons in the at least one hidden layer to a preset value; training the neural network based on the input data.

7. The method of claim 6, further comprising: determining the activation state of the inactivated neurons according to the trained information bottleneck of the inactivated neurons.

8. The method of claim 6, wherein, The information bottleneck of the activated neurons in the neural network remains unchanged during the training process.

9. The method of any one of claims 6-8, further comprising: determining the activation state of the inactivated neurons as activated based on the information amount corresponding to the trained information bottleneck of the inactivated neurons being greater than a preset threshold; maintaining the activation state of the inactivated neurons as inactivated based on the information amount corresponding to the trained information bottleneck of the inactivated neurons being not greater than the preset threshold.

10. The method of any one of claims 5-8, wherein, Training the neural network further comprises: training the neural network based on the information amount corresponding to the information bottleneck of each neuron included in the at least one hidden layer.

11. The method of any one of claims 5-8, wherein, Training the neural network further comprises: maintaining the first weight between the activated neurons in the at least one hidden layer and each neuron in the previous network layer corresponding to each of the at least one hidden layer unchanged during the training process, during training of the at least one hidden layer, training second weights between unactivated neurons in the at least one hidden layer and each neuron in a corresponding previous network layer for each of the at least one hidden layer, and wherein the maintaining includes resetting the first weights between activated neurons in one or more of the at least one hidden layer and unactivated neurons in the corresponding previous network layer to 0 prior to the maintaining.

12. The method of any one of claims 5-8, wherein, training the neural network further includes: based on the output layer being a multi-headed output layer: maintaining, during training, weights between activated neurons in the output layer and activated neurons in a previous network layer, maintaining, during training, weights between activated neurons in the output layer and unactivated neurons in the previous network layer, and during training of the output layer, training weights between unactivated neurons in the output layer and each neuron in the previous network layer, based on the output layer being a single-headed output layer, training, during training, weights between each neuron in the output layer and each neuron in the previous network layer, and wherein the maintaining includes resetting the weights between activated neurons in the output layer and unactivated neurons in the previous network layer to 0 prior to the maintaining.

13. The method of any of claims 5-8, further comprising: based on activation states of the at least one neuron included in the at least one hidden layer, increasing neurons in the at least one hidden layer.

14. The method of claim 13, wherein, increasing neurons includes, based on all neurons included in one of the at least one hidden layer being in an activated state, increasing neurons in the one of the at least one hidden layer.

15. The method of claim 1, wherein, the neural network is a neural network trained to minimize a loss function in a plurality of hidden layers.

16. The method of claim 1, wherein, the neural network is trained based on a regularizer term of the loss function in a manner that encourages sparsity by encouraging the use of as few neurons as possible based on the provided input.

17. The method of claim 11, determining the amount of information corresponding to the information bottleneck by the regularizer term.

18. The method of claim 17, the loss function when training the neural network includes: an accumulation of the amount of information corresponding to the information bottleneck for each neuron in a hidden layer.

19. The method of claim 18, wherein, the amount of information corresponding to the information bottleneck is an amount of information encoded by the neuron about a previous network layer.

20. The method of any of claims 5-8, further comprising at least one of: connections between activated neurons in a first layer of hidden layers and neurons of a previous network layer are maintained during training; during training, training connections between unactivated neurons in the first layer of hidden layers and neurons of the previous network layer; connections between activated neurons in a hidden layer other than the first layer and activated neurons in a previous network layer are maintained during training; connections between activated neurons in a hidden layer other than the first layer and unactivated neurons in a previous network layer are 0 and maintained during training; During the training process, connections between neurons in a hidden layer other than the first layer and neurons in the previous network layer are not activated.

21. The method of claim 17, wherein, The value of the regularizer for a neuron is determined based on a neuron-specific hyperparameter.

22. The method of claim 21, wherein the neuron-specific hyperparameter is determined by at least one of: not adjusting the hyperparameter when the neuron should be pruned and was actually pruned; multiplying the hyperparameter by a factor greater than 1 when the neuron should be pruned and was not actually pruned; resetting the hyperparameter to an initialization value when the neuron should not be pruned and was actually pruned; and multiplying the hyperparameter by a factor less than 1 when the neuron should not be pruned and was not actually pruned; when the determined neuron-specific hyperparameter is less than a minimum value of a predetermined value range, setting the neuron-specific hyperparameter to the minimum value; and when the determined neuron-specific hyperparameter is greater than a maximum value of a predetermined value range, setting the neuron-specific hyperparameter to the maximum value.

23. The method of claim 1, further comprising: obtaining unlabeled input data; outputting a predicted label for the unlabeled input data using the neural network in which the activation state of at least one neuron is determined.

24. An electronic device, comprising: at least one memory configured to store a neural network comprising a plurality of network layers, at least one of the plurality of network layers containing a plurality of neurons; at least one processor configured to perform the method of any one of claims 1-23.

25. An electronic device, comprising: a processor; a memory to store computer program instructions; wherein when the computer program instructions are loaded and run by the processor, the processor performs the method of any one of claims 1-23.

26. A computer-readable storage medium storing computer program instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1-23.

Images