A neural network system applying a center-standardized linear layer method
By introducing a centrally normalized linear layer into the neural network and imposing zero-mean and unit-length constraints, the gradient collapse and explosion problems in deep neural network training are solved, resulting in a more stable training process and better generalization performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2023-05-22
- Publication Date
- 2026-05-29
AI Technical Summary
Deep neural networks are prone to gradient collapse and explosion problems during training, which can cause the model to fail to converge effectively or diverge during training, especially on complex deep models and large-scale datasets.
A center-normalized linear layer is introduced into the linear layer of the neural network. By imposing zero-mean and unit-length constraints, the center-normalized linear layer is constructed, and the scale of weights and activation values is adjusted by gradient updates and learnable scale parameters to ensure the stable distribution of activation values.
It stabilizes the training process of neural networks, improves training efficiency, enhances the generalization ability of the model, reduces training difficulty, and avoids gradient collapse and explosion problems.
Smart Images
Figure CN116579419B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural networks, specifically relating to a neural network system that applies the centrally standardized linear layer method. Background Technology
[0002] Neural networks have achieved excellent results in various fields such as image classification, natural language processing, and speech recognition. Neural networks typically consist of a multi-layered structure, with transformations between each layer consisting of simple linear transformations with learnable weight parameters and non-linear activation functions. In the linear transformation part, each neuron is connected to the input through (learnable) weight parameters, and its output is a weighted sum of the input and weights. This multi-layered structure gives neural networks the ability to effectively learn hierarchical features. Furthermore, as the network deepens, the neural network model possesses more powerful expressive capabilities, enabling it to learn more complex functional relationships with higher levels of abstraction, thus achieving even better results.
[0003] However, while this deep hierarchical structure gives neural networks a significant advantage in learning features, it also makes training them difficult. One of the most classic problems in training neural networks is gradient collapse and exploding gradients. To effectively alleviate this problem, a fundamental approach is to carefully design the initialization method for the weight parameters of linear layers. This method can effectively ensure that the activation values of each hidden layer in the DNN are basically the same in the initial stage, greatly mitigating the gradient collapse and exploding problem in the initial stage of neural network training, thereby reducing the training difficulty of the neural network.
[0004] However, because the distribution of activation values in the hidden layers of a neural network changes during training, gradient collapse or explosion can still occur, causing the model to fail to converge effectively or suddenly diverge during training. This phenomenon is particularly pronounced in complex deep models and on large-scale datasets. Summary of the Invention
[0005] To address the aforementioned problems, this invention proposes a neural network system employing a center-normalized linear layer method. By imposing zero-mean and unit-length constraints on the linear layers of a deep neural network, a specific constraint-solving method is constructed, thereby creating a center-normalized linear layer. This type of linear layer can replace the original unconstrained linear layer in the neural network, and given input data, it obtains the desired output through function computation by the deep neural network.
[0006] The neural network system using the application center standardized linear layer method is as follows:
[0007] 1) In the linear layer of the original neural network, set the mean of the weight parameters connected to each output neuron to 0 and the length to 1, that is, apply zero mean and unit length constraints.
[0008] The linear layers include fully connected layers and convolutional layers:
[0009] For a fully connected layer: input x∈R d Where x is a d-dimensional real vector, the output value of the i-th neuron in the linear layer of the original neural network is h. i =W i x+b, i=1,2,...n; therefore, the weight parameter matrix of the fully connected layer is:
[0010] Where n is the number of neural units output by the neural network; Let be the weight parameter connected to the i-th neural unit, and b be the learnable bias parameter, which is usually initialized to 0.
[0011] The improved center-normalized linear layer requires the weight parameter W of the connection of the i-th neuron. i It is 0, that is And ||W i || = 1; where 1 is a d-dimensional vector and each element is 1.
[0012] For convolutional layer types: Input Where h and w are the height and bandwidth of the input feature map, then the weight parameters connected to the i-th neuron in the linear layer of the original neural network are... Where F h and F w This refers to the height and width of the filter.
[0013] The improved center-normalized linear layer requires the weight parameter W of the connection of the i-th neuron. i It is 0, that is, φ(W) i ) T 1 = 0 and ||φ(W) i )||=1, where φ(W i ) represents W i Transform into a d·F h ·F w A dimensional vector, 1 being d·F h ·F w A dimensional vector in which each element is 1.
[0014] 2) By updating the weight parameters of the improved center-normalized linear layer using gradients, it is ensured that the weight parameters connected to the output neurons have zero mean and unit variance.
[0015] The gradient update formula for the weight parameters connected to the i-th neuron is as follows:
[0016]
[0017] The loss function measures the difference between the output of the neural network and its true output. For the weight parameter W i The value in the t-th training iteration, For the weight parameter W i Gradient information in the t-th training iteration.
[0018] The centrally normalized linear layer uses reparameterization to ensure that the parameters are updated. It remains a vector with zero mean and unit length;
[0019] 3) Construct learnable scale parameters based on each neural unit. To adjust the scale of weights or activation values;
[0020] A learnable scale g is added to the weight parameters of the i-th neuron. i ∈R, ensuring that the length of the weight parameters of the i-th neuron during training is |g i By adjusting the scaling parameter g, we can ensure that the variance of activation values between layers of different neural network architectures remains stable, thus avoiding gradient collapse or explosion.
[0021] The advantages of this invention are:
[0022] 1. By introducing a centrally normalized linear layer into the neural network, the activation values of each neuron are more likely to have zero mean and stable variance, which stabilizes the distribution of activation values and thus stabilizes the training of the neural network.
[0023] 2. By introducing a centrally normalized linear layer into the neural network, the activation value of each neuron is more likely to have zero mean and stable variance. This can improve the conditional condition of the training loss landscape of the neural network from an optimization perspective, thus accelerating the training of deep neural networks.
[0024] 3. By introducing a centrally normalized linear layer into the neural network, the weights themselves are constrained, which can be regarded as a normalization method. Therefore, it can provide the generalization effect of the neural network. Attached Figure Description
[0025] Figure 1 Example of a neural network system including a centrally normalized linear layer
[0026] Figure 2An example of an implementation of a center-normalized linear layer based on a reparameterization method;
[0027] Figure 3 An example of a method for constructing learnable scale parameters based on each output neuron; Detailed Implementation
[0028] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.
[0029] This invention discloses a neural network system employing a center-normalized linear layer method. The center-normalized linear layer imposes zero-mean and unit-length constraints on the linear layers of the deep neural network and constructs a constraint-solving method, using reparameterization as a specific example. This type of linear layer can replace the original unconstrained linear layers in the neural network, and given input data, obtains the desired output through function computation by the deep neural network.
[0030] The centrally normalized linear layer has the following properties:
[0031] (1) It is a basic linearized layer of the neural network, and the weight parameters connected to each output neuron have a mean of 0 and a length of 1. This layer can replace the linear layer of the original neural network.
[0032] The basic linearization layer refers to the linear transformation layer of the neural network, which includes fully connected layers and convolutional layers:
[0033] For a fully connected layer: input x∈R d Where x is a d-dimensional real vector, and let the number of output neurons in the neural network be n, then the output value of the i-th neuron in the linear layer of the original neural network is h. i =W i x+b, i=1,2,...n; therefore, the weight parameter matrix of the fully connected layer is:
[0034] W i ∈R d Let be the weight parameter connected to the i-th neural unit, and b be the learnable bias parameter, which is usually initialized to 0.
[0035] The improved center-normalized linear layer requires the weight parameter W of the connection of the i-th neuron. i With a mean of 0 and a unit variance, i.e. And ||W i|| = 1; where 1 is a d-dimensional vector and each element is 1.
[0036] The centrally standardized linear layer can be viewed as a linear layer with zero-mean weight vectors and unit length constraints.
[0037] For convolutional layer types: Input Where h and w are the height and bandwidth of the input feature map; then the weights (filters) connected to the i-th neuron in the linear layer of the original neural network are... Where F h and F w This refers to the height and width of the filter.
[0038] The improved center-normalized linear layer requires the weight parameter W of the connection of the i-th neuron. i With a mean of 0 and a unit variance, φ(W) i ) T 1 = 0 and ||φ(W) i )||=1, where φ(W i ) represents W i Transform into a d·F h ·F w A dimensional vector, 1 being d·F h ·F w A dimensional vector in which each element is 1.
[0039] This expression demonstrates that the processing method is consistent for both convolutional and fully connected layers, except that for convolutional layers, only the operation φ() is needed to transform the tensor form of the convolution parameters into the vector form of the fully connected layers. Therefore, the following descriptions will be based on fully connected layers, but the same applies to convolutional layers.
[0040] (2) The centrally normalized linear layer can effectively adopt gradient-based training methods, and after parameter updates, it can not only ensure the zero-mean unit length property of the weight parameters connected to the output neurons of the centrally normalized linear layer, but also retain the backpropagation property of the neural network.
[0041] Gradient update methods include stochastic gradient descent (such as SGD) and gradient descent with adaptive learning rate (such as Adam). These gradient-based update methods calculate the gradient information of the linear layer and update the corresponding weight parameters using gradients. The formula for updating the gradient of the weight parameters connected to the i-th neuron is as follows:
[0042]
[0043] The loss function measures the difference between the output of the neural network and its true output. For the weight parameter W i The value in the t-th training iteration, For the weight parameter W i Gradient information in the t-th training iteration.
[0044] like Figure 2 As shown, the centrally normalized linear layer uses reparameterization to ensure that the parameters are updated... Still a zero-mean, unit-length vector; the weight parameters connected to each output neuron. A corresponding proxy parameter V was maintained. i ∈R d .
[0045] Before each input activation is processed, calculate This calculation guarantees And ||W i ||=1. W is obtained through the above calculation. i The central normalized linear layer undergoes a conventional linear transformation to calculate the output activation.
[0046] During the backpropagation gradient calculation process in training, the central normalization layer receives the backpropagation gradient information. The central normalized layer computes the gradient of its activation input x. and the gradient of its weight parameter W
[0047] E(·) represents the expected value calculated on batch training data;
[0048] Furthermore, the central normalization layer emphasizes gradient descent on the surrogate parameter V to stabilize training and ensure the properties of W.
[0049] Specifically, the weight W connected to the i-th output neuron is given. i gradient information calculate The central standardization layer updates the proxy parameters, namely: For V i The value in the t-th training iteration, For V i Gradient information in the t-th training iteration.
[0050] (3) For different neural network architectures, this layer can construct learnable scale parameters based on each output neuron. This is used to adjust the scale of weights or activation values to ensure that the variance of activation values between different layers of the neural network remains equal during training.
[0051] like Figure 3As shown, a learnable scale g is added to the weight parameters of the i-th neuron. i ∈R, ensuring that the length of the weight parameters of the i-th neuron during training is |g i The main reason for this design is to ensure that the variance of activation values between layers of different neural network architectures (such as feedforward neural networks and neural networks with residuals) remains stable by adjusting the scale parameter g, thus avoiding gradient collapse or explosion.
[0052] The centrally normalized linear layer can effectively ensure that the activation values of each hidden layer in the neural network are basically the same, which can alleviate the gradient collapse and explosion problems in neural network training, thereby reducing the training difficulty of the neural network.
[0053] The innovative aspects of this invention can be specifically implemented using the following methods:
[0054] A corresponding operating method including the centrally normalized linear layer; a computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the centrally normalized linear layer; and a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method for training the centrally normalized linear layer.
[0055] A system for one or more computers configured to perform a specific operation or action at a standardized linear layer means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, causes the system to perform the operation or action. A system for one or more computer programs configured to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0056] Example:
[0057] A neural network system 10 containing a central normalized linear layer, such as Figure 1 As shown, the neural network system 10 is an example of a system implemented as a computer program on one or more computers, wherein the modules, components, systems, and techniques described below can be implemented. The neural network system 10 receives a neural network input 11, processes the neural network input 11 through each layer within the neural network, and generates a neural network output 16 based on this. The neural network system 10 can be configured to receive images, text, or features extracted for any type of data as input, and generate a classification output or a continuous numerical vector based on that input.
[0058] For example, in an image classification task, the network input is an image or features extracted from an image, and the network output is a score for each class in the set of classes, where each score represents the likelihood (or probability) that the image belongs to that class.
[0059] For an image-based object detection task, the input 11 of the neural network system 10 is an image or features extracted from an image. The output 16 calculated by the neural network system 10 for a given image is the location of multiple objects and a score for each class in a set of categories, where each score represents the probability (or likelihood) that the image belongs to that category. The location of the object is a 4-dimensional continuous vector, including center point coordinates (x, y), width w, and height h.
[0060] For a text classification task, the network input 11 is a sequence of words, and the output 16 calculated by the neural network system 10 for a given text is a score for each class in the category set, where each score represents the probability (or likelihood) that the text belongs to that category.
[0061] For the machine translation task, the network input 11 is text in one language, and the network output 16 is a score for each of the sets of text segments in another language, where each score represents the estimated probability that the text segment in the other language is an appropriate translation of the input text into the other language.
[0062] Specifically, each layer of the neural network system 10 receives an input, either neural network input 11 or an output activation generated by another layer in the sequence, and generates an output activation from that input. Some or all of the neural network layers in the sequence generate outputs from the inputs based on the current values of the parameter set of that neural network layer. For example, a linear layer might use a matrix of the received input multiplied by the current parameter values as part of the output generated from the received input.
[0063] Some or all of the layers in the sequence of neural network layers are centrally normalized linear layers 13, such as... Figure 1 As shown, the neural network system 10 includes a centrally normalized linear layer 13 located between neural network layer A12 and neural network layer B15 in the sequence of neural network layers.
[0064] An input layer configured to receive training input from the network;
[0065] A centrally normalized linear layer that receives input activation generated by a layer preceding the centrally normalized linear layer in the sequence; the layer preceding the centrally normalized linear layer is a non-linear layer, a normalized layer, or another centrally normalized linear layer.
[0066] During the processing of neural network input 11 by neural network system 10, center-normalized linear layer 13 receives the layer A output activation generated by neural network layer A12, processes the layer A output activation to generate center-normalized linear layer 13 output activation 14, and then provides the center-normalized linear layer 13 output activation as input to neural network layer B15. Typically, center-normalized linear layer 13 requires that the mean of the weight parameters connected to each of its output neurons 14 be 0 and its length be 1. This layer can replace the linear layer of the original neural network.
[0067] The received activations are processed according to the centrally normalized linear layer parameter set to generate output activations;
[0068] Furthermore, such as Figure 2 The diagram illustrates a flowchart of an example process for processing input activation using a center-normalized linear layer, performed by a system of one or more computers located in one or more locations. The system obtains training data for training a neural network consisting of a sequence of neural network layers, each including at least one center-normalized linear layer. Typically, the training data includes multiple training inputs and a corresponding known output for each training input. Given a training input, the neural network's output should be the corresponding known output; if discrepancies exist, gradient information can be computed via backpropagation based on a defined loss function L. The system trains the neural network on the training data using gradient descent to determine the values of the corresponding parameters of the center-normalized linear layer, as well as the values of the parameters of other neural network layers in the sequence.
[0069] The center-normalized linear layer receives input activations generated by the layers preceding it in the sequence of layers. The center-normalized linear layer uses its own parameter set to process the input activations to generate the output activations.
[0070] And the neural network layer that provides the output activation as input to the centrally normalized linear layer in the sequence.
[0071] The embodiments of the neural network systems and functional operations described in this specification may be implemented in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. The embodiments of the neural network systems and functional operations described in this specification may be implemented as one or more computer programs. A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
Claims
1. A neural network system applying the center-normalized linear layer method, characterized in that: It includes several center-normalized linear layers, which impose zero-mean and unit-length constraints on the linear layers of the original neural network, and construct a constraint-solving method with reparameterization as a specific example; given the input data, the center-normalized linear layers are processed by the function of the deep neural network to obtain the desired output; The workflow of a neural network system is as follows: Step 1: Input image and text data, or feature data obtained after feature extraction of image and text data; Step 2: Each center-normalized linear layer in the neural network system receives the input activation generated by the previous layer, and uses its center-normalized weight parameters to perform a linear transformation on the activation value. The output result is used for calculation by the layers connected to it. At the same time, the center-normalized linear layer receives the gradient information backpropagated from the layers connected to it, performs backpropagation within it to obtain the gradient information of its weight parameters, and further backpropagates it to the previous layers, finally obtaining the gradient information of all weight parameters of the neural network system. The forward and backward propagation calculations for the centrally normalized linear layer are as follows: 1) In the linear layer of the original neural network, set the mean of the weight parameters connected to each output neuron to 0 and the length to 1, that is, apply zero mean and unit length constraints. Specifically: the input to the central normalized linear layer is a d-dimensional real vector. The output is If this layer has n output neurons, then the weight parameter matrix of the central normalized linear layer is: Deviation weight parameter ; In its forward computation expression, the first The output value of each neural unit is The centrally standardized linear layer imposes the following constraints on the weight parameter matrix: and ,in ; 2) By updating the weight parameters of the improved center-normalized linear layer using gradients, it is ensured that the weight parameters connected to the output neurons have zero mean and unit variance. First, when backpropagating to calculate gradients in the center-normalized linear layer during training, the center-normalized layer receives the backpropagation gradient information. ; Let n be the loss function and n be the number of output neurons. This refers to the output value of the output neural unit; Then, the central normalization layer calculates its activation input. gradient and its weight parameters gradient in This indicates that the expected value is calculated on the batch training data; For dimension values; Next, the central standardization layer emphasized the proxy parameters. Gradient descent is used to stabilize training and ensure the weight parameters are maintained. The attributes it possesses; The weight parameters connected to each output neuron A corresponding proxy parameter was maintained. ; Before each input activation is processed, calculate ,in Given a d-dimensional vector with each element equal to 1, this calculation guarantees... and ; Regarding the first The weights connected to each output neuron gradient information The calculation formula is: The central standardization layer updates the proxy parameters, namely: , for The value in the t-th training iteration, for Gradient information in the t-th training iteration, ; 3) Construct learnable scale parameters based on each neural unit. To adjust the scale of weights or activation values; Regarding the first The weight parameters of each neuron are added with a learnable scale. To ensure the first step in the training process The length of the weight parameters of each neuron is By adjusting the scale parameters This ensures that the variance of activation values between layers in different neural network architectures remains stable, avoiding gradient collapse or explosion. Step 3: Using gradient information, the neural network system uses gradient descent to update the values of the corresponding weight parameters of the central standardized linear layer, as well as the values of the weight parameters of other layers in the neural network, thereby completing the training process. Step four: The trained neural network system outputs either a classification output of image or text data, or a continuous numerical vector. For image classification tasks, the output is a score for each class in the set of classes, where each score represents the probability or likelihood that the image belongs to that class; For image-based object detection tasks, the output is the location of multiple objects and a score for each class in a set of categories, where each score represents the probability or likelihood that the image belongs to that class; For text classification tasks, the output is a score for each class in the set of categories, where each score represents the probability or likelihood that the text belongs to that category.
2. The neural network system using the center-normalized linear layer method as described in claim 1, characterized in that: The layer preceding the centrally normalized linear layer is a nonlinear layer, a normalized layer, or another centrally normalized linear layer.
3. The neural network system using the center-normalized linear layer method as described in claim 1, characterized in that: The linear layer includes fully connected layers and convolutional layers: For fully connected layers: Input ,in Let be a d-dimensional real vector, then the th... The output value of each neural unit is ; Therefore, the weight parameter matrix of the fully connected layer is ,in For the first Weight parameters for each neural unit connection This is a learnable bias parameter, typically initialized to 0; Improved center-normalized linear layer requirements Weight parameters of each neuron connection It is 0, that is and ;in It is a d-dimensional vector with each element being 1; For convolutional layer types: Input Where h and w are the height and bandwidth of the input feature map, then the first linear layer of the original neural network... The weight parameters connected to each neural unit ,in and For the height and width of the filter; Improved center-normalized linear layer requirements The weight parameters connected to each neural unit It is 0, that is: and ;in, express Transform into one 3D vector for A dimensional vector in which each element is 1.