Combined standardization layer module for improving robustness of neural network system
By alternately using batch standardization layer and non-batch standardization layer in neural networks, the combined standardization layer module is solved, and the problem of neural network not being robust when input data distribution differences or small batch data is achieved, faster training convergence is achieved and better robustness.
Patent Information
- Application Number
- CN202411468972.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-14
AI Technical Summary
Existing neural networks are not robust when facing differences in input data distribution or small batch data, and the inconsistency of batch standardization layers during training and testing leads to poor results.
A combined standardization layer module is proposed, by alternately using batch standardization layer and non-batch standardization layer, the specific structure is "linear layer-batch standardization layer-nonlinear layer-linear layer-non-batch standardization layer-nonlinear layer" to improve The robustness of neural networks.
This combined standardization layer module can accelerate neural network training while improving its robustness, especially when small batch data or data distributions vary greatly, it has better results and is simple and convenient.
Smart Images
Figure CN119443184A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of neural networks, and in particular relates to a combined normalization layer module for improving the robustness of a neural network system. Background Art
[0002] Neural networks have achieved good application results in many fields such as image classification, natural language processing and speech recognition. However, they are currently facing challenging problems such as non-robust model prediction. For example, neural networks require that the distribution of test data and training data is consistent. If this requirement is not met, there will be a significant performance drop in prediction. In addition, adding a little disturbance to the input of the neural network may cause a significant change in the output of the network.
[0003] The existing technology method to alleviate the prediction non-robustness of neural networks usually uses a normalization layer. By performing a normalization operation on the hidden layer of the neural network, it ensures that the output of the hidden layer of the neural network is within a certain range, alleviating the problem of very large changes in the neural network due to differences in the distribution of input data or disturbances in the input data. This can not only alleviate the prediction non-robustness problem of the neural network model, but also help accelerate the training convergence of the neural network.
[0004] The most famous of these is the batch normalization layer (BN). However, BN itself has a problem: it uses batch statistics during training, but requires the use of population statistics to be estimated during testing. This inconsistency between training and testing makes BN very ineffective when there are small batches of data or when the distribution of training and test data is very different.
[0005] Previous researchers have also proposed some normalization layers with consistent expressions during training and testing, such as layer normalization (LN) and group normalization (GN). Although these methods can achieve good results when BN is not applicable, they are usually inferior to BN.
[0006] Therefore, it is of great significance to improve the robustness of neural network models in a wider range of scenarios. Summary of the invention
[0007] In view of the above problems, the present invention proposes a combined normalization layer module for improving the robustness of a neural network system. The combined normalization layer module can effectively utilize the advantages of BN in accelerating neural network training by alternately using batch normalization layers (BN) and batch-free normalization layers (BFN) in a deep neural network, and alleviate its disadvantage of poor effect in the case of small batch data or large differences in the distribution of training and test data, thereby improving the robustness of the deep neural network.
[0008] The combined normalization layer module for improving the robustness of the neural network system is a sequence of "linear layer-batch normalization layer (BN)-nonlinear layer-linear layer-non-batch normalization layer (BFN)-nonlinear layer". When applied to multilayer perceptron (MLP) and convolutional neural network (CNN), the input of the combined normalization layer module is different, so the calculation process is different, as follows:
[0009] 1. For the input of the multi-layer perceptron, the calculation process of the combined normalization layer module is as follows:
[0010] (1) Calculate the output of the linear layer:
[0011] For the fully connected layer: input Where x is a d-dimensional real vector, represents an n-dimensional real vector space.
[0012] Then the output of the fully connected layer of the original neural network is h=Wx+b, W is the weight parameter matrix of the fully connected layer, and b is the learnable bias parameter.
[0013] (2) Calculate the output of the batch normalization layer (BN):
[0014] Assume the batch size is m, then the input of BN is Where x consists of m d-dimensional real vectors; the j-th dimension of the i-th sample is normalized to get the output:
[0015]
[0016] The vector after normalizing all dimensions of all samples As the output of BN.
[0017] in and are the mean and variance of the m samples in the jth dimension, ∈ is a selected fixed minimum value, usually around 10 -5 -10 -8 between.
[0018] (3) Calculate the output of the nonlinear layer:
[0019] For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained
[0020] max(0, x) means taking the larger value between 0 and x as the result.
[0021] (4) Calculate the output of the non-batch normalization layer (BFN):
[0022] The main non-batch normalization layers include layer normalization (LN) and group normalization (GN).
[0023] 1) For LN: Input Normalize the sample's ,th dimension and get the output:
[0024]
[0025] The vector after all dimensions of the sample are normalized As the output of LN;
[0026] in and are the mean and variance of the current sample in all dimensions respectively;
[0027] 2) For GN: Input Divide the number of dimensions d into g groups, each containing k dimensions, where d = k × g; normalize the jth dimension in the i-th group and get the output:
[0028]
[0029] The vector after normalizing all dimensions of all groups As the output of GN.
[0030] in and are the mean and variance of the current sample in all dimensions of the i-th group.
[0031] 2. For the input of the convolutional neural network, the calculation process of the combined normalization layer module is as follows:
[0032] (1) Calculate the output of the linear layer:
[0033] For the convolutional layer form: input Where h and w are the height and width of the input feature map, d is the number of channels of the feature map, that is, the dimension; let the height and width of the filter be F h and F w , the step size of the convolution operation is s, and the padding is p, then the output of each channel feature map is in
[0034] (2) Calculate the output of the batch normalization layer (BN):
[0035] Assume the batch size is m, then input Where x consists of m feature maps of dimension d×h×w, and the (k, l)th dimension of the jth channel of the i-th sample is normalized to get the output:
[0036]
[0037] The vector after normalizing all dimensions of all samples As the output of BN.
[0038] in and are the mean and variance of m samples on the jth channel respectively.
[0039] (3) Calculate the output of the nonlinear layer:
[0040] For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained
[0041] (4) Calculate the output of the non-batch normalization layer (BFN):
[0042] The main non-batch normalization layers include layer normalization (LN) and group normalization (GN).
[0043] 1) For LN: Input Normalize the (j, k)th dimension of the i-th channel and get the output:
[0044]
[0045] The vector after normalizing all dimensions of all channels As the output of LN.
[0046] in and are the mean and variance of the current sample on all channels respectively.
[0047] 2) For GN: Input Divide the number of channels d into g groups, each containing k features, where d = k × g; normalize the (i, j)th dimension of the cth channel of group G to get the output:
[0048]
[0049] The vector after normalizing all dimensions of all groups As the output of GN.
[0050] in and are the mean and variance of the current sample group G in all dimensions of width and height, respectively.
[0051] The combined normalization layer module for improving the robustness of the neural network system can be realized by directly connecting several combined normalization layer modules in series when used to construct a forward neural network, specifically including: a residual neural network using a combined normalization layer module in the form of Basic Block and BottleNeck;
[0052] (1) For the residual neural network using Basic_Block, the residual-combination normalization layer module in Basic_Block form is described as follows:
[0053] 1) BasicBlock_XBNBlock_P1 replaces the first BN in the original residual block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BFN-nonlinear layer-convolutional layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0054] Vector input form for convolutional neural networks For the residual connection part, the input x of the module is saved in the residual r at the starting point of the residual branch, that is, r=x; and the residual is directly transmitted to the end point of the residual branch. Therefore, the output y=y0+r obtained at the end point of the residual branch, where y0 represents the output obtained by the BN or downsampling layer before the end point of the residual branch.
[0055] For the optional downsampling layer, a 1×1 convolution layer and a BN are included to adjust the number of channels and spatial size.
[0056] 2) BasicBlock_XBNBlock_P2 (hereinafter referred to as Basic_XBNBlock) replaces the second BN in the original residual block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolution layer-BN-nonlinear layer-convolution layer-BFN-optional downsampling layer-residual branch end point-nonlinear layer".
[0057] (2) For the residual neural network using BottleNeck, the residual-combination normalization layer module in BottleNeck form is described as follows:
[0058] 1) Bottleneck_XBNBlock_P1 replaces the first BN in the original bottleneck block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BFN-nonlinear layer-convolutional layer-BN-nonlinear layer-convolutional layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0059] 2) Bottleneck_XBNBlock_P2 (hereinafter referred to as XBNBlock) replaces the second BN in the original bottleneck block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolution layer-BN-nonlinear layer-convolution layer-BFN-nonlinear layer-convolution layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0060] 3) Bottleneck_XBNBlock_P3 replaces the third BN in the original bottleneck block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BN-nonlinear layer-convolutional layer-BN-nonlinear layer-convolutional layer-BFN-optional downsampling layer-residual branch end point-nonlinear layer".
[0061] The advantages of the present invention are:
[0062] 1. A combined normalization layer module that improves the robustness of the neural network system. It has the advantages of BN and can reduce the distribution differences and disturbances of the input data to a certain extent, so that the neural network can converge faster during training. In general, it has better prediction effects and stronger robustness than BFN.
[0063] 2. A combined normalization layer module that improves the robustness of the neural network system. Since the expressions of BN are inconsistent during training and testing, while those of BFN are the opposite, BFN is more applicable and has better effects than BN in some cases.
[0064] 3. A combined normalization layer module for improving the robustness of a neural network system. It only requires the alternating use of BN and BFN. The implementation method is simple, and since it is a complete module, it can be plug-and-play in a deep neural network, which is more convenient and quick.
[0065] 4. A combined normalization layer module for improving the robustness of a neural network system. By introducing a combined normalization layer module into a neural network system, the advantages of BN in accelerating neural network training can be effectively utilized, and BFN can potentially hinder the accumulation of BN estimation offsets in the network, thereby alleviating the disadvantage that BN has poor effects in small batches of data or when the distribution of training and test data is very different, thereby improving the robustness of deep neural networks in a wider range of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 This is an example of a combined normalization layer module for improving the robustness of a neural network system according to the present invention;
[0067] Figure 2 This is an example of a residual-combined normalization layer module (Basic XBNBlock) in the form of a Basic Block of the present invention;
[0068] Figure 3 This is an example of a BottleNeck form of residual-combined normalization layer module (XBNBlock) of the present invention;
[0069] Figure 4 An example of a ResNet-18 (ResNet-XBNBlock-18) neural network system including Basic XBNBlock of the present invention;
[0070] Figure 5 An example of a ResNeXt-50 (ResNeXt-XBNBlock-50) neural network system including XBNBlock according to the present invention; DETAILED DESCRIPTION
[0071] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.
[0072] The present invention discloses a combined normalization layer module for improving the robustness of a neural network system, specifically comprising a combined normalization layer module composed of a sequence of "linear layer-batch normalization layer (BN)-nonlinear layer-linear layer-non-batch normalization layer (BFN)-nonlinear layer", and a neural network system using the combined normalization layer module;
[0073] like Figure 1As shown in the figure, the combined normalization layer module can be applied to multilayer perceptron (MLP) and various convolutional neural networks (CNN). Due to the different inputs of the combined normalization layer module, the calculation process is different, as follows:
[0074] 1. For the input of the multi-layer perceptron, the calculation process of the combined normalization layer module is as follows:
[0075] (1) Calculate the output of the linear layer:
[0076] For the fully connected layer: input Where x is a d-dimensional real vector, represents an n-dimensional real vector space.
[0077] Then the output of the fully connected layer of the original neural network is h = Wx + b, is the weight parameter matrix of the fully connected layer, b is the learnable bias parameter, and the output of the connection layer is h∈Rn.
[0078] (2) Calculate the output of the batch normalization layer (BN):
[0079] Assume the batch size is m, then the input of BN is Where x consists of m d-dimensional real vectors; the information of all samples in the batch under the current dimension is integrated, and the j-th dimension of the i-th sample is normalized to obtain the output:
[0080]
[0081] The vector after normalizing all dimensions of all samples As the output of BN.
[0082] in and are the mean and variance of the m samples in the jth dimension, ∈ is a selected fixed minimum value, usually around 10 -5 -10 -8 to prevent division by 0.
[0083] (3) Calculate the output of the nonlinear layer:
[0084] For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained
[0085] max(0, x) means taking the larger value between 0 and x as the result.
[0086] (4) Calculate the output of the non-batch normalization layer (BFN):
[0087] The main non-batch normalization layers include layer normalization (LN) and group normalization (GN).
[0088] 1) For LN: Input Combining all dimensional information of the current sample, normalize the j-th dimension of the sample and get the output:
[0089]
[0090] The vector after all dimensions of the sample are normalized As the output of LN;
[0091] in and are the mean and variance of the current sample in all dimensions respectively;
[0092] 2) For GN: Input Divide the number of dimensions d into g groups, each group contains k dimensions, where d = k × g; integrate all the dimensional information of the current sample in the current group, normalize the jth dimension in the i-th group, and get the output:
[0093]
[0094] The vector after normalizing all dimensions of all groups As the output of GN.
[0095] in and are the mean and variance of the current sample in all dimensions of the i-th group.
[0096] 2. For the input of the convolutional neural network, the calculation process of the combined normalization layer module is as follows:
[0097] (1) Calculate the output of the linear layer:
[0098] For the convolutional layer form: input Where h and w are the height and width of the input feature map, and d is the number of channels, i.e., the dimension, of the feature map. Then the weight parameter connected to the i-th neural unit in the linear layer of the original neural network is Assume the height and width of the filter are F h and F w , the stride of the convolution operation is s, and the padding is p, then the output of each channel feature map is in
[0099] (2) Calculate the output of the batch normalization layer (BN):
[0100] Assume the batch size is m, then input Where x consists of m feature maps of dimension d×h×w. The information of the current dimensions in height and width of all samples in the batch is integrated, and the (k, l)th dimension of the jth channel of the i-th sample is normalized to obtain the output:
[0101]
[0102] The vector after normalizing all dimensions of all samples As the output of BN.
[0103] in and are the mean and variance of m samples on the jth channel respectively.
[0104] (3) Calculate the output of the nonlinear layer:
[0105] For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained
[0106] (4) Calculate the output of the non-batch normalization layer (BFN):
[0107] The main non-batch normalization layers include Layer Normalization (LN) and Group Normalization (GN).
[0108] 1) For layer normalization (LN): Input By integrating the information of all dimensions of the current sample in width and height, the (j, k)th dimension of the i-th channel is normalized to obtain the output:
[0109]
[0110] The vector after normalizing all dimensions of all channels As the output of LN.
[0111] in and are the mean and variance of the current sample on all channels respectively.
[0112] 2) For group normalization (GN): Input Divide the number of channels d into g groups, each containing k features, where d = k × g; integrate all dimensional information of the current sample and the current group in width and height, normalize the (i, j)th dimension of the cth channel of group G, and get the output:
[0113]
[0114] The vector after normalizing all dimensions of all groups As the output of GN.
[0115] in and are the mean and variance of the current sample group G in all dimensions of width and height, respectively.
[0116] The combined normalization layer module for improving the robustness of the neural network system can be realized by directly connecting several combined normalization layer modules in series when used to construct a forward neural network. The present invention further proposes a method of using the combined normalization layer module in a residual neural network. For the residual neural network, a residual-combined normalization layer module is proposed, which specifically includes: a residual neural network using a combined normalization layer module in the form of Basic Block and BottleNeck;
[0117] (1) For residual neural networks using Basic Block, such as ResNet-18 and ResNet-34, the residual-combination normalization layer module in BasicBlock form is described as follows:
[0118] 1) BasicBlock_XBNBlock_P1 replaces the first BN in the original residual block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-3×3 convolution layer-BFN-nonlinear layer-3×3 convolution layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0119] Vector input form for convolutional neural networks The convolutional layers, BFN, nonlinear layers, and BN in the module are calculated in the same way as those introduced in the convolutional neural network.
[0120] For the residual connection part, the input x of the module is saved in the residual r at the starting point of the residual branch, that is, r=x; and the residual r is directly transmitted to the end point of the residual branch. Therefore, the output y=y0+r obtained at the end point of the residual branch, where y0 represents the output obtained by the BN or downsampling layer before the end point of the residual branch.
[0121] The optional downsampling layer refers to the actual neural network, which determines whether to process the dimension of the vector, i.e. downsampling, based on whether the vector dimension matches. It contains a 1×1 convolution layer and a BN to adjust the number of channels and spatial size.
[0122] 2) BasicBlock_XBNBlock_P2 (hereinafter referred to as Basic_XBNBlock) Figure 2 As shown in the figure, the second BN in the original residual block is replaced by BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-3×3 convolution layer-BN-nonlinear layer-3×3 convolution layer-BFN-optional downsampling layer-residual branch end point-nonlinear layer".
[0123] The calculation method of each layer is the same as BasicBlock_XBNBlock_P1, the only difference is that the positions of BN and BFN are swapped.
[0124] (2) For residual neural networks using BottleNeck, such as ResNet-50 and ResNet-101, the BottleNeck-based residual-combination normalization layer module is described as follows:
[0125] 1) Bottleneck_XBNBlock_P1 replaces the first BN in the original bottleneck block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-1×1 convolution layer-BFN-nonlinear layer-3×3 convolution layer-BN-nonlinear layer-1×1 convolution layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0126] Vector input form for convolutional neural networks The residual branch starting point, convolution layer, BFN, nonlinear layer, BN, optional downsampling layer, and residual branch end point in the module are all calculated in the same way as the convolutional neural network in the previous article.
[0127] 2) Bottleneck_XBNBlock_P2 (hereinafter referred to as XBNBlock) Figure 3 As shown in the figure, the second BN in the original bottleneck block is replaced by BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-1×1 convolution layer-BN-nonlinear layer-3×3 convolution layer-BFN-nonlinear layer-1×1 convolution layer-BN-optional downsampling layer-residual branch end point-nonlinear layer".
[0128] The calculation method of each layer is the same as Bottleneck_XBNBlock_P1, the only difference is that the positions of the first BN and BFN are swapped.
[0129] 3) Bottleneck_XBNBlock_P3 replaces the third BN in the original bottleneck block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-1×1 convolution layer-BN-nonlinear layer-3×3 convolution layer-BN-nonlinear layer-1×1 convolution layer-BFN-optional downsampling layer-residual branch end point-nonlinear layer".
[0130] The calculation method of each layer is the same as Bottleneck_XBNBlock_P1, the only difference is that the positions of the second BN and BFN are swapped.
[0131] The present invention uses a residual neural network system with a combined normalization layer module, including ResNet-18 using Basic_XBNBlock (hereinafter referred to as ResNet-XBNBlock-18, such as Figure 4 ) and ResNeXt-50 using XBNBlock (hereinafter referred to as ResNeXt-XBNBlock-50, as shown in Figure 5 As shown in the figure, the specific network structure is described as follows:
[0132] 1) ResNet-XBNBlock-18 replaces all residual blocks in the original ResNet-18 with Basic_XBNBlock. The neural network is a sequence of multiple layers consisting of "7×7 convolution layer-BN-nonlinear layer-max pooling layer-4 groups of residual block layers using Basic_XBNBlock-average pooling layer-optional BN-optional random deactivation layer (dropout layer)-fully connected layer".
[0133] Vector input form for convolutional neural networks The convolutional layers, BN, and nonlinear layers in the module are calculated in the same way as described in the above convolutional neural network, and the fully connected layers are calculated in the same way as described in the above multilayer perceptron.
[0134] For the maximum pooling layer, it is used to reduce the spatial size of the feature map. Assume that the input Assuming the pooling window size is p×p and the step size is s, for each channel k and each position (i, j) in the feature map Fk, calculate the maximum value within the window [i, i+p)×[j, j+p):
[0135] P k (i, j) = max i<m＜i+p max j≤n<j+p F k (m, n)
[0136] Among them, F krepresents the feature map of the kth channel, m and n represent the index of the window in the height and width directions respectively. The final output in
[0137] For the 4 groups of residual block layers using Basic_XBNBlock, each group of residual block layers consists of 2 Basic_XBNBlock residual blocks, and the 2 residual blocks of the first group do not contain downsampling layers, the first residual block of the other groups contains a downsampling layer, and the second residual block does not contain a downsampling layer.
[0138] For the average pooling layer, it is also used to reduce the spatial size of the feature map. Assume that the input Assume that the pooling window size is p×p, the step size is s, for each channel k, and the feature map F k For each position (i, j) in , calculate the average value within the window [i, i+p)×[j, j+p):
[0139]
[0140] For the random inactivation layer (dropout layer), the output of some neurons in the input is randomly set to 0 during the training phase, which can prevent the network from overfitting the training data and increase the generalization ability of the model. However, this layer is not effective during the test phase. The dimensions of the input vector and the output vector are the same.
[0141] 2) ResNeXt-XBNBlock-50 replaces all residual blocks in the original ResNeXt-50 with XBNBlock. The neural network is a sequence of multiple layers consisting of "7×7 convolution layer-BN-nonlinear layer-max pooling layer-4 groups of residual block layers using XBNBlock-average pooling layer-fully connected layer".
[0142] Vector input form for convolutional neural networks The BN, nonlinear layer, maximum pooling layer, average pooling layer, and fully connected layer in the module are calculated in the same way as described in ResNet-XBNBlock-18.
[0143] For the convolutional layer, grouped convolution is used. Assume the input Assume that the number of groups is G and the filter size is K × K. If the input channels are grouped evenly, the number of input channels in each group is For the g-th group (g from 0 to G-1), the convolution operation is expressed as
[0144] Among them g,ij is the value of the g-th group output feature map at position (h, w), i g,ijis the g-th group of input feature maps, w g,ij is the filter used by the gth group; assuming the step size of the convolution operation is s and the padding is p, the output of each channel feature map is in
[0145] For the 4 groups of residual block layers using XBNBlock, they are composed of 3, 4, 6, and 3 XBNBlock bottleneck blocks respectively. And the first bottleneck block of all groups contains a downsampling layer, while the other bottleneck blocks do not contain a downsampling layer.
[0146] The various innovative contents of the present invention can be specifically implemented by the following items:
[0147] A method for corresponding operation of the combined normalization layer module and a neural network system using the combined normalization layer module; a computer storage medium encoded with instructions, which, when executed by one or more computers, causes the one or more computers to execute the corresponding operation of the combined normalization layer module and the neural network system using the combined normalization layer module; and a system, which includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, causes the one or more computers to execute the corresponding operation of the method for training the combined normalization layer module and the neural network system using the combined normalization layer module.
[0148] A system of one or more computers configured to perform specific operations or actions of the combined normalization layer module and the neural network system applying the combined normalization layer module means that the system has installed thereon software, firmware, hardware or a combination thereof, which in operation enables the system to perform the operations or actions. One or more computer programs configured to perform specific operations or actions means that the one or more programs include instructions that, when executed by a data processing device, enable the device to perform the operations or actions.
[0149] Example:
[0150] like Figure 1As shown, the combined normalization layer module 1 is an example of a system implemented as a computer program on one or more computers, in which the modules, components, systems and techniques described below can be implemented. The combined normalization layer module 1 receives the combined normalization layer input 10, and processes the combined normalization layer input 10 through the linear layer 11, batch normalization layer (BN) 12, non-linear layer 13, linear layer 14, non-batch normalization layer (BFN) 15, and non-linear layer 16 inside the neural network in sequence, and generates a combined normalization layer output 17 based on this. The combined normalization layer module 1 can receive a vector in the hidden layer as an input, and generate a discrete numerical output or a continuous numerical vector based on the input.
[0151] A residual-combination normalization layer module (Basic_XBNBlock)2 in the form of Basic Block, such as Figure 2 As shown, Basic_XBNBlock module 2 is an example of a system implemented as a computer program on one or more computers in which the modules, components, systems and techniques described below can be implemented. Basic_XBNBlock module 2 receives Basic_XBNBlock input 20, processes Basic_XBNBlock input 20 through each layer inside the neural network, and generates Basic_XBNBlock output 29 based thereon. Basic_XBNBlock module 2 can receive a vector in a hidden layer as an input, and generate a discrete numerical output or a continuous numerical vector based on the input.
[0152] A BottleNeck form of residual-combination normalization layer module (XBNBlock)3, such as Figure 3 As shown, XBNBlock module 3 is an example of a system implemented as a computer program on one or more computers, in which the modules, components, systems and techniques described below can be implemented. XBNBlock module 3 receives XBNBlock input 300, processes XBNBlock input 300 through each layer inside the neural network, and generates XBNBlock output 312 based on it. XBNBlock module 3 can receive a vector in a hidden layer as an input, and generate a discrete numerical output or a continuous numerical vector based on the input.
[0153] A ResNet-18 (ResNet-XBNBlock-18) neural network system 4 including Basic_XBNBlock, such as Figure 4As shown, the neural network system 4 is an example of a system implemented as a computer program on one or more computers, in which the modules, components, systems and techniques described below can be implemented. The neural network system 4 receives input 400, processes the input 400 through each layer inside the neural network, and generates output 410 based on this. The neural network system 4 can be configured to receive images, texts, any kind of data or features extracted for these types of data as input, and generate a classification output or a continuous numerical vector based on the input.
[0154] A ResNeXt-50 (ResNeXt-XBNBlock-50) neural network system 5 including XBNBlock, such as Figure 5 As shown, the neural network system 5 is an example of a system implemented as a computer program on one or more computers, in which the modules, components, systems and techniques described below can be implemented. The neural network system 5 receives an input 50, processes the input 50 through each layer inside the neural network, and generates an output 58 based on this. The neural network system 5 can be configured to receive images, texts, any kind of data or features extracted for these types of data as input, and generate a classification output or a continuous numerical vector based on the input.
[0155] For example, for an image classification task, the network input 400 (or 50) is an image or a feature extracted from an image, and the network output 410 (or 58) is a score for each class in a set of categories, where each score represents the possibility (or probability) that the image belongs to that category;
[0156] For the image-based target detection task, the input 400 (or 50) of the neural network system 4 (or 5) is an image or a feature extracted from an image, and the output 410 (or 58) calculated by the neural network system 4 (or 5) for a given image is the location of multiple targets and the score for each class in the category set, where each score represents the possibility (or probability) that the image belongs to the category. The location of the target is a 4-dimensional continuous vector, such as the center point location coordinates (x, y) and the width w and height h;
[0157] For the text classification task, the network input 400 (or 50) is a word embedding matrix obtained after processing the word sequence, and the output 410 (or 58) calculated by the neural network system 4 (or 5) for a given text is a score for each class in the category set, where each score represents the possibility (or probability) that the text belongs to the category;
[0158] For the machine translation task, the network input 400 (or 50) is a word embedding matrix obtained after processing the text in one language, and the network output 410 (or 58) is a score for each of a set of text segments in another language, wherein each score represents the estimated possibility that the text segment in another language is a proper translation of the input text into another language.
[0159] Some of the sequences of layers of the neural network system 4 are residual-combination normalization layer modules 2, for example Figure 4 The 4 groups of residual block layers 405 in the embodiment include 8 residual-combination normalization layer modules 2. Some of the sequences of layers of the neural network system 5 are residual-combination normalization layer modules 3, for example Figure 5 The 4 groups of residual block layers 55 in include 16 residual-combined normalization layer modules 3.
[0160] Embodiments of the neural network systems and functional operations described in this specification may be implemented in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the neural network systems and functional operations described in this specification may be implemented as one or more computer programs. A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
Claims
1. A combined normalization layer module for improving the robustness of a neural network system, characterized in that: The combined normalization layer module is a sequence consisting of "linear layer-batch normalization layer BN-non-linear layer-linear layer-non-batch normalization layer BFN-non-linear layer". When applied to multi-layer perceptron and convolutional neural network, the input of the combined normalization layer module is different, so the calculation process is different, as follows: For the input of the multi-layer perceptron, the calculation process of the combined normalization layer module is as follows: (1) Calculate the output of the linear layer: For the fully connected layer: input Where x is a d-dimensional real vector, represents an n-dimensional real vector space; Then the output of the fully connected layer of the original neural network is h = Wx + b, W is the weight parameter matrix of the fully connected layer, and b is the learnable bias parameter; (2) Calculate the output of the batch normalization layer: Assume the batch size is m, then input Where x consists of m d-dimensional real vectors; the j-th dimension of the i-th sample is normalized to get the output: The vector after normalizing all dimensions of all samples As the output of BN; in and are the mean and variance of m samples in the jth dimension, respectively, and ∈ is the selected fixed minimum value; (3) Calculate the output of the nonlinear layer: For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained Where max(0,x) means taking the larger value between 0 and x as the result; (4) Calculate the output of the non-batch normalization layer: The main non-batch normalization layers include layer normalization layer LN and group normalization layer GN; 1) For LN: Input Normalize the jth dimension of the sample and get the output: The vector after all dimensions of the sample are normalized As the output of LN; in and are the mean and variance of the current sample in all dimensions respectively; 2) For GN: Input Divide the number of dimensions d into g groups, each containing k dimensions, where d = k × g; normalize the jth dimension in the i-th group and get the output: The vector after normalizing all dimensions of all groups As the output of GN; in and are the mean and variance of the current sample in all dimensions of the i-th group; For the input of the convolutional neural network, the calculation process of the combined normalization layer module is as follows: (1) Calculate the output of the linear layer: For the convolutional layer form: input Where h and w are the height and width of the input feature map, d is the number of channels of the feature map, that is, the dimension; let the height and width of the filter be F h and F w , the step size of the convolution operation is s, and the padding is p, then the output of each channel feature map is in (2) Calculate the output of the batch normalization layer: Assume the batch size is m, then input Where x consists of m feature maps of dimension d×h×w, and the (k,l)th dimension of the jth channel of the i-th sample is normalized to get the output: The vector after normalizing all dimensions of all samples As the output of BN; in and are the mean and variance of m samples on the jth channel respectively; (3) Calculate the output of the nonlinear layer: For the Relu function: input After the activation function ReLU(x)=max(0,x), the output of the nonlinear layer is obtained (4) Calculate the output of the non-batch normalization layer: The main non-batch normalization layers include layer normalization layer LN and group normalization layer GN; 1) For LN: Input Normalize the (j, k)th dimension of the i-th channel and get the output: The vector after normalizing all dimensions of all channels As the output of LN; in and They are the mean and variance of the current sample on all channels respectively; 2) For GN: Input Divide the number of channels d into g groups, each containing k features, where d = k × g; normalize the (i, j)th dimension of the cth channel of group G to get the output: The vector after normalizing all dimensions of all groups As the output of GN; in and are the mean and variance of the current sample group G in all dimensions of width and height, respectively.
2. A combined normalization layer module for improving the robustness of a neural network system as claimed in claim 1, characterized in that: The combined normalization layer module, when used to construct a forward neural network, is implemented by directly connecting several combined normalization layer modules in series, specifically including: a residual neural network using a combined normalization layer module in the form of Basic Block and BottleNeck; (1) For the residual neural network using Basic Block, the residual-combination normalization layer module in the Basic Block form is described as follows: 1) BasicBlock_XBNBlock_P1 replaces the first BN in the original residual block with BFN. The residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BFN-nonlinear layer-convolutional layer-BN-optional downsampling layer-residual branch end point-nonlinear layer"; Vector input form for convolutional neural networks For the residual connection part, the input x of the module is saved in the residual r at the starting point of the residual branch, that is, r=x; then the residual is directly transmitted to the end point of the residual branch; thus, the output y=y0+r obtained at the end point of the residual branch, where y0 represents the output obtained by the BN or downsampling layer before the end point of the residual branch; For the optional downsampling layer, it includes a 1×1 convolution layer and a BN to adjust the number of channels and spatial size; 2) BasicBlock_XBNBlock_P2 replaces the second BN in the original residual block with BFN; the residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BN-nonlinear layer-convolutional layer-BFN-optional downsampling layer-residual branch end point-nonlinear layer"; (2) For the residual neural network using BottleNeck, the residual-combination normalization layer module in BottleNeck form is described as follows: 1) Bottleneck_XBNBlock_P1 replaces the first BN in the original bottleneck block with BFN; the residual-combined normalization layer module is a sequence of "residual branch starting point-convolutional layer-BFN-nonlinear layer-convolutional layer-BN-nonlinear layer-convolutional layer-BN-optional downsampling layer-residual branch end point-nonlinear layer"; 2) Bottleneck_XBNBlock_P2 replaces the second BN in the original bottleneck block with BFN; the residual-combined normalization layer module is a sequence of "residual branch starting point-convolutional layer-BN-nonlinear layer-convolutional layer-BFN-nonlinear layer-convolutional layer-BN-optional downsampling layer-residual branch end point-nonlinear layer"; 3) Bottleneck_XBNBlock_P3 replaces the third BN in the original bottleneck block with BFN; the residual-combined normalization layer module is a sequence of multiple layers consisting of "residual branch starting point-convolutional layer-BN-non-linear layer-convolutional layer-BN-non-linear layer-convolutional layer-BFN-optional downsampling layer-residual branch end point-non-linear layer".