An apparatus and method for machine learning for image processing

Abstract

A device for machine learning includes one or more processors for implementing a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first and second layers. The normalization layer is used by the device to perform the following operations when training on a batch of training samples: receiving multiple outputs from the first layer of the neural network for multiple training samples, each output including multiple data values ​​with different indices on a first dimension and a second dimension, where the first dimension represents the channel dimension; dividing the outputs into multiple groups according to the indices on the first and second dimensions associated with the outputs; forming a normalized output for each group; and providing the normalized output as input to the second layer of the neural network. This enables the training of a high-performance deep convolutional neural network that runs stably at different batch sizes and can generalize to multiple visual tasks, accelerating training and improving training performance.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to a machine learning device and method for image processing. Background Technology

[0002] Deep Convolutional Neural Networks (DCNNs) are a widely used method for visual tasks such as image classification, object detection, and semantic segmentation. DCNNs typically consist of convolutional layers, normalization layers, and activation layers. Normalization layers are crucial for improving performance and accelerating the training process.

[0003] However, training DCNNs is typically difficult and time-consuming. Previous training methods also had limited performance.

[0004] The Batch Normalization (BN) described in Sergey Ioffe and Christian Szegedy's paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shifts" (pp. 448-456, 2015) at the International Conference on Machine Learning uses the mean and variance computed along with the batch, height, and width dimensions of the feature map to normalize it. The normalized feature map is then rescaled and retranslated to preserve the representational power of the DCNN. Based on BN, many other normalization methods have been proposed to compute mean and variance statistics across different dimensions. For example, Layer Normalization (LN), described in Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton's paper "Layer Normalization" (2016) in the NIPS Deep Learning Proceedings, has been proposed for computing statistics across the channel, height, and width dimensions of Recurrent Neural Networks (RNNs). The weight normalization (WN) described by Tim Salimans and Durk P Kingma in their paper "Weight Normalization: A Simple Reparameterization for Accelerating Training of Deep Neural Networks" (pp. 901-909, 2016) presented at the Conference on Advances in Neural Information Processing Systems has been proposed for parameterizing weight vectors in supervised image recognition, generative models, and deep reinforcement learning. Separate normalization, described by Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H Sinz, and Richard S Zemel in their paper "Normalizing the Normalization Element: Comparing and Extending Network Normalization Schemes" (2016) presented at the International Conference on Learning Representations, includes BN and LN (as special cases), which have been proposed for image classification, language modeling, and super-resolution. Instance normalization (IN), described in the article “Instance Normalization: The Missing Component in Fast Stylization” published by Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky in the arXiv preprint arXiv:1607.08022 (2016), is proposed for fast stylization, where statistics are calculated based on the height and width dimensions.Unlike calculating statistics from data, normalization propagation, as described in the paper "Normalization Propagation: A Parametric Technique for Eliminating Internal Covariate Shifts in Deep Networks" (pp. 1168–1176, 2016) by Devansh Arpit, Yingbo Zhou, Bhargava Kota, and Venu Govindaraju at the International Conference on Machine Learning, estimates data independently of the distributions within each layer. Group normalization, described in the paper "Group Normalization" (pp. 3–19, 2018) by Yuxin Wu and Kaiming He in the proceedings of the European Conference on Computer Vision (ECCV), divides channels into multiple groups and calculates statistics, height, and width dimensions for each group of channels to demonstrate stability with batch size. The positional normalization (PN) described in the paper "Positional Normalization" (pp. 1620–1632, 2019) presented by Boyi Li, Felix Wu, Kilian Q, Weinberger, and Serge Belongie at the Conference on Advances in Neural Information Processing Systems is proposed for computing statistics on the channel dimension of generative networks.

[0005] BN, IN, LN, GN, and PN share the same four steps: dividing the intermediate feature map into multiple feature groups; calculating the mean and variance of each feature group; normalizing the corresponding feature groups using the calculated mean and variance of each feature group; and applying two additional trainable parameters to each channel of the intermediate feature map to recover the representational power of the DCNN. The main difference between BN, IN, LN, GN, and PN lies in the division of feature groups.

[0006] Among these normalization methods, Batch Normalization (BN) generally achieves good performance with large batch sizes. However, its performance may degrade with smaller batch sizes. General Normalization (GN) is more stable for different batch sizes, but its performance is slightly inferior to BN with larger batch sizes. Other normalization methods (including In, LN, and PN) perform well on specific tasks, but generally do not generalize to multiple vision tasks compared to BN, and their performance is poor with larger batch sizes.

[0007] A normalization method needs to be developed to overcome these problems. Summary of the Invention

[0008] According to one aspect, the present invention provides an apparatus for machine learning, the apparatus comprising one or more processors configured to implement a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network, the normalization layer being configured to perform the following operations when the apparatus trains on a batch of training samples: receiving multiple outputs of the first layer of the neural network for the multiple training samples in the batch, each output including multiple data values ​​with different indices on a first dimension and a second dimension, the first dimension representing a channel dimension; dividing the outputs into multiple groups according to the indices on the first dimension and the second dimension associated with the outputs; forming a normalized output for each group; and providing the normalized outputs as inputs to the second layer of the neural network.

[0009] This could enable the training of high-performance DCNNs that run stably with different batch sizes and generalize to multiple vision tasks. It could also accelerate and improve the performance of DCNN training.

[0010] The second dimension represents one or more spatial dimensions, such as the height and width of the feature map of an image. This can provide an efficient way to perform machine learning on spatially expanded samples.

[0011] The step of generating a normalized output for each group may include: calculating aggregate statistical parameters of the outputs in the group. Such parameters are conveniently used to assist in the training of subsequent neural network layers.

[0012] The step of generating a normalized output for each group may include calculating the mean and variance of the outputs in the group. One or both of these quantities may be helpful in training subsequent neural network layers.

[0013] The step of grouping the outputs may include assigning each output to only a single group within the group. In this way, each output may not be overrepresented during the training of subsequent neural network layers.

[0014] The step of grouping the outputs may include assigning all outputs related to common metrics on the first dimension and common metrics on the second dimension to the same group. Therefore, these groups may include outputs related by having common metrics.

[0015] The step of grouping the outputs may include assigning outputs related to a common batch to different groups. Including the batch dimension in the statistical data calculation can further improve the performance and universality of normalization.

[0016] The step of grouping the outputs may include assigning the outputs to different groups based on the metric on the first dimension associated with the outputs. This may enable aggregated values ​​derived from the groups to provide information about the outputs having that metric.

[0017] The step of grouping the outputs may include assigning the outputs to different groups based on the metric on the second dimension associated with the outputs. This may enable aggregated values ​​derived from the groups to provide information about the outputs having that metric.

[0018] The normalization layer can be used to perform the following operations: receive control parameters; compare the control parameters with a predetermined threshold; and, based on the parameters, determine how to assign the output to different groups in the grouping step according to the indicators related to the output in the first and second dimensions. Selecting the size of the feature group used to calculate the statistical data can further improve the stability of normalization for different batch sizes.

[0019] The device can be used to generate the control parameters based on the number of training samples in the batch. For example, a smaller G can be used when the batch size is small, and a larger G can be used when the batch size is large.

[0020] The output can be a feature map formed by the first layer of the neural network. This may enable the device to be used for computer vision and image classification tasks.

[0021] The device can be used to train the second layer of the neural network based on the normalized output.

[0022] According to a second aspect, the present invention provides a method for training an apparatus for machine learning based on a batch of training samples, the apparatus including a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network, the method comprising: receiving multiple outputs of the first layer of the neural network for the batch of multiple training samples, each output including multiple data values ​​with different indices on a first dimension and a second dimension, the first dimension representing a channel dimension; dividing the outputs into multiple groups according to the indices on the first dimension and the second dimension associated with the outputs; forming a normalized output for each group; and providing the normalized outputs as inputs to the second layer of the neural network.

[0023] This method may enable the training of high-performance DCNNs that run stably with different batch sizes and can generalize to multiple vision tasks. The method can accelerate and improve the performance of DCNN training. Attached Figure Description

[0024] The invention will now be described by way of example with reference to the accompanying drawings.

[0025] In the attached diagram:

[0026] Figures 1(a)-(f) schematically illustrate the differences between BN, IN, LN, GN, PN, and the BatchGroup Normalisation (BGN) method described in this paper in terms of the dimension in which statistical data is computed. Each subplot shows a feature map tensor, where N is the batch axis, C is the channel axis, and (H,W) is the spatial axis. Pixels shown in shaded mode are used to compute the statistical data. Figures 1(a), 1(b), 1(c), 1(d), and 1(e) show examples of the BN, IN, LN, GN, and PN methods, respectively. Figure 1(f) illustrates the BGN method;

[0027] Figure 2 The figure shows a comparison between the method described in this paper and existing methods. The figure illustrates the Top-1 accuracy of training ResNet-50 on ImageNet, where batch sizes vary, and BN, IN, LN, GN, PN, and the BGN described in this paper are the normalization layers.

[0028] Figure 3 The Top-1 validation accuracy of BGN for ImageNet classification using the ResNet-50 model is shown. The hyperparameter G is set from 512 to 1.

[0029] Figure 4 The Top1 validation accuracy for ImageNet classification using the ResNet-50 model and different batch sizes (128 to 2) is shown.

[0030] Figures 5(a) and 5(b) illustrate examples of DARTS search space architectures consisting of a series of units, where each unit is a directed acyclic graph, nodes represent feature maps, and edges represent network operations such as convolutional or pooling layers. Figure 5(a) shows an example of a normal unit, and Figure 5(b) shows an example of a reduced unit in the DARTS search space.

[0031] Figure 6 The validated accuracy of CIFAR-10 using BN, IN, LN, GN, PN and the recommended BGN in DARTS is shown during the search and evaluation phases.

[0032] Figure 7Robust validation accuracy and net validation accuracy are shown for adversarial training using BN, IN, LN, GN, PN, and BGN as normalized layers in WideResNet. Net accuracy is evaluated on a clean dataset, and robust accuracy is evaluated on PGD attack data.

[0033] Figure 8 The average accuracy of performing few-shot learning tasks with imprinted weights on miniImageNet using ResNet-12 as the backbone network is shown. The normalization layers were replaced according to BN, IN, LN, GN, PN, and BGN. The average accuracy on 600 randomly generated test sets is reported with a confidence interval of 95%.

[0034] Figure 9 An example of a device for machine learning is shown, the device including a processor for implementing a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer of the neural network and the second layer of the neural network;

[0035] Figure 10 A method for training an apparatus for machine learning based on a batch of training samples is shown, the apparatus including a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network. Detailed Implementation

[0036] This paper describes a normalization method for training deep convolutional neural networks. Some implementations of the method have been shown to achieve better performance, stability and versatility than existing methods.

[0037] The method described herein can be implemented by a machine learning device with a processor for implementing a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network.

[0038] As will be described in more detail below, the normalization layer can be used to perform the following operations when the device is training on a batch of training samples: receiving multiple outputs of the first layer of the neural network for the multiple training samples in the batch, each output including multiple data values ​​at different indices on a first dimension and a second dimension, wherein the first dimension represents the channel dimension.

[0039] Preferably, the output is a feature map formed by the first layer of the neural network, as illustrated in the example below.

[0040] In one example, the first dimension is the channel C in the feature map. The second dimension represents one or more spatial dimensions of the feature map. For example, the second dimension could represent the height (H) and / or width (W) of the feature map.

[0041] Then, based on the indices on the first and second dimensions associated with the output, the output is divided into multiple groups, and a normalized output is formed for each group. Advantageously, the step of grouping the output may further include assigning outputs associated with a common batch to different groups.

[0042] In one example, consider the feature map output F of the first few layers of the network. N×C×H×W , where N is the batch size of the feature maps.

[0043] First, the channel, height, and width dimensions are merged into a new dimension to obtain F. N×M Where M = C × H × W.

[0044] Preferably, the step of forming a normalized output for each group includes: calculating aggregate statistical parameters of the outputs in that group, such as the mean and variance.

[0045] In this example, the average value μ g and variance The calculations are based on the batch size and the new dimensions (C, H, W) as follows:

[0046]

[0047] Where G is the number of groups in the new dimension, which is a hyperparameter, and S = M / G is the number of instances in each feature group of the partition.

[0048] The hyperparameter G can be used to control the number of feature instances or the size of feature groups to calculate the statistics.

[0049] Therefore, the normalization layer can also be used to receive control parameters (i.e., hyperparameters G) and compare the control parameters with a predetermined threshold. Based on this parameter, the normalization layer can determine how to assign the output to different groups in the grouping step according to the indicators related to the output in the first and second dimensions.

[0050] The device can be used to form parameter G based on the number of training samples in the batch.

[0051] When determining the batch size for a DCNN, a large full batch size can lead to disordered gradients, while a small batch size can result in noisy gradients. Accurate normalized statistics should cover an adequate number of feature instances. Therefore, the method described in this paper can introduce the feature set and the hyperparameter G to control the number of feature instances or the size of the feature set to compute the statistics. For example, when the batch size is small, a smaller G can be used to incorporate the entire new dimension into the statistics computation, while when the batch size is large, a larger G can be used to split the new dimension into smaller chunks to compute the statistics.

[0052] Then, for g∈[1,G], the feature map is normalized as follows:

[0053]

[0054] Here, ∈ is a small number added for stability of the split. After BN, IN, LN, GN, and PN, F... N×M Split back to F N×C×H×W To maintain the representational power of DCNN, additional trainable parameters are added to each functional channel:

[0055]

[0056] In BN, the μ used in the test phase c and It is the moving average of the values ​​during the training phase. The method described in this paper can also use this strategy because the normalization method is best independent of the batch size. IN, LN, GN, and PN typically use statistics calculated directly from the testing phase.

[0057] Therefore, the normalization layer divides the output into multiple groups based on the indices on the first and second dimensions associated with the output. Then, a normalized output is formed for each group. The normalized output is then provided as input to the second layer of the neural network.

[0058] The outputs can be grouped in different ways. The step of grouping the outputs may include: assigning each output to only a single group within the group. The step of grouping the outputs may also include: assigning all outputs related to common metrics on the first dimension and common metrics on the second dimension to the same group.

[0059] In another example, the step of grouping the output may include: assigning the output to different groups based on the metric on the first dimension associated with the output. Furthermore, the step of grouping the output may include: assigning the output to different groups based on the metric on the second dimension associated with the output.

[0060] In a preferred embodiment, the step of grouping the outputs includes assigning the outputs associated with a common batch to different groups. Therefore, groups can be additionally formed along the batch dimension (N). Referring to the representation shown in Figure 1(f), each group can extend along the N-axis as shown in the figure, or there can be subgroups along the N-axis and the (C,H,W) axes. In other words, in a preferred embodiment where the (C,H,W) dimension is compressed to a single axis, the group is shown for all N (i.e., it extends across the N-axis). However, the samples can also be grouped along the N-axis (batch grouping). Preferably, each of these groups contains multiple samples.

[0061] Figures 1(a)-(f) illustrate the differences in the dimension in which statistical data are computed by BN, IN, LN, GN, PN, and the method described in this paper (hereinafter referred to as Batch Group Normalisation (BGN)). Figures 1(a), 1(b), 1(c), 1(d), 1(e), and 1(f) show examples of the BN, IN, LN, GN, and PN methods, respectively. Figure 1(f) shows an example of the BGN method. Each subgraph displays a feature map tensor, where N is the batch axis, C is the channel axis, and (H, W) is the spatial axis. Pixels shown in shaded mode are used to compute the statistical data.

[0062] Figure 2 The Top-1 accuracy of ResNet-50 training on ImageNet is shown, with different batch sizes and BN, IN, LN, GN, PN, and BGN being the normalization layers. BGN achieves good performance and stability with different batch sizes without adding trainable parameters, using additional information, or requiring additional computation.

[0063] One application of the method described in this paper is image classification. In the example described below, ImageNet (see “ImageNet Classification with Deep Convolutional Neural Networks”, pp. 1097–1105, 2012, by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, at the Conference on Advances in Neural Information Processing Systems) was used, containing 1:28M training images and 50,000 validation images. The model used in the example was ResNet-50 (see “Image Recognition with Deep Residual Learning”, pp. 770–778, 2016, by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, in the proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition) with approximately 50 convolutional layers followed by normalization and activation layers, stacked with residual learning. Eight GPUs were used in the ImageNet experiments. The average gradient used for backpropagation was calculated across the eight GPUs, while the average and variance used in BN and BGN were computed independently on each GPU. c and β c The initial values ​​were 1 and 0, respectively, while all other trainable parameters were initialized in the same way as proposed by He et al. Training was performed for 120 epochs, with the learning rate decaying by a factor of 10 at epochs 30, 60, and 90. Based on the paper "Accurate Large Micro-Batch SGD: Training ImageNet in 1 Hour" published by PriyaGoyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, AapoKyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He in the arXiv preprint arXiv:1706.02677 (2017), the initial learning rates for experiments with batch sizes of 128, 64, 32, 16, 8, 4, and 2 were 0.4, 0.2, 0.1, 0.05, 0.025, 0.0125, and 0.00625, respectively. Stochastic Gradient Descent (SGD) is used as the optimizer. 10 –4 The weight decay is applied to all trainable parameters.

[0064] For validation, each image was cropped into 224x224 blocks from the center. Top-1 accuracy was used as the evaluation metric. All experiments were trained with the same programming implementation, but the normalization layer was replaced with BN, IN, LN, GN, PN, and BGN, respectively.

[0065] To explore the hyperparameter G, BGNs with 512, 256, 128, 64, 32, 16, 8, 4, 2, and 1 batches were used as the normalization layer in ResNet-50 for ImageNet classification. The maximum batch size (depending on GPU memory) and minimum batch size (128 and 2) were tested in the experiments. Figure 3 The Top1 accuracy of the validation dataset is shown.

[0066] Typically, the results indicate that a larger G (e.g., 512) is better suited for larger batch sizes (e.g., 128), while a smaller G (e.g., 1) is better suited for smaller batch sizes (e.g., 2). This suggests that the number of feature instances affects statistical data computation during normalization. Appropriately, when the batch size is large, a larger G can be used to split the new dimension to maintain an appropriate number of feature instances for statistical data computation. Appropriately, when the batch size is small, a smaller G can be used to combine the new dimension to maintain an appropriate number of feature instances for statistical data computation.

[0067] Figure 4 Further experimental results are shown, where BN, IN, LN, GN, PN, and BGN are used as the normalization layers in ResNet-50 with batch sizes of 128, 64, 32, 16, 8, 4, and 2, respectively. The number of groups in GN is set to 32. For batch sizes of 128, 64, 32, 16, 8, 4, and 2, the number of groups in BGN is set to 512, 256, 128, 64, 16, 2, and 1, respectively. Figure 3 G is chosen for the maximum and minimum batch sizes, while G is chosen for other batch sizes using interpolation. Figure 4 The Top-1 accuracy for each method is shown. In these examples, BGN outperforms the previous methods at all different batch sizes. BN's performance drops rapidly with smaller batch sizes. IN typically performs poorly in ImageNet classification.

[0068] The following example demonstrates the application of Neural Architecture Search (NAS) to image classification on the CIFAR-10 (Canadian Institute for Advanced Study) dataset. This shows that, in addition to manually designed, regular neural network architectures, BGN is also suitable for automatically designed, less regular neural network architectures. The following example uses a cell-based architecture automatically designed using NAS (specifically DARTS), as described in the paper "DARTS: Search for Differentiable Architectures" (2019) published by Hanxiao Liu, Karen Simonyan, and Yiming Yang at the International Conference on Learning Representations. For DARTS, both the search and training use normalization methods.

[0069] like Figure 5(a) and 5(b) As shown, the searched architecture family consists of a series of units, each of which is a directed acyclic graph (DAG), where nodes represent feature maps and edges represent network operations such as convolutional or pooling layers. Figure 5(a) shows an example of a normal unit, and Figure 5(b) shows an example of a reduced unit in the DARTS search space. Each unit has two input nodes 501, four internal nodes 502, and one output node 503. Multiple units are connected in a feedforward manner to create a deep neural network.

[0070] Given a set of possible operations, DARTS encodes the architecture search space with continuous parameters to form a single-sample model, and performs the search by training the single-sample model with two layers of optimization, wherein the weights and architecture parameters of the model are optimized alternately with training data and validation data.

[0071] For DARTS training, the same experimental setup as used by Liu et al. was employed. During the search and evaluation phases, the BN layers in DARTS were replaced with the normalized layers IN, LN, GN, PN, and BGN. In this embodiment, the method searched for 8 units over 50 epochs with a batch size of 64 and an initial number of channels of 16. SGD was used to optimize the model weights with an initial learning rate of 0.025, momentum of 0.9, and weight decay of 3 x 10⁻⁶. –4 The ADAM method described by Diederik P Kingma and Jimmy Ba in their arXiv preprint arXiv:1412.6980 (2014) is used to optimize architecture parameters with an initial learning rate of 3x10. –4 The momentum is (0:5; 0:999), and the weight decay is 10. –3 The model was evaluated using a 20-unit network with 36 initial channels to ensure its size was comparable to other baseline models. The entire training set was used to train the model for 600 epochs with a batch size of 96 to ensure convergence. For GN, a configuration of G=32 was used, while for BGN, a configuration of G=256 was used. Other hyperparameters were set to the same parameters as in the search phase. The optimal 20-unit architecture searched on CIFAR-10 using DARTS was trained from scratch using the corresponding normalization method used in the search phase. Figure 6 The verification accuracy of each method is shown. IN and LN fail to converge, while BGN outperforms GN, PN, and BN. These results suggest that, in some implementations, BGN can be generalized to NAS for search and evaluation.

[0072] It is well known that DCNNs are vulnerable to malicious perturbation examples (known as adversarial attacks). Adversarial training has been proposed to address this issue. In the following example, BGN is applied to adversarial training, and its results are compared with BN, IN, LN, GN, and PN. WideResNet, described in the paper "Wide Residual Networks" (pp. 87.1–87.12, BMVA Press, September 2016) published in the British Machine Vision Conference (BMVC) proceedings and edited by Edwin R. Hancock, Richard C. Wilson, and William APSmith, was used to perform an image classification task on the CIFAR-10 dataset, where the depth was set to 10 and the width factor was set to 2. The neural network was trained and evaluated against a four-step Projected Gradient Descent (PGD) attack. For the PGD, the step size was set to 0.00784 and the maximum perturbation norm was set to 0.0157. The training was conducted for 200 epochs until convergence. Due to the specific nature of adversarial training, G=128 was used in both GN and BGN. This divides the image into blocks, limiting adversarial attacks to a finite range by breaking the correlation between adversarial attacks between different image blocks, thereby improving the robustness of the image. The Adam optimizer was used with a learning rate of 0.01. Figure 7 Robust validation accuracy and net validation accuracy are shown when training WideResNet using BN, IN, LN, GN, PN, and BGN as normalization layers. Robust accuracy is more important than net accuracy when judging adversarial networks. PN encounters convergence problems and fails to converge. In this implementation, BGN outperforms the other methods.

[0073] The BGN method can also be implemented as part of a Few Shot Learning (FSL) task. FSL aims to train a model capable of recognizing previously unseen new categories using only a limited set of training samples. The training dataset, with sufficient annotated samples, includes the basic categories. The test dataset contains C new classes, each associated with only a small set of K labeled samples (e.g., 5 or fewer) containing the support set, while the remaining unlabeled samples comprise the query set and are used for evaluation. This can also be referred to as a C-class, K-samples-per-class FSL classification problem.

[0074] In one example, the imprinted weight model was used, as described in the paper "Few-shot Learning with Imprinted Weights" (pp. 5822–5830, 2018) by Hang Qi, Matthew Brown, and David GLowe in the proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. During training, a cosine classifier is learned on the feature extraction layer, with each column of classifier parameter weights considered as a prototype of the corresponding class. During testing, new class prototypes (new columns of classifier weight parameters) are defined by averaging the feature representations of the supporting images, and unlabeled images are classified using a nearest-neighbor strategy. The ResNet-12 backbone (see "Few-Shot Learning with Matching Networks" by Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al., NeurIPS 2016) was tested on miniImageNet (see "TADAM: Task-Relevant Adaptive Metrics for Improving Few-Shot Learning" by Boris Oreshkin, Pau Rodríguez Lópe, and Alexandre Lacoste, NeurIPS 2018) with settings including 5 classes, 1 sample per class, and 5 samples per class. In this example, the training scheme described in "Dynamic Few-Shot Visual Learning Without Forgetting" by Spyros Gidaris and Nikos Komodakis, CVPR 2018, was used. The BGN model was optimized using SGD, with Nesterov momentum set to 0:9, weight decay set to 0.0005, microbatch size set to 256, and epochs set to 60. All input images were resized to 84x84. The learning rate is initialized to 0.1 and then changed to 0.006, 0.0012, and 0.00024 at the 20th, 40th, and 50th epochs, respectively. Figure 8 The mean and variance of accuracy for training on miniImageNet and performing tasks with 5 classes, 1 sample per class, and 5 samples per class are shown, with the normalization layer of imprinted weights replaced by BN, IN, LN, GN, PN, and the proposed BGN. In these implementations, BGN outperforms other methods, demonstrating its versatility even with very limited available labeled data.

[0075] Figure 9 A machine learning device 900 for implementing the BGN method is schematically shown. The device 900 can be implemented on devices such as laptops, tablets, smartphones, or televisions.

[0076] The device 900 includes a processor 901 for processing the dataset as described herein. For example, the processor 901 may be implemented as a computer program running on a programmable device such as a central processing unit (CPU). The system 200 includes a memory 902 for communicating with the processor 901. The processor 902 may be non-volatile memory. The processor 901 may also include a cache (…). Figure 9 (Not shown in the image), the cache can be used to temporarily store data from memory 902. The system may include multiple processors and multiple memories. The memories may store data executable by the processors. The processors may be configured to run according to a computer program stored in a non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods as described herein.

[0077] Figure 10 A method 1000 for training a machine learning device based on a batch of training samples is summarized, the device including a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network. In step 1001, the method includes receiving multiple outputs of the first layer of the neural network for the batch of multiple training samples, each output including multiple data values ​​with different indices on a first dimension and a second dimension, the first dimension representing a channel dimension. The output may be a feature map formed by the first layer of the neural network. In step 1002, the method includes dividing the output into multiple groups according to the indices on the first and second dimensions associated with the output. In step 1003, the method includes forming a normalized output for each group. In step 1004, the method includes providing the normalized output as input to the second layer of the neural network. The method further includes training the second layer of the neural network based on the normalized output.

[0078] As described above, the method divides the intermediate feature maps into feature groups in different ways. In a preferred embodiment, each intermediate feature map has four dimensions, including batch size, height, width, and channel size. First, the height, width, and channel dimensions are merged into one dimension, and then this new dimension is divided into multiple feature groups. The hyperparameter G controls how many groups the intermediate feature maps are divided into. Then, statistics (e.g., mean and variance) for each feature group are calculated for the entire micro-batch.

[0079] The normalization method described in this paper exhibits good performance, runs stably with varying batch sizes, and generalizes to multiple vision tasks. This method does not utilize additional trainable parameters, information across multiple layers or iterations, or additional computation. It computes mean and variance statistics for both batch and group (channel, height, and width) dimensions, and the size of the partitioned feature groups can be controlled using the hyperparameter G. In some implementations, this normalization method can accelerate and improve the performance of DCNN training.

[0080] Advantageously, the method can consider the batch dimension in the statistical data calculation (i.e., include the batch dimension in the calculation of the mean and variance), and can control the size of the feature set used for statistical data calculation to an appropriate size (i.e., neither too large nor too small). Including the batch dimension in the statistical data calculation can further improve the performance and generality of normalization, while selecting the size of the feature set used for calculating the statistical data can further improve the stability of normalization for different batch sizes.

[0081] The method described herein does not use additional trainable parameters or computations or multiple iterations / multi-layer information. This method can be combined with other techniques that use additional trainable parameters or computations or multiple iterations / multi-layer information to further improve performance. Therefore, its implementation is intuitive, independent of many methods, and can be used in conjunction with them to further enhance performance.

[0082] In some implementations, BGN outperforms BN by nearly 10% on mini-batch ImageNet classification. In other implementations, it outperforms BN, IN, LN, GN, and PN in image classification, neural network architecture search, adversarial learning, few-shot learning, and unsupervised domain adaptation tasks.

[0083] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any of the problems disclosed herein, and without limitation to the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. Various modifications that can be made within the scope of the invention, given the foregoing description, will be apparent to those skilled in the art.

Claims

1. A machine learning device for image processing, characterized in that, The device includes one or more processors, the processors being configured to implement a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network, the normalization layer being configured to perform the following operations when the device trains on a batch of image training samples: For multiple image training samples in the batch, the first layer of the neural network receives multiple output feature maps. Each output feature map includes multiple data values ​​with different indices in the first and second dimensions. The first dimension represents the channel dimension, height dimension, and width dimension of the output feature map, and the second dimension includes the batch dimension of the output feature map. The channel dimension, height dimension, and width dimension of the output feature map are merged into a new dimension; Receive control parameters, which are determined based on the number of the plurality of image training samples in the batch; The control parameter is compared with a predetermined threshold. Based on the control parameters, and the new dimension and the index on the second dimension associated with the output feature map, the output feature map is divided into multiple groups, and the number of instances S in each group satisfies the following formula: S=M / G, where M represents the new dimension and G represents the control parameters; Normalized output feature maps are generated for each group; The normalized output feature map is provided as the input to the second layer of the neural network.

2. The device according to claim 1, characterized in that, The step of forming a normalized output feature map for each group includes: calculating the aggregate statistical parameters of the output feature maps in the group.

3. The device according to claim 1, characterized in that, The step of forming a normalized output feature map for each group includes: calculating the mean and variance of the output feature maps in the group.

4. The device according to any one of claims 1-3, characterized in that, The step of dividing the output feature map into multiple groups includes: assigning each output feature map to only a single group within the group.

5. The device according to any one of claims 1-3, characterized in that, The step of dividing the output feature maps into multiple groups includes: assigning all output feature maps related to the common indicators on the first dimension and the common indicators on the second dimension to the same group.

6. The device according to any one of claims 1-3, characterized in that, The step of dividing the output feature map into multiple groups includes: assigning the output feature maps related to the common batch to different groups.

7. The device according to any one of claims 1-3, characterized in that, The step of dividing the output feature map into multiple groups includes: assigning the output feature map to different groups according to an index on the first dimension associated with the output feature map.

8. The device according to any one of claims 1-3, characterized in that, The step of dividing the output feature map into multiple groups includes: assigning the output feature map to different groups according to an index on the second dimension associated with the output feature map.

9. The device according to any one of claims 1-3, characterized in that, The device is used to train the second layer of the neural network based on the normalized output feature map.

10. A method for training an apparatus for machine learning based on a batch of image training samples, characterized in that, The device includes a first layer of a neural network, a second layer of a neural network, and a normalization layer disposed between the first layer and the second layer of the neural network; the method includes: For multiple image training samples in the batch, the first layer of the neural network receives multiple output feature maps. Each output feature map includes multiple data values ​​with different indices in the first and second dimensions. The first dimension represents the channel dimension, height dimension, and width dimension of the output feature map, and the second dimension includes the batch dimension of the output feature map. The channel dimension, height dimension, and width dimension of the output feature map are merged into a new dimension; Receive control parameters, which are formed based on the number of image training samples in the batch; The control parameter is compared with a predetermined threshold. Based on the control parameters, and the new dimension and the index on the second dimension associated with the output feature map, the output feature map is divided into multiple groups, and the number of instances S in each group satisfies the following formula: S=M / G, where M represents the new dimension and G represents the control parameters; Normalized output feature maps are generated for each group; The normalized output feature map is provided as the input to the second layer of the neural network.

Images