Neural network with adaptive gradient clipping

Abstract

A computer-implemented method for training a neural network is disclosed. The method includes determining a gradient associated with a parameter of the neural network. The method also includes determining a ratio of a norm of the gradient to a norm of the parameter and comparing the ratio to a threshold. In response to determining that the ratio exceeds the threshold, reducing a value of the gradient such that the ratio is equal to or below the threshold. The value of the parameter is updated based on the reduced value of the gradient.

Description

Technical Field

[0001] This specification relates to systems and methods for training neural networks using the adaptive gradient clipping technique. Background Technology

[0002] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set.

[0003] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from it. Specifically, a recurrent neural network can use some or all of the network's internal states from previous time steps when computing the output at the current time step. An example of a recurrent neural network is a Long Short-Term Memory (LSTM) neural network, which includes one or more LSTM memory blocks. Each LSTM memory block can include one or more units, each including an input gate, a forget gate, and an output gate, which allows the unit to store the unit's previous state, for example, for use in generating the current activation or for other components of the LSTM neural network. Summary of the Invention

[0004] This specification generally describes how a system implemented as a computer program on one or more computers in one or more locations can perform a method for training a neural network (i.e., adjusting the parameters of the neural network).

[0005] In one aspect, a computer-implemented method for training a neural network is provided, comprising determining gradients associated with the parameters of the neural network. A ratio of the gradient norm to the parameter norm is determined and compared to a threshold. In response to determining that the ratio exceeds the threshold, the value of the gradient is reduced such that the ratio is equal to or below the threshold. The values ​​of the parameters are then updated based on the reduced gradient value.

[0006] This method provides an adaptive gradient pruning technique that ensures stable parameter updates. In some neural networks, such as very deep ones with hundreds or thousands of layers, batch normalization is required for efficient training. This method enables the efficient training of such neural networks without batch normalization layers, referred to in this paper as "normalizer-free" neural networks. Batch normalization introduces dependencies between training data items within a batch, making implementation on parallel or distributed processing systems more difficult. Batch normalization is also a computationally expensive operation.

[0007] By using the adaptive gradient pruning technique described in this paper to ensure that the ratio of the gradient norm to the parameter norm remains within an acceptable range during training, a normalizer-free network with the same properties as batch-normalized networks can be provided, replicating the beneficial effects of batch normalization in a normalizer-free network. This provides more stable parameter updates in a normalizer-free network, and this stability enables training with large batch sizes, reducing the overall training time while maintaining task performance. Removing the dependency on batch normalization and intra-batch training terms also makes training easier to implement on parallel or distributed processing systems. The independence of training data terms is also important for sequence modeling tasks.

[0008] Conventional gradient clipping methods only consider the magnitude of the gradient; they do not consider the magnitude of the parameters themselves or the ratio of the gradient norm to the parameter norm. Using conventional gradient clipping in networks without normalizers does not yield the full benefits offered by using this adaptive gradient clipping method. In particular, training with conventional gradient clipping is sensitive to depth, batch size, and learning rate, and requires fine-grained tuning when changing any of these factors. Diminishing returns are also observed for larger networks when using conventional gradient clipping. The use of a ratio for gradient clipping provides improved stability in parameter updates, replicating the properties and benefits of batch normalization that conventional gradient clipping fails to achieve.

[0009] In some existing methods, the ratio is used to adapt the learning rate and also has the effect of scaling the gradient during parameter update steps. However, in this adaptive gradient pruning method, the gradient value is reduced only when the ratio is outside an acceptable range. This has a significant impact on the network's ability to generalize and maintain task performance, especially when computational resources are limited and small batch sizes must be used.

[0010] The ratio of the gradient norm to the parameter norm can be defined as the gradient norm divided by the parameter norm.

[0011] The method may further include: in response to determining that the ratio is below the threshold, maintaining the value of the gradient and updating the value of the parameter based on the maintained gradient value. That is, when the ratio is below the threshold, the gradient may remain unchanged.

[0012] Reducing the value of a gradient can include multiplying the gradient value by a scaling factor. The scaling factor can be based on a ratio, and reducing the gradient value can include multiplying the gradient value by a scaling factor based on that ratio. For example, the scaling factor can be based on the reciprocal of the ratio. Alternatively or additionally, the scaling factor can be based on a threshold. For example, the threshold can be a value in the range of 0.01 to 0.16, including 0.01 and 0.16. The scaling factor can be based on a combination of a ratio and a threshold. For example, the scaling factor can be based on a threshold multiplied by the reciprocal of the ratio.

[0013] Alternatively, the threshold value can be based on the learning rate. For example, the threshold can be proportional to the inverse of the learning rate. The threshold value can also be based on the batch size. For example, a smaller value for the threshold can be chosen for larger batch sizes (which provides stronger clipping).

[0014] The gradient norm and parameter norm can be determined based on the parameters associated with a neuron in a neural network. That is, a neuron can be a single neuron, and the gradient norm and parameter norm can be unity norms.

[0015] The parameters of a neural network can be the weights of the neurons connected to the neural network, and the gradient norm can be determined based on the gradient associated with each corresponding weight connected to the neuron, and the parameter norm can be determined based on the weight value of each corresponding weight connected to the neuron.

[0016] The gradient norm and parameter norm can be determined based on the Frobenius norm. That is, the Frobenius norm of the gradient or parameter matrix associated with a neural network layer can be defined as the square root of the sum of the squares of each individual element of the matrix.

[0017] The gradient norm can be computed as the Frobenius norm over the gradient associated with the corresponding weights connected to the neuron, and the parameter norm can be computed as the Frobenius norm over the corresponding weights connected to the neuron.

[0018] The value of the gradient can be reduced based on the following equation:

[0019]

[0020] Among them, W l Let W be the weight matrix of layer l, where i is the index of a neuron in layer l (and therefore can be W).l (row vectors), It corresponds to parameter W i l The gradient of , where λ is the scalar threshold, and ||.|| F It is the Frobenius norm. ||W i l || F It can also be calculated as max(||W) i l || F This prevents zero initialization parameters from clipping its gradient to zero. ε can be 10. -3 Or other appropriate smaller values.

[0021] The neural network can be a deep residual neural network. The neural network can include residual blocks, where the residual blocks are unnormalized layers. That is, the residual blocks may not include batch normalization or other types of normalization layers. The residual blocks can include convolution, pooling, and / or nonlinear operations, but without activation normalization operations such as batch normalization. The nonlinearity can be a Gaussian Error Linear Unit (GELU) or a Corrected Linear Unit (ReLU). The convolution operation can be a grouped convolution. For example, the group width of a 3×3 convolution can be 128.

[0022] The parameters can be those associated with the convolutional layer. When the parameters are weights of the convolutional filter, the gradient and parameter norm can be computed over the fan-in range, including both channel and spatial dimensions. Adaptive gradient clipping methods can be applied to all layers of the network. However, the final output layer can be excluded. The initial convolutional layer can also be excluded.

[0023] The neural network can be a deep residual neural network comprising a four-stage backbone. Each stage can comprise a sequence of residual blocks with constant width and resolution. The backbone can comprise residual blocks in a ratio of 1:2:6:3 from stage one to stage four. That is, stage one can comprise one residual block, stage two can comprise two residual blocks, stage three can comprise six residual blocks, and stage four can comprise three residual blocks. Networks increasing in depth can have an increasing number of residual blocks consistent with the specified ratio. For example, the network could have five residual blocks in stage one, ten in stage two, thirty in stage three, and fifteen in stage four. Input layers, fully connected layers, and output layers typically do not form part of the backbone.

[0024] The width of each level can be twice the width of the previous level. For example, the width could be 256 at the first level, 512 at the second level, 1024 at the third level, and 2048 at the fourth level. In an alternative configuration, the width of the third and fourth levels could be 1536. For example, the width could be 256 at the first level, 512 at the second level, and 1536 at both the third and fourth levels. In another example, the width could be 256 at the first level, 1024 at the second level, and 1536 at both the third and fourth levels.

[0025] The residual block can be a bottleneck residual block. A bottleneck residual block can include a first grouped convolutional layer and a second grouped convolutional layer within the bottleneck. A typical bottleneck consists of only one convolutional layer within itself. It has been found that including a second convolutional layer within the bottleneck can significantly improve task performance with almost no impact on training time. For example, a bottleneck residual block can include a 1×1 convolutional layer that reduces the number of channels to form the bottleneck, the bottleneck comprising a first 3×3 grouped convolutional layer and a second 3×3 grouped convolutional layer, and a 1×1 convolutional layer to restore the number of channels.

[0026] The weights of convolutional layers in residual blocks can undergo scaled weight normalization. That is, the weights can be reparameterized based on the mean and standard deviation of the weights in the layer. Further details on scaled weight normalization can be found in “Characterizing signal propagation to close the performance gap in unnormalized resnets” by Brock et al. at the 9th International Conference on Learning Representations (ICL R) in 2021, the full text of which is incorporated herein by reference.

[0027] The input to the residual block can be reduced based on the variance of the input. The variance can be determined analytically. The final activation of the residual branch of the residual block can be scaled using a scalar parameter. The value of the scalar parameter can be 0.2. For example, the residual block can have the following form h. i+1 =h i +αf i (h i / β i ), where h i Let f represent the input up to the i-th residual block, and f i () denotes the function computed by the i-th residual branch. The function can be parameterized to preserve variance during initialization such that Var(f) holds for all i. i (z))=Var(z). As mentioned above, the scalar α can be 0.2. The scalar β can be determined by the standard deviation of the input predicted to the i-th residual block.i , Among them, Var(h i+1 ) = Var(h i )+α 2 In addition to skipping the path to reduce the input (h) i / β i Outside of the transition block where the operation is performed, and after the transition block, it is reset to h. i+1 =1+α 2 The expected variance. Further details can also be found in the paper by Brock et al. cited above.

[0028] The residual block can also include squeeze and excite layers. Squeeze and excite layers can process input activations according to the following sequence of functions: global average pooling, fully connected linear functions, scaled nonlinear functions, second fully connected linear functions, sigmoid functions, and linear scaling. For example, the layer output could be 2σ(FC(GELU(FC(pool(h))))))×h, where σ is the sigmoid function, FC is the fully connected linear function, pool is global average pooling, and h is the input activation. A scalar multiplier of 2 can be used to maintain the signal variance.

[0029] The residual block can also include a learnable scalar gain at the end of the residual branch of the residual block. The learnable scalar can be initialized with a zero value. The learnable scalar can be anything other than the scalar α discussed above.

[0030] As described above, this adaptive gradient clipping method enables training on data items within a batch to be independent, and therefore can be used in sequence modeling tasks where batch normalization is not feasible. Conventional gradient clipping is commonly used in language modeling, and this adaptive gradient clipping method can provide a favorable alternative in such applications. Further examples of suitable sequence modeling tasks are provided below. The neural network can be a transformer-type neural network, i.e., a neural network that includes one or more transformer layers. Transformer layers can typically include attention neural network layers, particularly self-attention neural network layers, optionally followed by a feedforward neural network. Transformer-type neural networks can be used for sequence modeling, and are explained in further detail below. The neural network can also be a generative adversarial network (GAN) type neural network. GANs are explained in further detail below.

[0031] The values ​​of the updated parameters can be based on a batch size of at least 1024 training data items. In previous work involving neural networks without normalizers, training on ImageNet with large batch sizes (such as 1024) was unstable. Using an adaptive gradient pruning method provides improved stability and enables training with a batch size of at least 1024. For example, a batch size of 4096 can be used.

[0032] Neural networks can be pre-trained. For example, a neural network may have already been trained on different datasets and / or training objectives before further training on a specific task of interest and / or using a specific dataset of interest. Therefore, the network can be pre-trained and then fine-tuned. This method can accept a neural network used for training as input and can provide an updated neural network as output.

[0033] The method may also include receiving a training dataset that includes image data. The gradient can be determined based on a loss function used to measure the performance of the neural network on the image processing task.

[0034] Gradient computation and parameter updates can be performed based on stochastic gradient descent or any other suitable optimization algorithm. This method can be combined with regularization methods such as dropout and stochastic depth. The dropout rate can increase with depth. The dropout rate can range from 0.2 to 0.5, inclusive. This method can also be combined with momentum-based update rules (such as Nesterov's momentum). Due to the improved stability of the training method, this method also enables the use of large learning rates to accelerate training.

[0035] Gradient determination can be based on sharpness-aware minimization. In sharpness-aware minimization, the loss function can include a regular loss based on the training task and a further loss based on the geometry of the minimum. This further loss seeks parameters located in a neighborhood with uniformly low loss values. In other words, it seeks a flatter minimum, which is considered to provide better generalization than a minimum with a sharp shape. Gradient determination can include performing a gradient ascent step to determine a modified version of the parameters, and performing a gradient descent step based on the modified version of the parameters to determine the gradient associated with the parameters. The gradient ascent step can be performed based on a subset of the training data items in the current batch. For example, one-fifth of the training data items in the current batch can be used. When combined with the adaptive gradient clipping method described above, it has been found that using a subset of the batch for the ascent step results in equivalent performance to using all the training data items in the batch. Therefore, the same benefits can be achieved at a much lower computational cost. When used in a distributed training system, the gradient in the gradient ascent step does not require synchronization between copies on different processing units. The gradient ascent step and the resulting modified parameters can remain local to the processing unit, and the gradient descent step is performed on the locally modified parameters. For distributed systems with fewer processing units or single-processor systems, the same effect can be achieved through gradient accumulation. Further details on sharpness-perceived minimization can be found in... https: / / openreview.net / forum?id=6Tm1mposlrM The full text of Foret et al.'s “Sharpness-awareminimization for efficiently improving generalization”, presented at the 9th International Conference on Learning Representation (ICLR) in 2021, is available at [link to document] and is incorporated herein by reference.

[0036] Data augmentation techniques such as RandAugment can be used to augment training datasets. The stability of the augmentations provided by the adaptive gradient cropping method allows for the use of strong augmentations without degrading task performance. For image data, RandAugment offers a selection of image transformations, including: identity, automatic contrast, equalization, rotation, exposure, color, tone separation, contrast, brightness, sharpness, cropping, and translation. Modified versions of training data items can be generated by randomly selecting one or more transformations. Further details about RandAugment can be found in “RandAugment: Practical automated data augmentation with a reduced search space” by Cubuk et al., 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, the entire contents of which are incorporated herein by reference. It should be understood that other sets of transformations may be appropriately used depending on the modality of the training data items.

[0037] Additionally or alternatively, other data augmentation techniques can be used. For example, a modified training data item can be generated by selecting a portion of a first training data item and replacing the corresponding portion in a second training data item with the selected portion from the first training data item. The position and size of the selected portion can be randomly selected. Multiple portions can be selected and used to replace the selected portion to generate the modified training data item. In the case of image data, the portion can be an image patch. The label of the modified training data item can be assigned based on the proportion of the first and second training data items present in the modified training data item. For example, if the selected portion of the first training data item constitutes 40% of the modified training data item and the second training data item constitutes the remaining 60%, the label of the modified training data item could be 0.4 for the class associated with the first training data item and 0.6 for the class associated with the second training data item. In similar data augmentation techniques, the selected portion of the first training data item can be hidden, i.e., the pixel value can be set to zero or a value representing black, or it can be replaced with random noise.

[0038] Another exemplary data augmentation technique involves generating modified training data items by interpolating a first training data item and a second training data item. The interpolation can be linear. Labels can be assigned to the modified training data items based on a weighted interpolation of the first and second training data items.

[0039] In one implementation, for a batch of training data items, RandAugment can be applied to all training data items in that batch, a partial selection / replacement technique can be applied to half of the training data items in that batch, and an interpolation technique can be applied to the remaining half of the training data items to generate augmented training data items for that batch. As mentioned above, the stability of the augmentations provided by the adaptive gradient pruning method allows for the use of strong augmentations without degrading task performance. Therefore, combinations of different data augmentation techniques can be beneficial in improving task performance, where task performance gradually improves with stronger data augmentation. Typical batch normalized neural networks do not benefit from using stronger data augmentations and may even have their performance compromised in some cases.

[0040] This method can be executed by a parallel or distributed processing system comprising multiple processing units. The method may further include receiving a training dataset comprising multiple training data items; generating multiple batches of training data items, each batch comprising a subset of the training data items from the training dataset; distributing the multiple batches of training data items to the multiple processing units; and training the neural network in parallel using the multiple processing units based on the distributed multiple batches of training data items. The multiple processing units may be part of different physical computing devices and / or located in different physical locations.

[0041] This method can be executed by one or more tensor processing units, one or more graphics processing units, or other types of accelerator hardware. Parallel or distributed processing systems may include one or more graphics processing units or tensor processing units.

[0042] According to another aspect, a system is provided that includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding methods described above.

[0043] The system can be a parallel or distributed processing system. It may include one or more tensor processing units or one or more graphics processing units.

[0044] According to another aspect, one or more computer storage media are provided that store instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding methods described above.

[0045] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0046] Batch normalization is an important technique used to enable the training of very deep neural networks (e.g., those with hundreds or even thousands of layers). Batch normalization improves training stability and allows for the use of large batch sizes during training, which can significantly reduce overall training time. However, batch normalization is computationally and memory-intensive, negating some of the benefits of using larger batch sizes. For example, it has been estimated that batch normalization accounts for approximately one-quarter of the training time on the ResNet-50 architecture using Titan X Pascal GPUs on ImageNet.

[0047] Furthermore, batch normalization introduces dependencies between training data items within a batch. This increases the difficulty of implementing training on parallel or distributed processing systems and using accelerator hardware (such as tensor processing units and graphics processing units), which may be necessary for effectively training very deep neural networks. Batch normalization is also particularly sensitive to the underlying hardware used to perform training, and the results may be difficult to replicate on other hardware systems.

[0048] Previous work replacing batch normalization has produced networks that offer comparable accuracy on benchmark datasets such as ImageNet. However, task performance begins to degrade in these “no normalizer” networks with large batch sizes (e.g., greater than 1024 on ImageNet).

[0049] As discussed above, the inventors have identified a significant difference in the gradient norm to parameter norm ratio between batch-normalized and non-normalized networks during training. Therefore, the adaptive gradient pruning technique described herein can replicate the beneficial effects of batch normalization in non-normalized networks, ensuring that the gradient norm to parameter norm ratio remains within an acceptable range during training, thus providing more stable parameter updates. This stability enables training with large batch sizes to improve the training efficiency of non-normalized networks while maintaining high task performance. For example, neural networks trained using the adaptive gradient pruning technique train up to 8.7 times faster than the current state-of-the-art EfficientNet-B7 network on ImageNet, achieving test accuracy comparable to ImageNet.

[0050] Furthermore, gradient pruning has significantly lower computational and memory costs than batch normalization. Additionally, since there are no dependencies on training data items within a batch, training can be more easily performed on parallel and distributed processing systems. No special consideration is required regarding how to allocate training data items to batches or the parallel computation of batch statistics. Therefore, this training method is particularly well-suited for parallel and distributed processing systems and accelerator hardware.

[0051] On the other hand, adaptive gradient pruning is effective for both small and large batch sizes, while batch normalization and other normalization optimizers often perform poorly. Therefore, adaptive gradient pruning is also effective when computational resources are limited.

[0052] The enhanced stability provided by the adaptive gradient pruning method also enables training with strong data augmentation (such as RandAugment), which further improves the network's generalization ability and task performance. Attached Figure Description

[0053] Figure 1 An example neural network training system is shown.

[0054] Figure 2 A schematic diagram of a neural network is shown.

[0055] Figure 3 This is a flowchart illustrating the process used to train a neural network.

[0056] Figure 4 A schematic diagram of the residual neural network architecture is shown.

[0057] Figure 5 A schematic diagram of the bottleneck residual block is shown.

[0058] Figure 6 This is a graph showing the training latency versus image recognition accuracy for exemplary embodiments and various prior art neural network models.

[0059] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0060] Figure 1 An example neural network training system 100 for training a neural network is illustrated. A set of neural network parameters 105 and a training dataset 110 can be provided as input to the neural network training system 100. The neural network training system 100 is configured to process the neural network parameters 105 and the training dataset 110 to provide updated neural network parameters 115. That is, the values ​​of the input neural network parameters 105 can be changed in an attempt to improve the performance of the neural network on a specific predefined task. In particular, the neural network training system 100 is configured to update the neural network parameters 105 using an adaptive gradient pruning technique. In the adaptive gradient pruning technique, the gradient associated with the neural network parameters 105 is determined. The ratio of the gradient norm to the parameter norm is determined and compared with a threshold. In response to determining that the ratio exceeds the threshold, the value of the gradient is reduced such that the ratio is equal to or below the threshold, and the values ​​of the parameters are updated based on the reduced gradient value. (Refer to below) Figure 3Further details regarding the adaptive gradient pruning technique are provided. The neural network training system 100 can be configured to provide updated neural network parameters 115 as output.

[0061] The neural network training system 100 may alternatively retrieve input neural network parameters 105 and / or training dataset 110 from local data storage 120 or memory 125 of the system 100. The neural network training system 100 may also be configured to generate an initial set of values ​​for the neural network parameters. The neural network training system 100 may also be configured to repeatedly update the neural network parameters 105 until a predefined stopping criterion is reached, and may provide a final set of updated neural network parameters 115 as output.

[0062] Training dataset 110 may include multiple training data items suitable for the task, and optionally include a set of labels corresponding to the target output that the neural network should produce when processing the training data items. For example, training dataset 110 may include image data, video data, audio data, speech data, sensor data, data representing environmental states, and other types of data, as discussed in more detail below. Tasks may include image recognition, object detection, image segmentation, speech recognition, machine translation, generating actions for controlling robots / mechanical / electrical agents, and other tasks, as discussed in more detail below.

[0063] Typically, a neural network training system 100 may include multiple processing units 130A…N, where each processing unit includes local memory 135A…N. Therefore, Figure 1 The neural network training system 100 can be considered a parallel or distributed processing system. It should be understood that the processing units 130A…N can be arranged in various different architectures and configurations that are deemed appropriate by those skilled in the art. For example, the neural network training system 100 can be implemented using a graphics processing unit (GPU) or a tensor processor unit (TPU) or any type of neural network accelerator hardware. It should be understood that the processing units 130A…N can be distributed across multiple separate hardware devices in different physical locations communicating via appropriate computer networks, and do not need to reside on a single hardware device.

[0064] The neural network training system 100 can be configured to generate multiple batches of training data items, each batch comprising a subset of training data items from the training dataset 110. Alternatively, the received training dataset 110 can be pre-divided into multiple batches. The neural network training system 100 can be configured to distribute the multiple batches of training data items to multiple processing units 130A…N. The neural network training system 100 can be configured to train the neural network using the parallel processing capabilities of the multiple processing units 130A…N based on the multiple batches of training data items distributed to each processing unit 130A…N. In this context, the term "batch" is intended to encompass any grouping of training data items used to distribute to the processing units 130A…N. For example, when training a neural network using stochastic gradient descent, gradients can be computed based on "mini-batches" of training data items. This "mini-batch" of training data items can be further subdivided to distribute to the multiple processing units 130A…N. For example, each processing unit 130A…N can be configured to process 32 training data items each. The term "batch" is intended to include this further subdivision in the context of distributing training data items to processing units 130A…N. When "batch size" is mentioned in this disclosure, this can refer to the number of training data items used to determine gradients and update values. Therefore, this can refer to the size of a "mini-batch" in stochastic gradient descent prior to the subdivision and distribution of mini-batches to processing units 130A…N.

[0065] Multiple processing units 130A…N can each be configured to compute, in parallel, the corresponding network output for each training data item assigned to it based on the current value of the neural network parameter 105. As discussed in more detail below, the adaptive gradient pruning technique does not introduce any dependencies between training data items when computing the network output, and therefore, the computation of the network output can be performed in parallel and independently by each processing unit 130A…N. This contrasts with neural networks that include batch normalization layers, which introduce dependencies between training data items and may therefore require communication between processing units 130A…N to perform batch normalization operations, or alternatively introduce data scrambling operations, leading to further overhead. The adaptive gradient pruning technique enables neural networks without batch normalization layers to achieve task performance comparable (if not better) to those with batch normalization layers, while also being easier to implement and more efficient on parallel and distributed systems.

[0066] Each processing unit 130A…N can be configured to compute error values ​​or other learning signals based on a determined network output and a specific loss function used to train the neural network. Error values ​​can be backpropagated through the network to compute gradient values ​​in parallel on specific batches allocated to the processing units 130A…N. The computed gradient values ​​determined by each of the processing units 130A…N can be combined to determine the ratio of the gradient norm to the parameter norm and to update the parameter values ​​according to an adaptive gradient pruning technique. Updates to the parameter values ​​can be sent to each of the processing units 130A…N to apply the updates to a local copy of the parameters, or the update values ​​themselves can be sent to each of the processing units 130A…N when further training is required. It should be understood that other parallel implementations can be applied to implement the adaptive gradient pruning technique. For example, an asynchronous parallel implementation can be used, thereby allowing different local copies of the neural network parameters used by the processing units 130A…N. The determination of the ratio of the gradient norm to the parameter norm, the comparison of the ratio with a threshold, and the updating of the parameter values ​​can be performed in parallel and independently based on a batch of training data items distributed to the processing units. For example, parameter value updates and the distribution of updated parameter values ​​to processing units 130A…N can be performed using an appropriate asynchronous stochastic gradient descent method.

[0067] Although Figure 1 Parallel / distributed processing systems are described, but it should be understood that the neural network training system 100 does not need to be implemented as a parallel or distributed system and can be implemented using a single processing unit.

[0068] Figure 2 An example neural network 200 comprising multiple hidden layers 205A…N is shown. The neural network 200 processes input 210 through the multiple hidden layers 205A…N to provide output 215. Typically, the neural network 200 is trained to perform a specific task. For example, the neural network 200 can be trained to perform an image recognition task. Input 210 may be an image including pixel values ​​(or other image data), and output 215 may be a set of scores representing the probability that a particular object is present in the image.

[0069] Neural network 200 can be trained using conventional techniques such as stochastic gradient descent or other gradient-based methods, but modified to use the adaptive gradient pruning technique described below. Typically, for gradient-based training methods, one or more training data items are provided as input to neural network 200 to generate corresponding outputs. A loss function, such as cross-entropy loss, can be constructed to compare the generated outputs with the corresponding target outputs. The error value or other learning signal calculated from the loss function can be propagated back through the network from the output, in reverse order through multiple hidden layers 205A…N, and back to the input. In this way, the gradient of the loss function with respect to each parameter of the neural network can be calculated and used to update the parameter values.

[0070] In adaptive gradient pruning, the gradients associated with the parameters of a neural network are computed normally. However, these gradients can be modified before they are used to update the parameters. Specifically, as... Figure 3 As shown in the processing, in step 305, after determining the gradient associated with the parameters of the neural network in step 301, a ratio of the gradient norm to the parameter norm is determined. This ratio can be defined as the gradient norm divided by the parameter norm. At step 310, the determined ratio is compared with a threshold. At step 315, in response to determining that the ratio exceeds the threshold, the gradient value is reduced so that the ratio is equal to or below the threshold, thereby "clipping" the gradient. At step 320, the parameter values ​​are updated based on the reduced gradient value. At step 325, if the ratio does not exceed the threshold, the gradient value can be maintained, and at step 330, the parameter values ​​can be updated based on the maintained gradient value. In either case, the parameter value update can be performed according to the specific parameter update rules of the specific gradient-based training method employed.

[0071] Adaptive gradient pruning ensures stable parameter updates because, considering the scale of the parameters, updates are limited to a specific size. In some neural networks, such as very deep ones with tens, hundreds, or even thousands of layers, batch normalization is required for efficient training. This adaptive gradient pruning technique enables the efficient training of such neural networks without the need for batch normalization layers. Neural networks without batch normalization layers are referred to in this paper as "normalizer-free" neural networks.

[0072] The batch normalization layer takes the output of the hidden layers in the neural network as input and recenters and rescales the input. Initially, the input is modified so that the data has approximately zero mean and unit variance. If the initial normalization result is suboptimal, further scaling and shifting based on learnable parameters can be applied.

[0073] The mean and variance used for batch normalization are calculated based on a batch of training data items used for a specific parameter update step. Therefore, batch normalization introduces dependencies between training data items within a batch, making implementation more difficult on parallel or distributed processing systems. This is because communication between processing units may be required to compute the mean and variance of a batch of data, where the batch is split among processing units when computing the neural network's output. Without batch normalization, processing units can independently compute the network output for each input data item, eliminating the need for communication between them. Therefore, replacing batch normalization with adaptive gradient pruning removes the dependencies between training data items within a batch and restores the ability of processing units to independently compute the network output. This makes training easier to implement on parallel or distributed processing systems and reduces the amount of communication required between processing units, thus improving the efficiency of parallel implementations. In some existing implementations, as an alternative to transferring batch normalization statistics between processing units, the training data items within a batch can be scrambled each time batch normalization is run, so that processing units may be assigned different subsets of the batch each time. However, this scrambling operation also incurs additional overhead that reduces the efficiency of parallel / distributed implementations. The use of adaptive gradient pruning avoids the need for scrambling and reduces overhead in parallel / distributed implementations.

[0074] Neural networks without normalizers trained with adaptive gradient pruning offer task performance comparable to, if not better than, those with batch normalization. The increased stability achieved through adaptive gradient pruning enables training with large batch sizes, reducing overall training time while maintaining task performance. Batch normalization is also a computationally expensive operation, and its replacement helps reduce the computational requirements for training large-scale neural networks.

[0075] Conventional gradient pruning methods only consider the magnitude of the gradient; they do not consider the magnitude of the parameters themselves or the ratio of the gradient norm to the parameter norm. Using conventional gradient pruning methods in networks without normalizers does not yield the full benefits offered by using adaptive gradient pruning techniques. In particular, training with conventional gradient pruning is sensitive to depth, batch size, and learning rate, requiring fine-grained tuning when changing any of these factors. Diminishing returns are also observed for larger networks when using conventional gradient pruning. The use of a ratio for gradient pruning provides improved stability in parameter updates, replicating the properties and benefits of batch normalization that conventional gradient pruning fails to achieve.

[0076] Further details of the adaptive gradient technique will now be described. The gradient value can be reduced by multiplying it by a scaling factor. In one example, the scaling factor is based on a threshold. In another example, the scaling factor is based on a ratio, and could be based on the reciprocal of that ratio. The scaling factor can be based on a combination of a threshold and a ratio; for example, the scaling factor could be based on the threshold multiplied by the reciprocal of the ratio.

[0077] Gradient norm and parameter norm can be based on the Frobenius norm. The Frobenius norm of matrix A is defined as the square root of the sum of the squares of each individual element of the matrix:

[0078]

[0079] The norm can be a unit-wise norm, meaning it can be computed based on the gradient / parameter values ​​associated with a specific neuron in a neural network within a particular layer. For example, the norm can be computed based on the parameters associated with the incoming connections to the neuron and their corresponding gradients. Alternatively, outgoing connections can be used if appropriate.

[0080] In one implementation, the gradient value can be reduced and updated based on the following equation:

[0081]

[0082] Among them W l W is the weight matrix of layer l, where i is the index of a neuron in layer l (and therefore can be W when the norm is computed in unit mode). l (row vectors), It corresponds to parameter W i l The gradient of , λ is the scalar threshold, and ||.|| F It is the Frobenius norm. ||W i l || F It can also be calculated as max(||W) i l || F This prevents zero initialization parameters from clipping its gradient to zero. ε can be 10. -3 Or other appropriate smaller values.

[0083] In one example, the threshold could be a value in the range of 0.01 to 0.16, inclusive. It should be understood that other thresholds may be appropriately chosen depending on the type of network and the batch size of the training data items processed in a particular parameter update step. The value of the threshold can be based on the batch size. For example, a smaller value of the threshold could be chosen for a larger batch size (which provides stronger gradient clipping).

[0084] The values ​​of the updated parameters can be based on a batch size of at least 1024 training data items. In previous work involving neural networks without normalizers, training on ImageNet with large batch sizes (such as 1024) was unstable. As discussed above, the use of adaptive gradient pruning provides improved stability and enables training with batch sizes of at least 1024. For example, a batch size of 4096 can be used.

[0085] Adaptive gradient pruning is effective for both small and large batch sizes. Batch normalization and other normalization optimizers often perform poorly with small batch sizes. Therefore, adaptive gradient pruning is effective when computational resources are limited and small batch sizes are necessary.

[0086] Adaptive gradient pruning can be combined with regularization methods such as dropout and random depth. The dropout rate can increase with depth. That is, for networks with more layers, the dropout rate may be larger. The dropout rate can range from 0.2 to 0.5, inclusive. Adaptive gradient pruning can also be combined with momentum-based update rules (such as Nesterov's momentum). Due to the improved stability of training methods, adaptive gradient pruning also enables the use of large learning rates to accelerate training.

[0087] Gradient determination can be based on sharpness-aware minimization. In sharpness-aware minimization, the loss function can include a regular loss based on the training task and a further loss based on the geometry of the minimum. This further loss seeks parameters located in a neighborhood with uniformly low loss values. In other words, it seeks a flatter minimum, which is considered to provide better generalization than a minimum with a sharp shape. Gradient determination can include performing a gradient ascent step to determine a modified version of the parameters, and performing a gradient descent step based on the modified version of the parameters to determine the gradient associated with the parameters. The gradient ascent step can be performed based on a subset of the training data items in the current batch. For example, one-fifth of the training data items in the current batch can be used. When combined with adaptive gradient clipping techniques, it has been found that using a subset of the batch for the ascent step results in equivalent performance to using all training data items in the batch. Therefore, the same benefits can be achieved at a much lower computational cost. When used in a distributed training system, the gradient in the gradient ascent step does not require synchronization between copies on different processing units. The gradient ascent step and the resulting modified parameters can remain local to the processing unit, and the gradient descent step is performed on the locally modified parameters. For distributed systems with fewer processing units or single-processor systems, the same effect can be achieved through gradient accumulation. Further details on sharpness-perceived minimization can be found in... https: / / openreview.net / forumid=6Tm1mposlrM The full text of “Sharpness-aware minimization for efficiently improving generalization” by Foret et al., presented at the 9th International Conference on Learning Representation (ICLR) in 2021, is available at [link to relevant document] and is incorporated herein by reference.

[0088] Return to reference Figure 1 The neural network training system 100 can be configured to augment the training dataset 110 to generate further training data items. Additionally or alternatively, the received training dataset 110 can be an augmented training dataset that includes a set of unmodified training data items as well as modified training data items.

[0089] The enhanced stability provided by adaptive gradient cropping allows for the use of strong augmentations without degrading task performance. One exemplary data augmentation technique that can be used is called "RandAugment". Details about RandAugment can be found in "RandAugment: Practical automated data augmentation with a reduced search space" by Cubuk et al., 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, the entire contents of which are incorporated herein by reference. In short, however, for image data, RandAugment provides a selection of image transformations, including: recognition, automatic contrast, equalization, rotation, exposure, color, tone separation, contrast, brightness, sharpness, cropping, and translation. It should be understood that, depending on the modality of the training data items, other sets of transformations may be appropriately used. Modified versions of the training data items can be generated by randomly selecting one or more transformations. In one example, four transformations are randomly selected and applied sequentially to the training data items to generate modified training data items for use in training a neural network using adaptive gradient cropping.

[0090] Additionally or alternatively, other data augmentation techniques can be used. For example, a modified training data item can be generated by selecting a portion of a first training data item and replacing the corresponding portion in a second training data item with the selected portion from the first training data item. The position and size of the selected portion can be randomly selected. Instead of a single portion, multiple portions can be selected and used to replace the selected portion to generate the modified training data item. In the case of image data, the portion can be an image patch.

[0091] Training data items modified in this way can be labeled based on the proportion of the first and second training data items present in the modified training data items. For example, if a selected portion of the first training data item constitutes 40% of the modified training data item, and the second training data item constitutes the remaining 60%, then the label of the modified training data item could be 0.4 for the class associated with the first training data item and 0.6 for the class associated with the second training data item. In similar data augmentation techniques, the selected portion of the first training data item can be hidden, i.e., the pixel value can be set to zero or a value representing black, or it can be replaced with random noise.

[0092] Another exemplary data augmentation technique suitable for use with adaptive gradient clipping involves generating modified training data items by interpolating a first training data item and a second training data item. The interpolation can be linear. Labels can be assigned to the modified training data items based on a weighted interpolation of the first and second training data items.

[0093] In one implementation, for a batch of training data items, RandAugment can be applied to all training data items in that batch, a partial selection / replacement technique can be applied to half of the training data items in that batch, and an interpolation technique can be applied to the remaining half of the training data items to generate further training data items for that batch. As mentioned above, the stability of the augmentation provided by the adaptive gradient pruning method allows for the use of strong augmentations without degrading task performance. Therefore, combinations of different data augmentation techniques can be beneficial in improving task performance. It has been observed that task performance can gradually improve with stronger data augmentation. Typical batch normalized neural networks do not benefit from using stronger data augmentations and may even have their performance compromised in some cases.

[0094] The received neural network parameters 105 can be parameters of a pre-trained neural network, and the neural network training system 100 can be used to further train the neural network. For example, the neural network may have already been trained on different datasets and / or training objectives before further training on a specific task of interest and / or using a specific dataset of interest. Therefore, the neural network training system 100 can be used in the context of transfer learning. In one example, the neural network is pre-trained on a dataset comprising approximately 300 million labeled images from 18,000 classes. The neural network is then fine-tuned for image recognition on the ImageNet dataset. Both the pre-training and fine-tuning phases can be performed using the neural network training system 100 and adaptive gradient clipping techniques.

[0095] Adaptive gradient pruning can be applied to neural networks with deep residual neural network architectures. Residual neural network architectures consist of residual blocks, and as discussed above, using adaptive gradient pruning, these residual blocks can be without normalized layers. Residual blocks can include operations such as convolution, pooling, and / or other linear and non-linear operations, but without batch normalization operations.

[0096] In convolutional layers, gradients and parameter norms can be computed over a fan-in range that includes both channel and spatial dimensions. Adaptive gradient pruning techniques can be applied to all layers of the network; however, the final output layer can be excluded, as can the initial convolutional layers.

[0097] Figure 4A schematic diagram of the architecture of a residual neural network 400 is provided. The residual neural network 400 can be a neural network without a normalizer. The residual neural network 400 includes an initial set of one or more hidden layers referred to as the "stem" 405. Following the stem, the residual neural network 400 includes another set of hidden layers referred to as the "backbone" 410. Finally, the residual neural network 400 includes another set of one or more layers 415, which can be specific to the task being performed, such as a classification layer.

[0098] The backbone 410 of the residual neural network 400 may include multiple repeating residual blocks. Each residual block may include the same sequence of operations (a sequence of neural network layers), and more than one type of residual block may exist. The residual blocks may be arranged in levels, where each level includes a sequence of residual blocks with a constant width and resolution. Figure 4 In the structure, the backbone 410 includes a first stage 410A with one residual block, a second stage 410B with two residual blocks, a third stage 410C with six residual blocks, and a fourth stage 410D with three residual blocks. The backbone 410 can include multiple residual blocks in a ratio of 1:2:6:3 from the first stage to the fourth stage. The depth of the neural network can be increased by consistently increasing the number of residual blocks in each stage according to a specified ratio. For example, the neural network could have five residual blocks in the first stage, ten residual blocks in the second stage, thirty residual blocks in the third stage, and fifteen residual blocks in the fourth stage.

[0099] The width of each level can be twice the width of the previous level. For example, the width could be 256 at the first level, 512 at the second level, 1024 at the third level, and 2048 at the fourth level. In an alternative configuration, the width of the third and fourth levels could be 1536. For example, the width could be 256 at the first level, 512 at the second level, and 1536 at both the third and fourth levels. In another example, the width could be 256 at the first level, 1024 at the second level, and 1536 at both the third and fourth levels. Transition blocks can be used between levels. Figure 4 (Not shown in the image) is used to handle changes in width.

[0100] As mentioned above, residual blocks can include nonlinearities. The nonlinearity can be a Gaussian Error Linear Unit (GELU), a Corrected Linear Unit (ReLU), or other suitable nonlinear operations. The convolution operation can be a grouped convolution. For example, the group width of a 3×3 convolution can be 128.

[0101] The residual block can be a bottleneck residual block. Figure 5An exemplary bottleneck residual block 500 is shown. The bottleneck residual block 500 includes a 1×1 convolutional layer 505 that reduces the number of channels to form a bottleneck. For example, the number of channels can be halved. A first grouped convolutional layer 510 and a second grouped convolutional layer 515 are present within the bottleneck. A typical bottleneck consists of only one convolutional layer within the bottleneck. It has been found that including a second convolutional layer in the bottleneck can improve task performance with almost no impact on training time. Figure 5 In the middle, the bottleneck consists of two 3x3 grouped convolutional layers 510 and 515. Another 1×1 convolutional layer 520 is provided, which recovers the number of channels. Nonlinearity ( Figure 5 (Not shown in the image) can be followed by one or more convolution operations.

[0102] Residual block 500 also includes two scaling parameters, β525 and α530. The β parameter 525 reduces the input of residual block 500 and can be based on the variance of the input. The variance can be determined analytically. The final activation of the residual branches of residual block 500 (including bottleneck paths) can be scaled by the α scalar parameter 530.

[0103] Using scaling parameters 525 and 530, residual block 500 can implement a function h of the following form. i+1 =h i +αf i (h i / β i ), where h i Let f represent the input up to the i-th residual block 500, and f i () denotes the function computed by the i-th residual branch. The function can be parameterized to preserve variance during initialization such that Var(f) holds for all i. i (z))=Var(z). The scalar α530 can be 0.2. The scalar β can be determined by the standard deviation of the input predicted to the i-th residual block. i 525, Var(h) i+1 ) = Var(h i )+α 2 In addition to skipping the path to reduce the input (h) i / β i Outside of the transition block where the operation is performed, and after the transition block, it is reset to h. i+1 =1+α 2The expected variance. Further details can be found in “Characterizing signal propagation to close the performance gap in unnormalized resnets” by Brock et al. at the 9th International Conference on Learning Representations (ICLR) in 2021, the full text of which is incorporated herein by reference.

[0104] The weights of the convolutional layer with residual block 500 can undergo scaled weight normalization. That is, the weights can be reparameterized based on the mean and standard deviation of the weights in the layer. Further details on scaled weight normalization can be found in “Characterizing signal propagation to close the performance gap in unnormalized resnets” by Brock et al. at the 9th International Conference on Learning Representations (ICLR) in 2021, the full text of which is incorporated herein by reference.

[0105] The residual block can also include squeezing and activation layers. These layers can process input activations according to a sequence of functions: global average pooling, a fully connected linear function, a scaled nonlinear function, a second fully connected linear function, a sigmoid function, and linear scaling. For example, the layer output could be 2σ(FC(GELU(FC(pool(h)))))×h, where σ is the sigmoid function, FC is the fully connected linear function, pool is global average pooling, and h is the input activation. A scalar multiplier of 2 can be used to maintain the signal variance. In one example, the squeezing and activation layers are provided after the final 1×1 convolutional layer 520 and before scaling by α530.

[0106] The residual block can also include a learnable scalar gain at the end of the residual branch of the residual block. The learnable scalar can be initialized with a zero value. The learnable scalar can be anything other than the scalar α530 discussed above.

[0107] As discussed above, residual neural networks may include transition blocks between backbone levels. These transition blocks may have... Figure 5 The bottleneck residual block 500 shown is similar in form. However, the first 3×3 grouped convolutional layer 510 can be modified to increase the stride value; for example, the convolution operation can use a stride of 2 to change the width of the output activation. Additionally, the skip path (the path that bypasses the bottleneck layer) can include pooling layers and 1×1 convolutional layers with varying widths. The skip path can also be modified to occur after β scaling 525 instead of before the branch as in residual block 500.

[0108] Now for reference Figure 6 The graph illustrates the relationship between training latency and image recognition accuracy, comparing an exemplary non-normalizer neural network trained using the aforementioned techniques (solid line) with representative samples (dashed line) of the best-performing image recognition neural network model based on residual neural networks. More specifically, the exemplary non-normalizer neural networks trained using the aforementioned techniques (labeled NFNet-F0 to F5) include, as... Figure 5 The bottleneck residual blocks are shown. Each exemplary neural network has a four-level backbone with the ratio 1:2:6:3 as described above. The F0 neural network is the basic network with the minimum number of residual blocks (i.e., 1, 2, 6, and 3 residual blocks in each corresponding level). Each subsequent network has the next integer value in the ratio, i.e., the F1 neural network has 2, 4, 12, and 6 residual blocks in each corresponding level, the F2 neural network has 3, 6, 18, and 9 residual blocks in each corresponding level, and so on. From the first level to the fourth level, the width of each level is [256, 512, 1536, 1536].

[0109] Figure 6 The graph in the figure shows the training latency, which is measured as the median of the observed wall-clock time required to perform a single training step using a TPUv3 with a batch size of 32 devices and 32 training data items on each device over 5000 training steps. The neural network is evaluated using the ImageNet top-1 accuracy benchmark.

[0110] from Figure 6 As can be seen, the exemplary non-normalizer neural network provides higher image recognition accuracy and is also trained more efficiently.

[0111] As mentioned above, adaptive gradient pruning can be used to train neural networks to perform specific tasks, and examples are discussed below.

[0112] Neural networks can be configured to accept any kind of digital data input and generate any kind of score, classification, or regression output based on the input.

[0113] For example, if the input to a neural network is an image or features already extracted from an image, the output generated by the neural network for a given image can be a score for each of a set of object categories, where each score represents an estimated probability that the image contains an object belonging to that category. In other words, a neural network can perform image / object recognition tasks. A neural network can also provide an indication of the location of detected objects within the image as output, and thus can perform image segmentation.

[0114] As another example, if the input to a neural network is a sequence of texts in one language, the output generated by the neural network can be a score for each of a set of text fragments in another language, where each score represents an estimated probability that the text fragment in the other language is an appropriate translation of the input text into the other language.

[0115] As another example, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could be a score for each of a set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the utterance.

[0116] More generally, neural networks can be used in language modeling systems, image processing systems, or action selection systems. Neural networks can be used for both supervised and unsupervised learning tasks. For example, supervised learning tasks can include classification tasks, such as image processing tasks, speech recognition tasks, natural language processing tasks, word recognition tasks, or optical character recognition tasks. Unsupervised learning tasks can include reinforcement learning tasks, where an agent interacts with one or more real or simulated environments to achieve one or more objectives.

[0117] The input data for a neural network can include one or more of the following: image data, motion image / video data, motion data, speech data, audio data, electronic documents, data representing environmental states, and / or data representing actions. For example, image data can include color or monochrome pixel value data. Such image data can be captured from image sensors such as cameras or LiDAR sensors. Audio data can include data defining an audio waveform, such as a series of values ​​in the time and / or frequency domains defining the waveform; the waveform can represent speech in natural language. Electronic document data can include text data representing words in natural language. Data representing environmental states can include any kind of sensor data, including, for example: data characterizing the state of a robot or vehicle, such as posture data and / or position / velocity / acceleration data; or data characterizing the state of an industrial plant or data center, such as sensed electronic signals, such as sensed current and / or temperature signals. Data representing actions can include, for example, position, velocity, acceleration, and / or torque control data, or data used to control the operation of one or more devices in an industrial plant or data center. This data can typically relate to a real or virtual (e.g., simulated) environment.

[0118] The output data of a neural network can similarly include any kind of data. For example, in a classification system, the output data may include class labels for the input data items. In a regression task, the output data may predict the values ​​of continuous variables, such as control variables used to control electronic or electromechanical systems (such as robots, vehicles, data centers, or factories). In another example of a regression task operating on image or audio data, the output data may define one or more locations in the data, such as the location of an object or the location of one or more corners of an object's bounding box, or the temporal location of a sound feature in an audio waveform. In a reinforcement learning system, the output data may include, for example, data representing an action, as described above, to be performed by an agent operating in the environment (e.g., a mechanical agent, such as a robot or vehicle).

[0119] Data representing actions can include, for example, action-value (Q-value) data defining the action, or parameterized probability distributions sampled to determine the action, or data that directly defines the action, such as in a continuous action space. Therefore, in a reinforcement learning system, the neural network can directly parameterize the probability distribution of the action selection policy, or it can learn to estimate the value of the action-value function (Q-value). In the latter case, multiple memories and corresponding output networks can share a common embedding network to provide Q-values ​​for each available action.

[0120] A Transformer neural network is a self-attention feedforward sequence model. A Transformer neural network consists of an encoder and a decoder. The encoder maps the input sequence to an encoding. The decoder processes the encoding to provide an output sequence. Examples of input and output sequences are provided below. Both the encoder and decoder use self-attention, which guides the encoder / decoder to focus on the most relevant part of the sequence at the current time step, replacing the need for recurrent connections. Further details of the Transformer model can be found in "Attention Is All You Need" presented by Vaswani et al. at the 31st Conference on Neural Information Processing Systems (NIPS 2017) in Long Beach, California. https: / / papers.nips.cc / paper / 7181- attention-is-all-you-need.pdf It is available at [website address], and its entire content is incorporated herein by reference.

[0121] Transformer neural networks can be configured to receive an input sequence (i.e., a sequence of inputs, each with a corresponding input at each of a plurality of input positions) and process the input sequence to generate an output or an output sequence.

[0122] For example, a transformer neural network can be part of a reinforcement learning system that selects actions to be performed by a reinforcement learning agent interacting with the environment. It should be understood that other types of neural networks can be used in conjunction with reinforcement learning systems. To enable the agent to interact with the environment, the reinforcement learning system can receive an input sequence comprising a sequence of observations representing different states of the environment. The system can generate an output specifying one or more actions to be performed by the agent in response to the received input sequence (i.e., in response to the last observation in the sequence). That is, the observation sequence includes a current observation representing the current state of the environment and one or more historical observations representing past states of the environment.

[0123] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment. For example, the agent could be a robot that interacts with the environment to perform a specific task, such as locating an object of interest in the environment, moving the object of interest to a specified location in the environment, or navigating to a specified destination in the environment; or the agent could be an autonomous or semi-autonomous land, air, or sea vehicle that navigates in the environment.

[0124] In these implementations, observations can include one or more of, for example, images, object location data, and sensor data, to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or actuators.

[0125] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the global or relative pose of an object held by the robot.

[0126] In the case of robots or other mechanical agents or vehicles, observation can similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative posture of one or more parts of the agent. Observation can be defined in one, two, or three dimensions and can be absolute and / or relative.

[0127] Observations may also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or, for example, image or video data from a camera or LIDAR sensor, such as data from a sensor located separately from the agent in the environment.

[0128] In the case of electronic agents, observation may include data from one or more sensors monitoring a part of the plant or service facility, such as current, voltage, power, temperature, and data from other sensors, and / or electronic signals representing the function of the electronic and / or mechanical items of the equipment.

[0129] In these implementations, the action can be a control input for controlling the robot, such as torque or higher-level control commands for the robot's joints, or a control input for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as torque or higher-level control commands for the vehicle's control surfaces or other control elements.

[0130] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or a component of another mechanical agent. Action data may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which has an impact on the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions for controlling navigation (such as steering) and movement (e.g., braking and / or acceleration of the vehicle).

[0131] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment. Training the agent in a simulated environment allows the agent to learn from a large amount of simulated training data while avoiding the risks associated with training an agent in a real-world environment, such as damaging the agent due to performing poorly chosen actions. The agent trained in the simulated environment can then be deployed in a real-world environment.

[0132] For example, the simulation environment could be a simulation of a robot or vehicle, and a reinforcement learning system could be trained on the simulation. Alternatively, the simulation environment could be a motion simulation environment, such as a driving or flight simulation, and the agent could be a simulated vehicle navigating through the motion simulation. In these implementations, actions could be control inputs used to control the simulated user or the simulated vehicle.

[0133] In another example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game.

[0134] In another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediate or precursor chemicals, and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, actions are possible folding actions for folding the protein chain or actions for assembling the precursor chemical / intermediate, and the desired outcome can include, for example, folding the protein so that it is stable and enables it to perform a specific biological function or provide an efficient synthetic pathway for the chemical. As another example, the agent can be a mechanical agent that automatically performs or controls protein folding actions selected by the system without human interaction. Observations can include direct or indirect observation of protein states and / or can be derived from simulation.

[0135] In a similar manner, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharmaceutical chemical, and the agent is a computer system used to determine the elements of the pharmaceutical chemical and / or the synthetic pathway of the pharmaceutical chemical. The drug / synthesis can be designed based on rewards derived from the drug target, for example, in a simulation. As another example, the agent can be a mechanical agent that performs or controls the drug synthesis.

[0136] In some applications, the agent can be a static or mobile software agent, i.e., a computer program configured to operate autonomously and / or alongside other software agents or personnel to perform tasks. For example, the environment can be an integrated circuit routing environment, and the system can be configured to learn to perform routing tasks for routing interconnects of integrated circuits such as ASICs. The reward (or cost) can then depend on one or more routing metrics, such as interconnect resistance, capacitance, impedance, loss, speed or propagation delay, physical line parameters (such as width, thickness, or geometry), and design rules. Observations can be observations of component locations and interconnects; actions can include component placement actions (e.g., defining component locations or orientations) and / or interconnect routing actions (e.g., interconnect selection and / or placement actions). Thus, a routing task can include placing components, i.e., determining the location and / or orientation of components in an integrated circuit, and / or determining the routing of interconnects between components. Once the routing task has been completed, an integrated circuit, such as an ASIC, can be manufactured based on the determined placement and / or routing. Alternatively, the environment can be a data packet communication network environment, and the agent is a router that routes data packets on the communication network based on observations of the network.

[0137] Typically, in a simulated environment, observations may include simulated versions of one or more of the previously described observations or observation types, and actions may include simulated versions of one or more of the previously described actions or action types.

[0138] In some other applications, the agent can control actions in real-world environments, including, for example, equipment items in a data center or mains power or water distribution system, or in a plant or service facility. Observations can then relate to the operation of the plant or facility. For example, observations could include observations of the power or water usage of equipment, or observations of power generation or distribution control, or observations of resource usage or waste generation. The agent can control actions in the environment to, for example, improve efficiency by reducing resource usage, and / or reduce the environmental impact of operations in the environment, for example, by reducing waste. Actions can include actions that control equipment items in a plant / facility or impose operating conditions on equipment items in a plant / facility, and / or actions that cause changes in settings during the operation of the plant / facility, such as adjusting or turning on / off components of the plant / facility.

[0139] In some further applications, the environment is a real-world environment, and the agent manages task distribution across computing resources (e.g., on mobile devices and / or in data centers). In these implementations, actions may include assigning tasks to specific computing resources.

[0140] Typically, in the applications described above, the environment is a simulated version of a real-world environment. Once the system / method has been trained in the simulation, it can then be applied to the real-world environment. That is, control signals generated by the system / method can be used to control the agent to perform tasks in the real-world environment in response to observations from the real-world environment. Optionally, the system / method can continue training in the real-world environment based on one or more rewards from the real-world environment.

[0141] Optionally, in any of the above implementations, observations at any given time step may include data from previous time steps, such as actions performed at previous time steps, rewards received at previous time steps, etc., which may be helpful in characterizing the environment.

[0142] In another example, a transformer neural network can be part of a neural machine translation system. That is, if the input sequence is a sequence of words in the source language, such as a sentence or phrase, the output can be a translation of the input sequence into the target language, i.e., a sequence of words in the target language representing the sequence of words in the source language.

[0143] As another example, a transducer neural network can be part of a speech recognition system. That is, if the input sequence is a sequence of audio data representing spoken utterances, the output can be a sequence of morphemes, characters, or words representing the utterances; that is, a transcription of the input sequence. As another example, if the input to the neural network is a sequence representing spoken utterances, the output generated by the neural network can indicate whether a particular word or phrase (“hot word”) was spoken in the utterance. As yet another example, if the input to the neural network is a sequence representing spoken utterances, the output generated by the neural network can identify the natural language in which the utterances were spoken. Therefore, typically, network input can include audio data used to perform audio processing tasks, and network output can provide the results of audio processing tasks, such as recognizing words or phrases or converting audio to text.

[0144] As another example, a transformer neural network can be part of a natural language processing system. For instance, if the input sequence is a sequence of words in a natural language, such as a sentence or phrase, the output could be a summary of the input sequence in the natural language—a sequence with fewer words than the input sequence but retaining the basic meaning of the input sequence. As another example, if the input sequence is a sequence of words that forms a question, the output could be / definite a sequence of words that forms the answer to the question. As another example, the task could be a natural language understanding task, such as entailment tasks, paraphrasing tasks, text similarity tasks, sentiment analysis tasks, sentence completion tasks, grammar tasks, etc., which operates on a sequence of text in some natural language to generate an output that predicts a certain attribute of the text. Or automatic code generation from natural language (automatically generating TensorFlow code snippets from natural language). As another example, the task could be a text-to-speech task, where the input is text in natural language or features of text in natural language, and the network output defines a spectrogram or other data including audio defining the text spoken in natural language.

[0145] As another example, the task could be a text generation task, where the input is a sequence of text and the output is another sequence of text, such as the completion of the input text sequence, a response to a question posed in the input sequence, or a sequence of text about a topic specified by the first text sequence. As yet another example, the input to a text generation task could be anything other than text, such as an image, and the output sequence could be text describing the input.

[0146] As another example, a transformer neural network can be part of a computer-aided medical diagnostic system. For instance, the input sequence could be a data sequence from electronic medical records, and the output could be a predicted treatment sequence.

[0147] As another example, a transformer neural network can be part of an image processing system. For instance, the input sequence can be an image, i.e., a sequence of color values ​​from an image, and the output can be a sequence of text describing the image or video. As yet another example, the input sequence can be a text sequence or a different context, and the output can be an image describing the context.

[0148] Generative Adversarial Networks (GANs) are generative models trained using an adversarial process, in which a generator network and a discriminator network are trained simultaneously. During training, the generator network produces samples, and the discriminator network attempts to identify these samples as generated by the generator network, rather than as real training data items. The discriminator network's determination is used as a learning signal for the generator network to improve its generation capabilities, aiming to ensure that the generated samples cannot be distinguished from real training data items. Simultaneously, the discriminator network is also trained to improve its detection capabilities; thus, the two networks work together to enhance the generator network's abilities. Further details can be found in "Generative Adversarial Networks" by Goodfellow et al. in 2014, arXiv preprint arXiv:1406.2661. https: / / arxiv.org / pdf / 1406.2661.pdf It is available at [website address], and its entire content is incorporated herein by reference.

[0149] The generator can generate data items, which can be data representing still or moving images. In this case, the individual values ​​contained in the data items can represent pixel values, such as the values ​​of one or more color channels of a pixel. The training images used to train the discriminator network (and thus jointly train the generator network with it) can be real-world images captured by a camera.

[0150] For example, in one implementation, a user can use a trained generator network to generate images (still or moving images) from an image distribution (e.g., a distribution that reflects a database of training images used by the generator network, such as images that reflect real-world images).

[0151] Alternatively, the data item can be data representing a sound signal, such as the amplitude value of an audio waveform (e.g., natural language; in this case, the training examples could be samples of natural language, such as speech recorded by a microphone from a human speaker). In another possibility, the data item can be text data, such as a text string or other representations of words and / or sub-word units (word fragments) in a machine translation task. Therefore, the data item can be one-dimensional, two-dimensional, or higher-dimensional.

[0152] A generator network can generate data items conditioned on a conditional vector (target data) input to the generator network, which represents the target used to generate the data items. The target data can represent data of the same or different type or modality as the generated data items. For example, when trained to generate image data, the target data can define a label or class for one of the images, and the generated data items can then include example images of that type (e.g., an African elephant). Alternatively, the target data can include an image or an encoding of an image, and the generated data items can define another similar image—for example, when trained on facial images, the target data can include an encoding of a human face, and the generator network can then generate data items representing similar faces with different poses / lighting conditions. In another example, the target data can show an image of an object and include data defining the movement / change of viewpoint, and the generator network can generate an image of the object from a new viewpoint.

[0153] Alternatively, the target data may include text strings or spoken sentences, or encodings thereof, and the generator network may generate images corresponding to the text or speech (text-to-image synthesis) and vice versa. Alternatively, the target data may include text strings or spoken sentences, or encodings thereof, and the generator network may then generate corresponding text strings or spoken sentences in different languages. The system may also autoregressively generate video, particularly given one or more previous video frames.

[0154] In another implementation, the generator network can generate sound data, such as speech, in a similar manner. This can be conditioned on audio data and / or other data such as text data. Typically, the target data can define local and / or global features of the generated data items. For example, for audio data, the generator network can generate an output sequence based on a set of target data values. For example, the target data can include global features (which are the same when the generator network is generating a sequence of data items) that may include information defining the voice of a particular person, or voice style, or speaker identity, or language identity. The target data may additionally or alternatively include local features (i.e., not the same for the sequence of data items) that may include language features derived from the input text, optionally with intonation data.

[0155] In another example, the target data can define the motion or state of a physical object, such as the action and / or state of a robotic arm. The generator network can then be used to generate data items that predict future image or video sequences seen by a real or virtual camera associated with the physical object. In such an example, the target data could include one or more previous image or video frames seen by the camera. This data could be useful for reinforcement learning, for example, facilitating planning in a visual environment. More generally, the system learns to encode a probability density (i.e., distribution) that can be directly used for probabilistic planning / exploration.

[0156] In a further example, the generator network can be used for image processing tasks, such as denoising, deblurring, and image completion, by taking target data of a noisy or incomplete image; to perform image modification tasks by taking target data of a modified image; and for image compression, such as when using a generator network in an autoencoder. The system can be similarly used to process signals representing things other than images.

[0157] The input target data and output data items can typically be any kind of numerical data. Therefore, in another example, the input target data and output data items could each include tokens defining sentences in natural language. A generator network can then be used, for example, in a system used for machine translation, or it can be used to generate sentences representing concepts expressed with latent values ​​and / or additional data. Latent values ​​can be used, additionally or alternatively, to control the style or sentiment of the generated text. In a further example, the input and output data items could typically include speech, video, or time-series data.

[0158] In another example, the generator network can be used to generate further examples of data items for training another machine learning system. For instance, a generator network and a discriminator network can be jointly trained on a set of data items, and then the generator network can be used to generate new data items similar to those in the training dataset. A set of latent values ​​can be determined by sampling from the latent distribution of latent values. If the generator network has already been trained conditioned on additional data (e.g., labels), new data items can be generated conditioned on additional data (e.g., labels provided to the generator network). In this way, additional labeled data items can be generated, for example, to supplement the lack of unlabeled training data items.

[0159] For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, cause the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0160] The embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals), generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. However, the computer storage medium is not a propagation signal.

[0161] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. Apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0162] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts in a markup language document, as a single file dedicated to the program in question, or as multiple harmonizing files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0163] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, including one or more processors and computer-readable media, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0164] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, and the device can also be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, the processes and logic flows can be executed by a graphics processing unit (GPU), and the device can also be implemented as a GPU.

[0165] Computers suitable for executing computer programs include, for example, those based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.

[0166] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0167] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0168] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0169] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0170] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification within the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0171] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific or sequential order shown, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0172] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for training a neural network, comprising: Determine the gradients associated with the parameters of the neural network; Determine the ratio of the gradient norm to the parameter norm; Compare the ratio to a threshold; In response to determining that the ratio exceeds the threshold, the value of the gradient is reduced such that the ratio is equal to or lower than the threshold; and The parameter values ​​are updated based on the decreasing gradient values. The neural network is configured to process input data, which includes one or more of the following: image data, video data, audio data, and electronic documents.

2. The method according to claim 1, further comprising: In response to determining that the ratio is below the threshold, the value of the gradient is maintained and the value of the parameter is updated based on the maintained gradient value.

3. The method according to claim 1, wherein, Reducing the value of the gradient includes multiplying the value of the gradient by a scaling factor based on the threshold to reduce the value of the gradient.

4. The method according to claim 1, wherein, Reducing the value of the gradient includes multiplying the value of the gradient by a scaling factor based on the ratio to reduce the value of the gradient.

5. The method of claim 1, further comprising determining the gradient norm and the parameter norm based on parameters associated with a neuron of the neural network.

6. The method according to claim 5, wherein, The parameters of the neural network are the weights of the neurons connected to the neural network, and the method includes determining the gradient norm based on the gradient associated with each corresponding weight connected to the neuron, and determining the parameter norm based on the weight value of each corresponding weight connected to the neuron.

7. The method of claim 6, further comprising calculating the gradient norm as a Frobenius norm on the gradient associated with the corresponding weights connected to the neuron, and calculating the parameter norm as a Frobenius norm on the corresponding weights connected to the neuron.

8. The method according to claim 1, wherein, The value of the gradient is reduced based on the following equation: if in, Let be the weight matrix of the l-th layer, and i be the index of a neuron in the l-th layer. It corresponds to the parameter gradient, It is a scalar threshold, and || . || F It is the Frobenius norm.

9. The method according to claim 1, wherein, The neural network includes residual blocks, wherein the residual blocks are unnormalized layers.

10. The method according to claim 1, wherein, The neural network is a deep residual neural network that includes a four-level backbone.

11. The method according to claim 10, wherein, The backbone comprises residual blocks in a ratio of 1:2:6:3, from the first level to the fourth level.

12. The method according to claim 10 or 11, wherein, The width of each level is twice the width of the level before it.

13. The method according to claim 9, wherein, The residual block is a bottleneck residual block.

14. The method according to claim 1, wherein, The neural network is a transformer-type neural network.

15. The method according to claim 1, wherein, The value of the parameter is updated based on a batch size of at least 1024 training data items.

16. The method according to claim 1, wherein, The neural network has been pre-trained.

17. The method of claim 1, further comprising receiving a training dataset including image data, wherein, The gradient is determined based on a loss function used to measure the performance of the neural network on the image processing task.

18. The method according to claim 1, wherein, The method is executed by a parallel processing system or a distributed processing system comprising multiple processing units, and the method further includes: Receives a training dataset that includes multiple training data items; Generate multiple batches of training data items, each batch comprising a subset of the training data items in the training dataset; Distribute the multiple batches of training data items to the multiple processing units; and The neural network is trained in parallel using the multiple processing units based on multiple batches of training data items distributed.

19. The method according to claim 18, wherein, The parallel processing system or distributed processing system includes one or more tensor processing units or one or more graphics processing units.

20. A system for training a neural network, comprising one or more computers and one or more storage devices storing instructions, said instructions, when executed by said one or more computers, causing said one or more computers to perform operations of a corresponding method according to any one of claims 1-19.

21. The system according to claim 20, wherein, The system is a parallel processing system or a distributed processing system.

22. The system according to claim 21, wherein, The system includes one or more tensor processing units or one or more graphics processing units.

23. One or more computer storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method according to any one of claims 1-19.

Images