Layer normalization for calibrated uncertainty in deep learning

Abstract

Layer normalization in a machine learning application, including sampling a set of random activations corresponding to a fixed score of overall activations to provide a plurality of sub-sampled activations; calculating an average value of activation across the plurality of sub-samples; and calculating a standard deviation activated across the plurality of sub-samples. Layer normalization further includes employing two statistical data including an average value of sub-sample activations and a standard deviation across a plurality of sub-sample activations to normalize all activations as layer normalization.

Description

Background Technology

[0001] This invention relates generally to artificial intelligence, and more specifically to layer normalization in deep learning applications.

[0002] Enabling neural networks to efficiently estimate the uncertainty of their predictions is a problem that is gaining increasing attention as these models are deployed in critical decision-making settings. In real-world applications such as autonomous driving, robotics, or medical diagnostic models, they often operate outside the distribution compared to the training data. In such cases, calibrated prediction uncertainty is crucial for meaningfully comparing competing predictions and deciding when to trust them or ignore them if deemed unreliable.

[0003] Bayesian methods, such as Bayesian deep neural networks (DNNs), provide a principle for computing uncertainty measurements. Their drawback is that obtaining uncertainty measurements on complex models (e.g., large neural networks) quickly becomes tractable due to the computational challenges of posterior estimation and updating their parameters. To overcome these problems, proposed approximate Bayesian methods have been attempted, utilizing variational approximations, thus conveniently leveraging implementations of sampling mechanisms inherent in modern DNNs, such as loss and batch normalization. MC loss, for example, cleverly utilizes the fact that loss noise can be interpreted as a sampling mechanism on the variational distribution of the posterior of DNN parameters approximating given training data. Following this work, the stochastic nature of the mini-batch sampling process exhibited by batch normalization (Batch Norm) layers has also been used to perform approximate Bayesian inference for uncertainty estimation, as well as domain-adaptive prediction time calibration for uncertainty. Summary of the Invention

[0004] According to some embodiments of this disclosure, methods, systems, and computer program products for a computer-implemented method of layer normalization for calibration uncertainty in deep learning are provided.

[0005] Developing mechanisms for efficiently estimating the uncertainty of predictions made by neural networks is a challenge. Accurate uncertainty estimation is advantageous for real-world applications where there may be outward shifts in distribution compared to the training data. In these scenarios, calibrated prediction uncertainty plays a crucial role in determining when to trust the model's outputs or discard them as untrustworthy.

[0006] This disclosure describes a deep learning module that can be used as an insertion replacement for layer normalization blocks to seamlessly endow neural networks with uncertainty estimation capabilities. The deep learning module described herein, also known as Monte Carlo (MC) layer normalization, provides complementary applicability to related modules such as MC dropout and MC batch normalization. In some embodiments, the deep learning module (MC layer normalization) described herein can be used when the latter two modules (e.g., MC dropout and MC batch normalization) are technically unapplicable or not optimal. Although MC layer normalization is easy to deploy and has no significant computational overhead during training, it is stimulated from an approximate Bayesian perspective.

[0007] The methods, systems, and computer program products described herein can be empirically demonstrated to be competitive with existing alternatives in terms of prediction accuracy and uncertainty calibration across a variety of settings, including established benchmarks for outward-distributed image classification. Furthermore, the model possesses zero-shot prediction-temporal adaptive capabilities and its applicability in situations where known alternatives are unsuitable due to architectural incompatibility.

[0008] MC layer normalization can be used in neural networks that provide artificial intelligence in applications such as autonomous driving, object detection, and decision support.

[0009] In one aspect, a computer-implemented method for layer normalization in machine learning applications is provided. In some embodiments, the computer-implemented method enables deep neural networks to perform reliable predictions with calibrated uncertainty by using a Monte Carlo normalization module. Monte Carlo layer normalization is a normalization layer used when training a neural network. Monte Carlo layer normalization is used to normalize the activations of its upstream layers. Activation functions are typically applied to the output of each neuron in the network. It receives a weighted input and produces an output that is subsequently passed to the next layer.

[0010] In one embodiment, a computer-implemented method for layer normalization in a machine learning application is provided, comprising: sampling a random set of activations corresponding to a fixed score of overall activations to provide multiple subsampled activations; and calculating an average across the multiple subsampled activations. The computer-implemented method may further include calculating a standard deviation across the multiple subsampled activations. The computer-implemented method may then employ two statistics, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations, to normalize all activations as layer normalization.

[0011] In another embodiment, a computer-implemented method for layer normalization in machine learning applications is provided, comprising a training sequence including: sampling a random set of activations corresponding to a fixed score of overall activations to provide a plurality of subsampled activations; and calculating an average of the plurality of subsampled activations. The training sequence of the computer-implemented method may further include calculating a standard deviation across the plurality of subsampled activations. The training sequence of the computer-implemented method may then employ two statistics, including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations, to normalize all activations as a layer normalization. Following the training sequence, the computer-implemented method may perform an evaluation sequence employing a Monte Carlo approximation, which includes using subsampled input features for calculating the normalization statistics, and utilizing the layer normalization from the training sequence to calculate an average that allows the network to run forward multiple times, wherein the average is the output providing predictions.

[0012] In another embodiment, a computer-implemented method for layer normalization in machine learning applications is provided, comprising a training sequence including: sampling a random set of activations corresponding to a fixed portion of the overall activations to provide a plurality of subsampled activations; and calculating an average of the plurality of subsampled activations. The training sequence of the computer-implemented method may further include calculating a standard deviation across the plurality of subsampled activations. The training sequence of the computer-implemented method may then employ two statistics, including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations, to normalize all activations as a layer normalization. Following the training sequence, the computer-implemented method may perform an evaluation sequence employing a Monte Carlo approximation, which includes calculating a normalized static using subsampled input features, and calculating an average that allows the network to run forward multiple times using the layer normalization from the training sequence, wherein the average is the output providing the prediction.

[0013] In another embodiment, a computer-implemented method for layer normalization in machine learning applications is provided, comprising a training sequence that includes sampling a random set of activations corresponding to a fixed score of overall activations to provide multiple subsampled activations; and calculating an average across the multiple subsampled activations. The training sequence of the computer-implemented method may further include calculating a standard deviation across the multiple subsampled activations. The training sequence of the computer-implemented method may then employ two statistics, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations, to normalize all activations to a Monte Carlo layer normalization. Following the training sequence, the computer-implemented method may perform an evaluation sequence, including a sampling process that transforms the Monte Carlo layer normalization into the training sequence, and providing predictions using a Laplacian approximation without subsampling the input features.

[0014] On the other hand, a system for layer normalization in machine learning applications is described, including a hardware processor and a memory storing a computer program product. In one example, the computer program product of the system includes instructions to sample a set of random activations corresponding to a fixed score of overall activations using the hardware processor to provide multiple subsampled activations; and to compute an average across the multiple subsampled activations using the hardware processor. The computer program product also includes using the hardware process to compute the standard deviation of the activations across the multiple subsamples. The computer program product of the system can then use the hardware processor to normalize all activations as layer normalization using two statistics, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations.

[0015] In another aspect, this disclosure describes a computer program product for layer normalization in machine learning applications. The computer program product may include a computer-readable storage medium having computer-readable program code embodied therein.

[0016] The program instructions are processor-executable to cause the processor to sample a random set of activations corresponding to a fixed score of the overall activations to provide multiple subsampled activations; and to use the hardware processor to calculate an average across the multiple subsampled activations. The computer program product also includes instructions for using the hardware process to calculate a standard deviation across the multiple subsampled activations. The computer program product of the system can then use the hardware processor to normalize all activations as a layer normalization using two statistical data points, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations. Attached Figure Description

[0017] The following description will provide details of preferred embodiments with reference to the following figures, in which:

[0018] Figure 1 This is a flowchart / block diagram depicting a method for Monte Carlo (MC) layer normalization of calibrated uncertainty in a deep learning method according to an embodiment of the present disclosure.

[0019] Figure 2 This is a flowchart / block diagram depicting a method for normalizing calibrated uncertainty in a Monte Carlo (MC) layer in a deep learning method according to another embodiment of the present disclosure.

[0020] Figure 3 This is a diagram illustrating an example environment for a neural network used in an artificial intelligence model.

[0021] Figure 4This is a graph showing a comparison of LayerNorm, BatchNorm, prediction time BatchNorm, and MC-LayerNorm with expected calibration error (ECE), Brier score, and accuracy from CIFAR-10-C.

[0022] Figure 5 This is a graph from TinyImageNet-C, representing the first dataset used for calibration against inward distribution (test), outward distribution (OOD), and real-world scenes (mixed).

[0023] Figure 6 This is a graph from the second dataset of CIFAR-10-C, used for calibration for inward distribution (test), outward distribution (OOD), and real-world scenes (mixed).

[0024] Figure 7 This is a graph from TinyImageNet-C, representing the second dataset used for calibration against inward distribution (test), outward distribution (OOD), and real-world scenes (mixed).

[0025] Figure 8 A graph depicting the results on the Criteo dataset shows MC-LayerNorm when it reaches all three considered metrics: area under the curve (AUC), expected calibration error (ECE), and Brier score.

[0026] Figure 9 This is a block diagram of a system for machine learning according to an embodiment of the present disclosure, which employs Monte Carlo (MC) layer normalization in a deep learning method to calibrate uncertainties.

[0027] Figure 10 A computing environment according to embodiments of the present disclosure is described. Detailed Implementation

[0028] The methods, systems, and computer program products described in this disclosure can provide layer normalization for calibration uncertainties in deep learning methods. Training deep neural networks is a difficult task involving several problems to be solved. Despite their enormous potential, they can be slow and prone to overfitting. Batch normalization and layer normalization are two strategies for training neural networks faster without having to pay excessive attention to initialization and other regularization techniques. In layer normalization, all neurons in a particular layer have virtually the same distribution across all features of a given input. Normalizing all features except for each input in a particular layer eliminates the dependency on batches. This makes layer normalization well-suited for sequence models such as transformers and recurrent neural networks (RNNs). Some features of layer normalization include that it independently normalizes each input in a batch across all features. Since batch normalization depends on batch size, it is ineffective for small batch sizes. Layer normalization is independent of batch size, so it can also be applied to batches with smaller sizes. Furthermore, batch normalization requires different processing during training and inference time. Since layer normalization is performed along the length of the input to a particular layer, the same set of operations can be used at both training and inference times.

[0029] In some embodiments, the methods, systems, and computer program products described herein can provide deep learning modules that cleverly insert themselves into the research and development line of sampling-based deep learning layers for estimating the uncertainty of predictions on neural network models. In particular, the layer normalization provided herein, which may be referred to as Monte Carlo (MC) layer normalization, can be used as an insertion replacement of layer normalization blocks to seamlessly impart uncertainty calibration capabilities to neural networks.

[0030] As further described below, layer normalization, known as Monte Carlo (MC) layer normalization (MC-LayerNorm), includes randomized variants of layer normalization (LayerNorm), which replace related modules and provide supplementary applicability to related modules, such as MC dropout and MC BatchNorm.

[0031] Monte Carlo loss (MC loss) approximates Bayesian inference in neural networks to use loss for uncertainty estimation. During training time, loss sets a portion of the input features to zero, while during testing, no input features are lost. MC loss introduces randomness at inference time by using the training behavior at test time. Inference time is how long the forward propagation takes. To get the frames per second, we divide by 1 / inference time. In deep learning, inference time is the amount of time a machine learning model spends processing new data and making predictions. Turning to test time, the purpose of testing is to compare the output from the neural network with the target in an independent set (test instances).

[0032] If all test metrics are deemed acceptable, the neural network can move to the so-called deployment phase, also known as prediction.

[0033] Monte Carlo Batch Normalization (MC BatchNorm) introduces an alternative based on the properties of batch normalization statistics that utilize batch statistics during training time. Batch normalization is a technique used to train very deep neural networks, which normalizes the inputs of a layer for each mini-batch. In the formula used for batch normalization, given a batch of activations for a particular layer, it first calculates the mean and standard deviation of that batch. Then, it subtracts the mean and divides by the standard deviation to normalize these values. ε (epsilon) is added to the standard deviation of numerical stability.

[0034] When performing Monte Carlo Batch Normalization (MC BatchNorm), instead of utilizing the batch statistics run at inference time, the MC BatchNorm method approximates Bayesian inference by running multiple forward passes with different sets of training-time normalized statistics. Prediction-time batch normalization acts as a countermeasure to covariate shifts associated with outward-distributed data samples. This method works by discarding the running batch statistics of the batch normalization layer instead of using the batch statistics of each individual batch at test time. This effectively counteracts the covariate shift and significantly improves model calibration.

[0035] The layer normalization employed in methods, systems, and computer program products, known as Monte Carlo (MC) layer normalization (MC-LayerNorm), can be used for training setups and architectures where the latter two modules, Monte Carlo dropout (MC-dropout) and Monte Carlo batch normalization (MC-BatchNorm), are technically unsuitable or lead to suboptimal performance. Furthermore, MC-LayerNorm inherits the advantages of layer normalization over batch normalization, including the fact that its training behavior is independent of mini-batch size and invariant to rescaling of individual input data. This last characteristic has been identified as particularly attractive because it endows the layer normalization disclosed herein, MC-LayerNorm, with the ability to perform zero-shot domain adaptation at prediction time.

[0036] MC-LayerNorm can be driven from an approximate Bayesian perspective, while being easy to deploy in practice and having no significant computational overhead during training. In various experimental settings, including established benchmarks for outward distribution (OOD) image classification, the layer normalization provided in this disclosure, MCLayerNorm, outperforms existing alternatives such as Monte Carlo Loss (MC-Loss) and Monte Carlo Batch Normalization (MC BatchNorm) in terms of prediction accuracy and uncertainty calibration.

[0037] The layer normalization method provided in this disclosure, MCLayerNorm, is suitable for CTR prediction on the Criteo dataset, where known competing methods such as Monte Carlo dropout (MC-dropout) and Monte Carlo batch normalization (MCBatchNorm) cannot be used because they are incompatible with the relevant architecture. For example, current prior art models do not include dropout or batch normalization layers in the case of CTR prediction on the Criteo dataset, which excludes the use of MC dropout or MC BatchNorm.

[0038] Now for reference Figures 1-10 The methods, systems, and computer program products of this disclosure are described in more detail.

[0039] Figure 1 This is a flowchart / block diagram depicting Monte Carlo (MC) layer normalization for calibrated uncertainty in a deep learning method according to an embodiment of the present disclosure.

[0040] The normalization module described herein and referred to as MC layer normalization is similar in some respects to Monte Carlo batch normalization because MC layer normalization allows for uncertainty estimation. In some examples, MC layer normalization allows for uncertainty estimation, specifically through Monte Carlo sampling on random sources. In some embodiments, instead of sampling originating from the randomness of mini-batches in Monte Carlo batch normalization (MC BatchNorm), MC layer normalization is a variant of layer normalization where randomness is injected by subsampling the features when normalization is computed, and the normalization is a statistical method used to normalize the feature vector. References Figure 1 Block 1, the learning phase of the MC layer normalized neural network may include random activation sets corresponding to a fixed score of overall activation. Perform sampling, such as pre-activation.

[0041] In some embodiments, MC layer normalization can begin with a normalization layer, where the first layer in the feedforward neural network is considered. l There are hidden layers, and it is assumed that... a l This is a vector representation of the summation input (preactivation) of neurons in this layer.

[0042] Activation functions play a crucial role in neural networks, performing important functions in hidden layers to solve complex problems and throughout the analysis and transmission of data in deep learning algorithms. Many activation functions exist, including binary, linear, and numerous non-linear variants. An activation function defines the output of a node based on a specific set of inputs in machine learning, deep neural networks, and artificial neural networks. The activation functions in artificial neural networks are analogous to cells and neurons in the human brain. Pre-activation is the value calculated before the activation function is applied, while post-activation is the value calculated after the activation output of the neuron is applied.

[0043] Pre-activation via weighted matrix and the opposite layer The input matrix-vector multiplication is computed as: Equation (1) , in It is an element-wise activation function such as ReLU, and is Bias parameters.

[0044] Note that neural networks and feedforward computation will refer to Figure 3 To explain further, an artificial neural network (ANN) is an information processing system inspired by a biological nervous system, such as the brain. An element of an ANN is the structure of this information processing system, which comprises a large number of highly interconnected processing elements (called "neurons") working in parallel to solve a specific problem. Furthermore, the ANN is trained using a set of training data, where learning involves adjusting the weights that exist between the neurons. Through this learning process, the ANN is configured for specific applications, such as pattern recognition or data classification.

[0045] Now for reference Figure 2 The diagram illustrates a generalized representation of a neural network. While a specific structure of an ANN with three layers and a set number of fully connected neurons is shown, it should be understood that this is for illustrative purposes only. In practice, this embodiment can take any suitable form, including any number of layers and any pattern of one or more connections between them.

[0046] ANNs have demonstrated the ability to derive meaning from complex or imprecise data and can be used to extract patterns and detect trends that are too complex to be detected by humans or other computer-based systems. The structure of a neural network is typically known to have input neurons 102 that provide information to one or more "hidden" neurons 104. Connections 108 between the input neurons 102 and the hidden neurons 104 are weighted, and these weighted inputs are then processed by the hidden neurons 104 according to a function within the hidden neurons 104. Layers of any number of hidden neurons 104, and neurons performing different functions, can exist. Different neural network architectures, such as convolutional neural networks, Maxout networks, etc., exist, which can vary depending on the structure and function of the hidden layers and the weight patterns between layers. Each layer can perform a specific function and can include convolutional layers, pooling layers, fully connected layers, softmax layers, or any other suitable type of neural network layer. Finally, a set of output neurons 106 receives and processes the weighted inputs from the last set of hidden neurons 104.

[0047] This represents a "feedforward" computation, where information propagates from input neuron 102 to output neuron 106. Upon completion of the feedforward computation, the output is compared to the expected output available from the training data. Then, in the "backpropagation" computation, the error relative to the training data is processed, where hidden neuron 104 and input neuron 102 receive information about the error propagated back from output neuron 106. Once backpropagation is complete, a weighted update is performed, where weighted connections 108 are updated to address the received error. It should be noted that these three operational modes—feedforward, backpropagation, and weight update—do not overlap. This represents only one type of ANN computation and can be replaced by any suitable form of computation.

[0048] To train an ANN, the training data can be divided into a training set and a test set. The training data consists of pairs of inputs and known outputs. During training, the inputs from the training set are fed into the ANN using feedforward propagation. After each input, the ANN's output is compared with the corresponding known output. The difference between the ANN's output and the known output associated with that particular input is used to generate an error value, which is then backpropagated through the ANN, allowing the ANN's weights to be updated. This process continues until all pairs in the training set are exhausted.

[0049] After training is complete, the ANN can be tested against a test set to ensure that training has not led to overfitting. If the ANN can generalize to new inputs beyond those it has been trained on, it is ready for use. If the ANN does not accurately reproduce the known outputs of the test set, additional training data may be needed, or the ANN's hyperparameters may need to be tuned.

[0050] An ANN can be implemented in software, hardware, or a combination of both. For example, each weight 108 can be characterized as a weight value stored in computer memory, and the activation function for each neuron can be implemented by a computer processor. The weight values ​​can store any suitable data value, such as a real number, a binary value, or a value selected from a fixed number of possibilities, which is multiplied by the associated neuron output. Alternatively, the weights 108 can be implemented as a resistive processing unit (RPU) that generates a predictable current output when an input voltage is applied according to a settable resistor.

[0051] Returning to the reference layer normalization, note that layer normalization is proposed as a method to mitigate covariate bias in correlated inputs, as an alternative to batch normalization, and it lies in using the following calculations... and pre-activation variance Normalize the hidden units in a given layer (layer normalization) : Equation (2): , in It is a layer l The number of units in the array.

[0052] Return to reference Figure 1 Block 2 of this method may include calculating the average value of activations across multiple subsamples. Refer to equation (2). It is the average value of multiple subsample activations according to an embodiment of this disclosure.

[0053] Figure 1 Block 3 of the computer implementation shown can continue to calculate the standard deviation across multiple subsampling activations. Refer to equation (2). It is the average value of activations across multiple subsamples according to an embodiment of this disclosure.

[0054] In some embodiments, one advantage of layer normalization over batch normalization is that it does not rely on any assumptions about the size of the mini-batch and can therefore be used when the batch size is 1.

[0055] In some embodiments, the MC-LayerNorm layer includes modifying equation (2) by running an average on only a random subset of the cells. In some embodiments, this is achieved by applying a fixed score to the pre-activated cells. f Sampling (or equivalently, by using 1- f The rate of decrease of the pre-activated fixed fraction f ) obtained To set the definition, and calculate the normalized statistics on the sampling unit as follows: Equation (3): , in Indicates the size of the set.

[0056] refer to Figure 1 The computer-implemented method can continue to block 4. Figure 1 Block 4 of the computer-implemented method shown may include using two statistics, such as the pre-activated average across multiple subsamples. (mean) and standard deviation, to normalize all activations as layer normalization.

[0057] As a result, the normalized preactivation and layer calculated by MC-LayerNorm l The output is:

[0058] Equation (4): , ,

[0059] In some embodiments, and It is a random variable, and its implementation is determined by random sampling index. The subset is determined, and and output This is also true. This observation has inspired a probabilistic viewpoint on the structure of MC-LayerNorm.

[0060] This represents one instance of the training set.

[0061] The probabilistic view of MC-LayerNorm can assume, without loss of generality, that for the training dataset In a supervised learning setup, the objective of supervised learning can be formulated as training parameters. (It includes each floor) l weight and deviation To maximize the dataset under the predicted probability D Possibility: Equation (5): ,

[0062] in Parameterized as a neural network. Variables It refers to a network with MC-LayerNorm during subsampling of a neural network. Variables x y represents the sample features, and y represents the sample label. The network is assigned an MC-LayerNorm layer. It can be modeled as a network distribution ,in Is in all layers lThe set of implementations of the random statistical data in equation (3) in [the context]. In some embodiments, an equivalent way to see this is to consider [the context] for a given set. sampling network As the corresponding network ,in Now from the approximately defined distribution Sampling.

[0063] In some embodiments, distribution In a specific sense, towards the parameters ( The Gaussian distribution of the variable converges. These are the parameters of the neural network.

[0064] In some embodiments, this method can approximate the normality of MC-LayerNorm networks. For example, for Theorem 1, given a network with MC-LayerNorm layers... The corresponding network All C-LayerNorms (Equation (3)) are replaced by the regular LayerNorm (Equation (2)). The LayerNorm-induced model is replaced by MC-LayerNorm. exist Its distribution. Furthermore. exist parameters The neighborhood is asymptotically normal. (Variable) It is a network with LayerNorm during subsampling. Variables It has a distribution Sampling parameters ( ) parameterized neural network

[0065] The proof of the above MC layer normalization theorem is as follows (which can be called Lemma A.1): express ,and . Assumption →∞, when →∞. Assumption There are limits. To provide →∞, → . Assumption There are limits. To provide →∞, → .

[0066] This proof is based on standard regulations and rules.

[0067] Another proposal for the proof of the above theorem used for MC layer normalization is as follows (and may be referred to as Proposition A.2): Assume that in equations (1) and (4) f If ( ) is basically differentiable everywhere, then Lemma A.1 implies the existence of variance. V , making

[0068] → .

[0069] The proof follows the Delta method.

[0070] The proof in Proposition A.2 implies that the effect of MC layer normalization is to approximately add Gaussian noise to the activation of the corresponding model with layer normalization.

[0071] This method also includes approximate Bayesian inference using MC-LayerNorm. Theorem 1 above allows for the use of methods employing stochastic tuning techniques in neural networks. They perform actual approximate inferences in the space because they introduce random variables that allow sampling on the network. More specifically, this can begin with computing the predictive distribution ( (Given a new input) x and training dataset D In cases where a Bayesian model is used for averaging, but q is approximated using easily tractable variational approximations. (θ) Replace the difficult-to-handle posterior p(θ|D) on the parameter, then marginalize it on θ using Monte Carlo integration: Equation (6) ,in

[0072] refer to Figure 1 In some embodiments, after training and during evaluation, the computer-implemented method may continue to block 5, employing a sampling process that converts the training sequence into a currency calo layer normalization, as described above in reference equation (6).

[0073] In fact, in the mini-batch stochastic graded descent (SGD) training setup, for each training point and each MC-LayerNorm layer, when computing the stochastic graded descent (SGD) steps, a subset... l Sampling. This allows for orientation towards q. ( ) Optimize qθ ( The process of prediction. At the prediction time, equation (6) represents the process of prediction based on given input data. x In the case of making the network move forward N The Monte Carlo integral is implemented 1 time, each time with a subset. l Different implementations.

[0074] refer to Figure 1 Block 6 may include calculating the average value of multiple forward runs of the network using layer normalization from the training sequences, where this average value is the output providing predictions. These predictions can be used for machine learning applications in autonomous vehicles, object identification, and decision support. (Reference) Figure 1 The described processing flow employs a Monte Carlo approximation mode during the prediction time. In the Monte Carlo approximation mode, subsampled input features provide the calculation of normalized statistics. A Bayesian perspective is also employed in the Monte Carlo approximation mode. For the Bayesian perspective, the process is as follows: N The process involves several Monte Carlo iterations, with the average value used for prediction.

[0075] However, for networks with MC-LayerNorm, there is an additional approximation to the inference process that allows predictions to be computed more efficiently by utilizing the second part of Theorem 1.

[0076] Figure 2 This is a flowchart / block diagram depicting a method for normalizing calibrated uncertainty in a Monte Carlo (MC) layer in a deep learning method according to another embodiment of the present disclosure. Figure 2 An approximation pattern for prediction time using a trained neural network is described. In one embodiment of the single-approximation pattern, the sampling process is converted to MC integral and connected to a Bayesian model average. Furthermore, a single-approximation pattern can be achieved using the Laplace approximation, as shown in equation (7) below, where there is no subsampling of the input features. Figure 2 The process flow shown is consistent with Figure 1 The process flow shown is similar; for example, refer to Figure 1 Blocks 1, 2, 3, and 4 mentioned above are in relation to the reference. Figure 2 Blocks 1, 2, 3, and 4 are similar, and the above are... Figure 1 The descriptions of blocks 1, 2, 3, and 4 provided in the process flow diagram are suitable for providing a description Figure 2 Examples of blocks 1, 2, 3 and 4 in the example.

[0077] In some embodiments, according to the theorem, the distribution ˆθ~qθ(ˆθ) is approximately normally distributed around the parameter θ of the corresponding network with layer normalization, and we can approximate the Monte Carlo integral in equation (6) with the following equation: Equation (7): ,in = ,

[0078] For models with regular LayerNorm instead of MC-LayerNorm Based on θ The asymptotic normality is obtained using equation (5) and the Laplace approximation.

[0079] refer to Figure 2 Box 7 includes a sampling process that translates into Monte Carlo layer normalization of the training sequences. Block 8 includes employing a Laplacian approximation to provide predictions without subsampling the input features.

[0080] This shows that at prediction time, the method can be easily run by replacing the MC-LayerNorm with the regular LayerNorm as a cheap further approximation of the MC integral in equation (6). We will refer to this mode of using the MC-LayerNorm at prediction time as the LayerNorm approximation.

[0081] use Figure 2 The predictions provided by the process described can be used in machine learning applications for autonomous vehicles, object identification, and decision support.

[0082] Previous results are summarized in the pseudocode snippet, and the practical application of MC-LayerNorm for training neural networks with SGD using backpropagation algorithm 1 is described in detail below:

[0083] Algorithm 1: MC-LayerNorm module (training mode) Input: Input vector Input: (parameter) The fraction of the sampled cells f sample: Random selection calculate: calculate: Output: ,in

[0084] The pseudocode snippet summarizes the previous results and details the use of MC-LayerNorm and Algorithm 2 in the prediction time, as follows: Algorithm 2: MC-LayerNorm module (evaluation mode) Input: Input vector Input: Number of MC samples If mc_integration, then / / execute MC integration: for { / / Run Algorithm 1} End for Output: Else / / LayerNorm approximation: calculate: calculate: Output: ,in End if

[0085] In the case of convolutional input, MC-LayerNorm normalizes its input in both the channel and spatial dimensions. Experimental results

[0086] MC-LayerNorm was evaluated to assess the quality of predictions (using standard metrics including accuracy). MC-LayerNorm was also evaluated to assess the calibration of the prediction distribution. The aim of the experiments was to determine whether the MC-LayerNorm layers truly facilitated the model's appropriate representation of uncertainty. Specifically, when the training and test distributions were not simultaneous due to covariant shifts, i.e., when the marginal distributions of features differed, the experiments focused on the outward distribution (OOD) setting, but preserved the conditional label distribution. It was then expected that the predictions would reflect an appropriately calibrated uncertainty that tended to be less confident in OOD inputs that were not well represented in the training distribution. Two measures were used to quantify the calibration of the prediction uncertainty: Expected Calibration Error (ECE) and the Brier score. Expected calibration error (ECE)

[0087] ECE quantifies the difference between the confidence score of the model and its accuracy calculated on intervals of samples sorted by confidence score. In ECE, a lower value is better. Specifically, N predictions are grouped into intervals based on confidence score. In the middle, calculate the average forecast within the interval. Then calculate ECE = As an artifact resulting from confidence binning, ECE can overemphasize the tail of the probability. Brier rating

[0088] The Brier score is the squared distance between the vector of predicted probabilities and the one-hot encoded true label. The Brier score monotonically decreases to zero as the predicted probabilities approach the true target distribution. We experiment with two types of application settings: Setting 1: MC-LayerNorm serves as a viable alternative to existing methods, such as for real-world scenarios with sparse occurrences of outward-distributed samples (e.g., image classification). Setting 2: MC-LayerNorm is used as an option for accurate uncertainty estimation in applications where competing methods cannot be used because they are incompatible with the architecture (e.g., models that do not incorporate BatchNorm and / or Dropout due to architectural reasons or training architecture constraints). Image classification

[0089] Based on experience, MC layer normalization was compared with prediction time batch normalization. Model calibration metrics (accuracy, ECE, and Brier score) were evaluated on CIFAR-10-C and TinyImageNet. Both datasets were corrupted versions of their original test sets generated by applying one of 15 corruptions (e.g., frost, motion blur, rain). The evaluation focused on the ConvNext architecture, which has been shown to achieve object classification performance with BatchNorm and LayerNorm normalization. For CIFAR-10-C, the analysis was limited to the pre-trained architecture size "pico," while for TinyImageNet-C, a pre-trained "tiny" variant was used. No data augmentation was applied during training except for horizontal image flipping.

[0090] For each hyperparameter configuration, train three randomly initialized models as follows: Hyperparameter adjustment range

[0091] All models were trained using the default parameters β1, β2, and ∈, with the AdamW optimizer provided by the Timm library (Wightman et al., 2023). A grid search was used to adjust the learning rate from values ​​between [e-3, 0.5e-4], and for weight decay values ​​between [0.1, 1e-4], fine-tuning was run over 50 epochs with a batch size of 512 for both CIFAR-10 and TinyImageNet. Learning rate scheduling began at […]. The learning rate was set to 0 and increased linearly for one warm-up epoch. Then, the learning rate was decreased by 0.1 for epochs [20, 35, 45]. For Criteo, training was run for 100 epochs with a batch size of 1000, and the learning rate was decreased by 0.1 on a patience 2 platform checked on the validation set. All models were trained on an internal cluster consisting of NVIDIA A100 GPUs.

[0092] After training the initialized model, a single model is selected for each normalization method based on the accuracy measured on the validation set. Finally, model calibration is performed for three different cases: 1. Within the distribution (test): The original test set of the corresponding dataset is used as the baseline for all metrics. 2. Outward Distribution (OOD): Evaluation metric for the corrupted C-variant test set (severity 5). This is beneficial for prediction time BatchNorm because batches consist only of outwardly distributed samples. 128 batches are used. 3. Real-world use case (hybrid): This scenario simulates a zero-prediction-temporal adaptive setting where the deployed model first encounters an OOD sample and does not collect enough observations to re-adapt (because prediction-time BatchNorm requires multiple OOD samples). Such a scenario is simulated by creating batches of size N consisting of N-1 distributed samples and a single outward distributed sample. The batch size is the same as the OOD setting and equals N = 128.

[0093] Then, the calibration metric is measured only for samples distributed outwards.

[0094] Figure 4 and Figure 5 The results for CIFAR-10-C and TinyImageNet-C are shown. The graph identified by figure 25 is from the dataset normalized by the BatchNorm layer. The graph identified by figure 26 is from the dataset normalized by the LayerNorm layer. The graph identified by figure 27 is the prediction-BatchNorm layer normalized graph. The graph identified by figure 28 is the Monte Carlo layer normalized graph.

[0095] Figure 4 This is a graph of data from CIFAR-10-C for the distribution (test), outward distribution (ODD), and real-world scenarios (mixture). Figure 4 This section compares LayerNorm, BatchNorm, PredictionTimeBatchNorm, and MC-LayerNorm for Expected Calibration Error (ECE), Brier Score, and Accuracy. While PredictionTimeBatchNorm is superior in the Fully Outward Distribution (OOD) setting, MC-LayerNorm (f=0.8) outperforms all other methods in real-world use cases (mixed). Error bars are constructed from evaluations of 15 corrupted samples applied to the original test set.

[0096] Figure 5 This is a graph from TinyImageNet-C data, used for calibration against in-distribution (test), out-of-distribution (OOD), and real-world scenes (mixed). Figure 5 This section compares LayerNorm, BatchNorm, prediction-time BatchNorm, and our novel method MC-LayerNorm (f=0.9) for expected calibration error (ECE), Brier score, and accuracy. As demonstrated in the CIFAR-10-C experiments, LayerNorm is a superior method for real-world use cases (mixed) despite not exceeding the prediction-time batch standard in out-of-distribution settings. Error bars are constructed from evaluation runs applied to 15 corrupted samples on the original test set.

[0097] Although MC-LayerNorm also achieves better calibration than prediction-time BatchNorm in OOD settings, it outperforms all other normalization methods in real-world use cases (hybrid).

[0098] Prediction timing BatchNorm sacrifices much of its performance in real-world (mixed) use cases because it heavily relies on batches consisting only of outward-distributed samples. In contrast, LayerNorm methods do not rely on batch statistics because they compute normalized statistics across channels and spatial dimensions. Therefore, the calibration capability of the MC-LayerNorm method is independent of batch size and applies not only to small batches but also to mixed inward and outward-distributed batches.

[0099] Figure 2 The TinyImageNet-C experiments shown in the figure demonstrate the significant advantage of MC-LayerNorm in ECE. The base LayerNorm performs comparably to MC-LayerNorm in Brier scores, but outperforms the base version in ECE. Since ECE is independent of model accuracy, MC-LayerNorm shows a clear benefit for model calibration.

[0100] Although the purpose of the experiment was to evaluate model calibration under distribution shift, these results also indicate that MCLayerNorm slightly improves the calibration of in-distribution samples.

[0101] Figure 6 and Figure 7 This illustrates a comparison of calibration performance for different components of the sampling features of MCLayerNorm. The graph identified by label 25 is from the dataset normalized by the BatchNorm layer. The graph identified by label 26 is from the dataset normalized by the LayerNorm layer. The graph identified by label 27 is from the prediction-BatchNorm layer normalization. The graph identified by label 28 is from the Monte Carlo layer normalization.

[0102] Figure 6This is a graph from CIFAR-10-C data used for calibration of inward distribution (test), outward distribution (OOD), and real-world scenes (mixed).

[0103] Figure 7 This is a graph from TinyImageNet-C data, used for calibration against in-distribution (test), out-of-distribution (OOD), and real-world scenes (mixed).

[0104] The results show that MC-LayerNorm is stable only in terms of its hyperparameter, namely the score of the subsampled features. Even for a configuration where normalization is computed only from 60% of the input features, the model performance and calibration remain stable.

[0105] Criteo Display Ad Challenge

[0106] Here, the experiment demonstrates the value of MC-LayerNorm when competing methods are unusable due to architectural incompatibility. For the experiment, the model was trained on the Criteo Display Advertising Challenge (CriteoLabs, 2014). This experiment influences the current state-of-the-art MaskNet model (Wang et al., 2021), relying on an implementation from (Zhu, 2023, Apache-2.0 license). Two types of models were trained: the original MaskNet model containing standard LayerNorms and a patched model where all LayerNorm layers were replaced by MC-LayerNorm layers.

[0107] At prediction time, the MC-LayerNorm models are run using the LayerNorm approximation method, making them as computationally cheap as regular MaskNet models for evaluation.

[0108] The trained model was evaluated on the original test set and a corrupted version thereof. We simulated outwardly distributed data by applying corruption in the form of Gaussian noise to the numerical features. Shift strengths [6.25%, 12.5%, 25%, 50%] were based on the standard deviation of the original eigenvalues ​​per unit. Finally, the performance difference between the original model treated with LayerNorm and the adjusted model treated with MC-LayerNorm was determined.

[0109] Figure 8 The results on the Criteo dataset are illustrated, showing the performance of MC-LayerNorm when all three considered metrics are employed. As the test group is further shifted outwards, the MC-LayerNorm method retains more of the original calibration and model performance compared to the baseline of traditional LayerNorm. Figure 8In the figure shown, the line indicated by reference numeral 30 represents data from layers normalized using LayerNorm. Figure 8 In the figure shown, the line marked by reference numeral 29 represents data from layers normalized by Monte Carlo layer normalization.

[0110] refer to Figure 8 The area under the curve (AUC), Brier score, and expected calibration error (ECE) for four shift intensities are shown to compare the performance of the standard LayerNorm versus the MC-LayerNorm when an increased amount of noise is applied to the test group. This experiment demonstrates the significant advantage of the MC-LayerNorm approach, preserving more of the original model performance compared to the classic LayerNorm, as the data are further outwardly distributed (the figure shows the average of 10 runs; the error bars indicate the standard error around the average).

[0111] This section describes a novel normalization layer, MC-LayerNorm, which improves model calibration for both inward and outward (OOD) data by injecting randomness through subsampling features when computing normalized statistics. Compared to prediction-time BatchNorm, MC-LayerNorm has the advantage of not requiring readjustment on OOD samples because it does not rely on batch-level statistics, as it computes statistics per channel rather than per batch. Therefore, the approach using MC-LayerNorm layers exhibits calibration capability independent of batch size, effective not only for small but also for mixed inward and outward distributions. It can even perform inference on a single sample at a time.

[0112] MC-LayerNorm is able to perform zero-prediction-time-domain adaptation, which offers advantages in real-world OOD use cases where the deployed model is first exposed to new OOD samples and there is no time to properly adapt to covariate shifts.

[0113] Figure 9 This is a block diagram of System 200 for machine learning, which uses Monte Carlo (MC) layer normalization in deep learning methods to calibrate uncertainties. Figure 9An embodiment of a system for layer normalization in machine learning applications is illustrated, the system including a hardware processor and a memory storing a computer program product. The computer program product of the system includes instructions that include sampling a random set of activations corresponding to a fixed score of overall activations using the hardware processor to provide multiple subsampled activations; calculating an average across the multiple subsampled activations using the hardware processor; and calculating a standard deviation across the multiple subsampled activations using a hardware process. The system can also use the hardware processor to employ two statistics, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations, to normalize all activations as layer normalization.

[0114] refer to Figure 9 Subset counter 201 collects a randomly activated set, as shown in the reference. Figure 1 and Figure 9 As described in block 1 of the above method, Monte Carlo layer normalization 202 provides as referenced. Figure 1 and Figure 2 The training steps described in blocks 1-4.

[0115] Still referencing Figure 9 System 200 also includes a Monte Carlo approximation for prediction engine 203. Further details of the Monte Carlo approximation for prediction engine 203 can be found in... Figure 1 As described in the descriptions of blocks 6 and 7, in the Monte Carlo approximation mode, subsampled input features provide the computation of normalized statistics. In the Monte Carlo approximation mode, a Bayesian perspective is employed. For the Bayesian perspective, N Monte Carlo iterations are run, and the average value is used for prediction.

[0116] refer to Figure 9 System 200 may also include a Laplace approximation without subsampling for prediction engine 204. This prediction engine can provide a single-approximation pattern. The sampling process is transformed into an MC-integral and connected to a Bayesian model average.

[0117] Using the Laplacian approximation, a single-shot approximation pattern without subsampling of input features can be provided. Further details regarding the approximate use of the Laplacian approximation without subsampling of the prediction engine 204 in a single-shot trigger can be found in the above section. Figure 2 It can be found in the descriptions of blocks 7 and 8.

[0118] refer to Figure 10 In some embodiments, components of system 200 for normalizing MC layer calibration uncertainties in deep learning communicate with at least one hardware processor (processor set 310), wherein the hardware processor can communicate with... Figure 9 The other components depicted work together to provide the aforementioned functionality. Figure 10Further shown is a processing system 300, which may include a reference Figures 1-9 The system 200 described is for MC layer normalization.

[0119] refer to Figure 10 The computing environment 300 includes examples of environments for executing at least some of the computer code involved in performing the methods of the present invention, such as a method for normalizing MC layers for calibration uncertainties in deep learning 200. In addition to block 200, the computing environment 300 includes, for example, a computer 301, a wide area network (WAN) 302, an end-user equipment (EUD) 303, a remote server 304, a public cloud 305, and a private cloud 306. In this embodiment, the computer 301 includes a processor set 310 (including processing circuitry 320 and a cache 321), a communication structure 311, volatile memory 312, persistent storage device 313 (including an operating system 322 and block 200, as described above), a peripheral device set 214 (including a user interface (UI), a device set 323, a storage device 324, and an Internet of Things (IoT) sensor set 325), and a network module 315. The remote server 304 includes a remote database 330. Public cloud 305 includes gateway 340, cloud orchestration module 341, host physical machine set 342, virtual machine set 343, and container set 344.

[0120] Computer 301 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future capable of running programs, accessing networks, or querying databases such as remote database 330. As is well known in the field of computer technology, and depending on the technology, the performance of a computer-implemented method can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 300, the detailed discussion focuses on a single computer, specifically computer 301, to keep the presentation as simple as possible.

[0121] Computer 301 can be located in the cloud, even if Figure 10 The document does not show that the computer is in the cloud; on the other hand, the computer 301 does not need to be in the cloud unless it can be definitively indicated to any extent.

[0122] Processor set 310 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 320 may be distributed across multiple packages, such as multiple cooperating integrated circuit chips. Processing circuitry 320 may implement multiple processor threads and / or multiple processor cores. Cache 321 is memory located within the processor chip package(s) and is typically used for data or code that should be readily accessible by the threads or cores running on processor set 310. Cache memory is typically organized into multiple levels based on its relative proximity to the processing circuitry. Alternatively, some or all of the caches in the processor set may be located “off-chip.” In some computing environments, processor set 310 may be designed to work with qubits and perform quantum computing.

[0123] Computer-readable program instructions are typically loaded onto computer 301 to cause processor set 310 of computer 301 to perform a series of operational steps to implement a computer-implemented method, such that the instructions thus executed instantiate the method specified in the flowchart and / or the narrative description of the computer-implemented method included in this disclosure (collectively, the "method of the invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 321 and other storage media discussed below. The program instructions and associated data are accessed by processor set 310 to control and direct the execution of the method of the invention. In computing environment 300, at least some of the instructions for performing the method of the invention may be stored in persistent storage device 313 in block 200.

[0124] Communication structure 311 is a signal transmission path that allows the various components of computer 301 to communicate with each other. Typically, this structure consists of switches and conductive paths, such as switches and conductive paths that form buses, bridges, physical input / output ports, etc. Other types of signal communication paths can be used, such as fiber optic communication paths and / or wireless communication paths.

[0125] Volatile memory 312 is any type of volatile memory now known or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, but this is not necessary unless explicitly stated otherwise. In computer 301, volatile memory 312 is located in a single package and inside computer 301; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located outside computer 301.

[0126] The persistent storage device 313 is any form of non-volatile memory for a computer, now known or to be developed in the future. The non-volatility of this memory means that the stored data is retained regardless of whether power is supplied to the computer 301 and / or directly to the persistent storage device 313. The persistent storage device 313 may be a read-only memory (ROM), but typically at least a portion of the persistent storage device allows for data writing, data deletion, and data rewriting. Some common forms of persistent storage include disks and solid-state storage devices. The operating system 322 may take several forms, such as various known proprietary operating systems or operating systems employing an open-source portable operating system interface type with a kernel. The code included in block 200 generally includes at least some of the computer code involved in performing the methods of the present invention.

[0127] Peripheral device set 314 includes the peripheral device set of computer 301. Data communication connections between peripheral devices and other components of computer 501 can be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, UI device set 323 may include components such as displays, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage device 324 is an external storage device, such as an external hard drive, or a pluggable memory, such as an SD card. Storage device 324 may be persistent and / or volatile. In some embodiments, storage device 324 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 301 requires substantial storage (e.g., where computer 301 locally stores and manages a large database), this storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 325 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, while another could be a motion detector.

[0128] Network module 315 is a collection of computer software, hardware, and firmware that allows computer 301 to communicate with other computers via WAN 302. Network module 315 may include hardware such as a modem or Wi-Fi transceiver, software for packetizing and / or depacketizing data transmitted over the communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control and network forwarding functions of network module 315 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of network module 315 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the methods of the present invention can typically be downloaded to computer 301 from an external computer or external storage device via a network adapter card or network interface included in network module 315. WAN 302 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances using any technology now known or to be developed in the future for transmitting computer data. In some embodiments, a WAN may be replaced by and / or supplemented by a local area network (LAN), which is designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0129] End User Equipment (EUD) 303 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 301) and can take any of the forms discussed above in conjunction with computer 301. EUD 303 typically receives helpful and useful data from the operation of computer 301. For example, assuming computer 301 is designed to provide recommendations to the end user, these recommendations are typically transmitted from network module 315 of computer 301 to EUD 303 via WAN 302. In this way, EUD 303 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 303 can be a client device, such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0130] Remote server 304 is any computer system that provides at least some data and / or functionality to computer 301. Remote server 304 can be controlled and used by the same entity operating computer 301. Remote server 304 represents multiple machines that collect and store helpful and useful data for use by other computers such as computer 301. For example, if computer 301 is designed and programmed to provide recommendations based on historical data, that historical data can be provided to computer 301 from a remote database 330 of remote server 304.

[0131] Public cloud 305 is any computer system that can be used by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities (particularly data storage (cloud storage) and computing power) without direct active management by users. Cloud computing typically leverages resource sharing to achieve scalability consistency and economy. Direct and active management of the computing resources of public cloud 305 is performed by the computer hardware and / or software of cloud orchestration module 341. The computing resources provided by public cloud 305 are typically implemented by virtual computing environments running on various computers constituting host physical set 342, which is the entire domain of physical computers in and / or available to the public cloud 305. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 343 and / or containers from container set 344. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the VCE is instantiated. Cloud orchestration module 341 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages the active instantiation of VCE deployments. Gateway 340 is a collection of computer software, hardware, and firmware that allows public cloud 305 to communicate via WAN 302.

[0132] Now, we will provide some further explanation of Virtualized Computing Environments (VCEs). A VCE can be stored as an "image." New active instances of a VCE can be instantiated from this image. Two common types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user-space instances, called containers, to exist. From the perspective of the programs running within them, these isolated user-space instances typically appear as actual computers. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container; this is a characteristic known as containerization.

[0133] Private cloud 306 is similar to public cloud 305, except that computing resources are available only to a single enterprise. While private cloud 306 is depicted as communicating with WAN 302, in other embodiments, private cloud may be completely disconnected from the Internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types) typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by normalization or proprietary technology that enables coordination, management, and / or data / application portability across the multiple component clouds. In this embodiment, public cloud 305 and private cloud 306 are both part of a larger hybrid cloud.

[0134] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. For example, in some embodiments, a computer program product for layer normalization in machine learning applications is provided. The computer program product may include a computer-readable storage medium having computer-readable program code embodied therein. The program instructions are processor-executable to cause the processor to use a hardware processor to sample a random set of activations corresponding to a fixed score of overall activations to provide multiple subsampled activations. The computer instructions may also utilize the hardware processor to calculate an average across the multiple subsampled activations; and to calculate a standard deviation across the multiple subsampled activations. Furthermore, the computer program product may employ two statistical data, including the average of the subsampled activations and the standard deviation across the multiple subsampled activations, to normalize all activations as layer normalization.

[0135] A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention. The computer program product may also be non-transitory.

[0136] Computer-readable storage media can be tangible devices capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0137] The computer-readable program instructions described in this disclosure can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0138] Computer-readable program instructions for performing the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0139] As used herein, the terms "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or a combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuitry, processing circuitry, instruction execution device, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated to use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0140] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0141] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0142] According to embodiments of the present invention, these and other variations of the hardware processor subsystem are also contemplated.

[0143] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, again according to the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0144] Computer Program Product Embodiment (“CPP Embodiment” or “CPP”) is a term used in this disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively included in a collection of one or more storage devices, the collection of one or more storage devices collectively including machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions used by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as perforated cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing.

[0145] Computer-readable storage media, as used in this disclosure, should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As those skilled in the art will understand, data is typically moved at certain incidental points in time during normal operation of the storage device, such as during access, defragmentation, or garbage collection; however, this does not make the storage device transient, because the data is not transient when it is stored.

[0146] References to "one embodiment" or "embodiment" and other variations of the invention in this specification mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification, as well as any other variations, do not necessarily refer to the same embodiment.

[0147] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to many of the listed items, as will be apparent to those skilled in the art and related fields.

[0148] Preferred embodiments of systems and methods for Monte Carlo normalization of calibration uncertainty in deep learning have been described (these are intended to be illustrative and not restrictive). It should be noted that modifications and variations can be made by those skilled in the art based on the above teachings. Therefore, it should be understood that changes can be made to the specific embodiments disclosed, and these changes are within the scope of the invention as summarized by the appended claims. Thus, aspects of the invention have been described in the details and features required by patent law, and the claimed and desired protection under a patent certificate is set forth in the appended claims.

Claims

1. A computer-based implementation method for layer normalization in machine learning applications, comprising: Sample the set of random activations corresponding to a fixed score of overall activation to provide multiple subsampled activations; Calculate the average value of activations across the multiple sub-samples; Calculate the standard deviation across the multiple subsampling activations; as well as Two statistical data, including the average value of the subsampled activations and the standard deviation across the plurality of subsampled activations, are used to normalize all activations of the activations as layer normalization.

2. The computer-implemented method of claim 1, wherein the sampling of the random activation set comprises selecting a set of pre-activated neurons from the total number of neurons in the neural network.

3. The computer-implemented method of claim 1, wherein calculating the average value and the standard deviation from the plurality of subsampled activations to normalize all activations of the activations as the layer normalization comprises: The subsampling activation is equal to: , The average value from the activation of the plurality of sub-samples is equal to , The standard deviation from the activation of the plurality of sub-samplings is equal to ,and The layer normalization is , where α is the activation in a layer of the neural network.

4. The computer-implemented method of claim 1, wherein the layer normalization is a step for training sequences of a neural network used in machine learning of a model for controlling a vehicle in an autonomous vehicle application.

5. The computer-implemented method of claim 1, wherein the layer normalization is a step for training sequences of neural networks used in machine learning of models in object identification applications.

6. A computer-based implementation of layer normalization for machine learning applications, comprising: Execute a training sequence comprising sampling a random set of activations corresponding to a fixed score of total activations to provide multiple subsampled activations, calculating an average across the multiple subsampled activations, calculating a standard deviation across the multiple subsampled activations, and using two statistics including the average of the subsampled activations and the standard deviation across the multiple subsampled activations to normalize all activations of the activations as a layer normalization. as well as The evaluation sequence is performed using a Monte Carlo approximation, which includes using subsampled input features for the calculation of normalized statistics, and using the layer normalization based on the training sequence to calculate the average value that makes the network run forward multiple times, wherein the average value is the output that provides the prediction.

7. The computer-implemented method of claim 6, wherein sampling the random activation set comprises selecting a set of pre-activated neurons from the total number of neurons in the neural network.

8. The computer-implemented method of claim 6, wherein calculating the average value and the standard deviation from the plurality of subsampled activations to normalize all activations of the activations as the layer normalization comprises: The subsampling activation is equal to: , The average value from the activation of the plurality of sub-samples is equal to , The standard deviation from the activation of the plurality of sub-samplings is equal to ,and The layer normalization is , where a ~l .

9. A computer implementation method for layer normalization in machine learning applications is provided, comprising: Execute a training sequence comprising sampling a random set of activations corresponding to a fixed score of total activations to provide multiple subsampled activations, calculating an average across the multiple subsampled activations, calculating a standard deviation across the multiple subsampled activations, and using two statistics including the average of the subsampled activations and the standard deviation across the multiple subsampled activations to normalize all activations of the activations as a layer normalization. as well as The evaluation sequence is performed using a sampling process that is converted into a Monte Carlo layer normalized version of the training sequence, and a Laplace approximation is used to provide predictions without subsampling the input features.

10. The computer-implemented method of claim 9, wherein the sampling of the activated random set comprises selecting a set of pre-activated neurons from the total number of neurons in the neural network.

11. The computer-implemented method of claim 9, wherein calculating the average value and the standard deviation from the plurality of subsampled activations to normalize all activations of the activations as the layer normalization comprises: The subsampling activation is equal to: , The average value from the activation of the plurality of sub-samples is equal to , The standard deviation from the activation of the plurality of sub-samplings is equal to ,and The layer normalization is , where a ~l .

12. The computer-implemented method according to claim 9, wherein the Laplace approximates: 。 13. A system for layer normalization in machine learning applications, comprising: Hardware processor; The system also includes a memory for storing computer program products, wherein the computer program product includes instructions, the instructions comprising: The hardware processor is used to sample a random set of activations corresponding to a fixed score of total activations to provide multiple subsampled activations; The hardware processor is used to calculate the average value across the multiple sub-sample activations; The hardware processor is used to calculate the standard deviation across the plurality of subsampling activations; and The hardware processor uses two statistical data, including the average value of the subsampled activations and the standard deviation across the plurality of subsampled activations, to normalize all activations of the activations as a layer normalization.

14. The system of claim 13, further comprising using the hardware processor to execute an evaluation sequence employing a Monte Carlo approximation, the Monte Carlo approximation including using subsampled input features for calculating normalized statistics, and using the layer normalization to calculate the average value that makes the network run forward multiple times, wherein the average value is an output providing a prediction.

15. The system of claim 13, further comprising using the hardware processor to execute an evaluation sequence, the evaluation sequence employing a sampling process converted to Monte Carlo layer normalization and employing a Laplace approximation to provide predictions without subsampling the input features.

16. The system of claim 13, wherein the sampling of the random activation set comprises selecting a set of pre-activated neurons of the neural network from the total number of neurons in the neural network.

17. The system of claim 13, wherein calculating the average value and the standard deviation from the plurality of subsampled activations to normalize all activations of the activations as the layer normalization comprises: The subsampling activation is equal to: , The average value from the activation of the plurality of sub-samples is equal to , The standard deviation from the activation of the plurality of sub-samplings is equal to ,and The layer normalization is , where a ~l .

18. The system of claim 13, wherein the layer normalization is a step for training sequences of a neural network used in machine learning of a model for controlling a vehicle in an autonomous vehicle application.

19. The system of claim 13, wherein the layer normalization is a step for training sequences of neural networks used in machine learning of models in object identification applications.

20. A computer program product for layer normalization in machine learning applications, the computer program product comprising a computer-readable storage medium having computer-readable program code embodied therein, the program code being executable by a hardware processor to cause the hardware processor to: Sample the set of random activations corresponding to a fixed score of overall activation to provide multiple subsampled activations; Calculate the average value of activations across the multiple sub-samples; Calculate the standard deviation across the multiple subsampling activations; as well as Two statistical data points, including the average value of the subsampled activations and the standard deviation across the plurality of subsampled activations, are used to normalize all activations of the activations as layer normalization.

21. The computer program product of claim 20, further comprising using the hardware processor to execute an evaluation sequence employing a Monte Carlo approximation, the Monte Carlo approximation including using subsampled input features for calculating normalized statistics, and using the layer normalization to calculate the average value that causes the network to run forward multiple times, wherein the average value is an output providing a prediction.

22. The computer program product of claim 20, further comprising executing an evaluation sequence using the hardware processor, the evaluation sequence employing a sampling process converted to Monte Carlo layer normalization and employing a Laplace approximation to provide predictions without subsampling the input features.

23. The computer program product of claim 20, wherein calculating the average value and the standard deviation from the plurality of subsampled activations to normalize all activations of the activations as the layer normalization comprises: The subsampling activation is equal to: , The average value from the activation of the plurality of sub-samples is equal to , The standard deviation from the activation of the plurality of sub-samplings is equal to ,and The layer normalization is , where a ~l .

24. The computer program product of claim 20, wherein the layer normalization is a step of training a neural network used in machine learning for a model controlling a vehicle in an autonomous vehicle application.

25. The computer program product of claim 20, wherein the layer normalization is a step for training sequences of neural networks used in machine learning of models in object identification applications.

Images