Method and apparatus for converting pre-trained model data formats without retraining

Abstract

The processor-implemented method for converting a pre-trained model from a first floating-point format to a second floating-point format involves receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate with values ​​in the first floating-point format. The layers of the ANN model to which scaling parameters will be applied are selected based on the layer type and topology of the ANN model. The maximum median is determined according to the weight distribution in the selected layers of the ANN model. The scaling parameter is calculated based on the ratio of the maximum median to the maximum representable value. The scaling parameter is applied to the weights of the selected layers of the ANN model.

Description

[Technical Field]

[0001] (Cross-reference of related applications)

[0001] This application claims the benefit of Indian Patent Application No. 202441010214, filed on 14 February 2024, entitled “METHODS AND APPARATUS FOR CONVERTING PRE-TRAINED MODEL DATA FORMAT WITHOUT RETRAINING,” and PCT Application No. PCT / US2023 / 034975, filed on 11 October 2023, entitled “METHODS AND APPARATUS FOR CONVERTING PRE-TRAINED MODEL DATA FORMAT, WITHOUT RETRAINING,” which in turn claims the benefit of Indian Patent Application No. 202341047622, ​​filed on 14 July 2023, whose disclosures are expressly incorporated in their entirety by reference.

[0002]

[0002] The aspects of the present disclosure relate in general to data processing, and more specifically to floating-point data in machine learning models. The aspects relate in more detail to converting a pre-trained model from one that operates on floating-point data in a first format to one that operates on floating-point data in a second format, without retraining the model. [Background technology]

[0003]

[0003] An artificial neural network may comprise an interconnected group of artificial neurons (e.g., neuron models). An artificial neural network may be a computing device or may be represented as a method to be executed by a computing device. Convolutional neural networks (CNNs) are one type of feedforward artificial neural network. A convolutional neural network may comprise a collection of neurons, each having a receptive field and tiling the input space together. Convolutional neural networks, such as deep convolutional neural networks (DCNs), have numerous applications. Specifically, these neural network architectures are used in a variety of technologies, including image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks.

[0004]

[0004] The use of floating-point values ​​makes calculations in many neural network applications sufficiently accurate. This is especially true for applications such as autonomous driving. Representing floating-point values ​​in computing is very difficult considering the constraints of device size and memory. Floating-point values ​​are generally represented using scientific or engineering notation. Considering bit formats (e.g., 16-bit or 32-bit), the number of bits can be divided to represent different parts of a floating-point value in scientific notation. For example, one bit may represent the sign of a floating-point value. One set of bits may represent the mantissa (also called the significant bit), and another set of bits may represent the exponent.

[0005]

[0005] In computing, half-precision is a format for representing floating-point values. Half-precision can be called a floating-point bit format that uses 16 bits in computer memory. However, half-precision formats can offer lower precision. The industry standard for half-precision formats is the Institute for Electrical and Electronic Engineers (IEEE) 16-bit floating-point (FP16) format. The IEEE FP16 format consumes less memory and enables faster calculations than high-precision formats such as single-precision (32-bit format) or double-precision (64-bit format), but offers lower precision and a narrower range.

[0006]

[0006] To provide higher accuracy in neural network applications, some conventional methods have attempted to use non-standard FP16 formats (e.g., formats other than IEEE FP16). However, when pre-trained models are designed in such non-standard FP16 formats, it can be difficult, and sometimes impossible, to utilize inference accelerators that are often designed to support standard data formats. [Overview of the Initiative]

[0007]

[0007] Each of the present disclosures is described in an independent claim. Some aspects of the present disclosures are described in dependent claims.

[0008]

[0008] In some aspects of the present disclosure, a processor-implemented method includes taking an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having bit lengths. The processor-implemented method also includes selecting layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. The processor-implemented method further includes determining the maximum median value according to the weight distribution in the selected layers. The processor-implemented method further includes calculating the scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the processor-implemented method also includes applying the scaling parameters to the weights of the selected layers of the ANN model.

[0009]

[0009] Various aspects of the present disclosure relate to an apparatus that includes means for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having bit length. The apparatus also includes means for selecting layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. The apparatus further includes means for determining the maximum median value according to the weight distribution in the selected layers. The apparatus additionally includes means for calculating scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the apparatus includes means for applying scaling parameters to the weights of the selected layers of the ANN model.

[0010]

[0010] In some aspects of the present disclosure, a non-temporary computer-readable medium recording program code is disclosed. The program code is executed by a processor and includes program code for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having bit length. The program code also includes program code for selecting layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. The program code further includes program code for determining the maximum median value according to the weight distribution in the selected layers. The program code additionally includes program code for calculating scaling parameters based on the ratio of the maximum median value to the maximum representable value. The program code also includes program code for applying scaling parameters to the weights of the selected layers of the ANN model.

[0011]

[0011] Various aspects of the present disclosure relate to a device having at least one memory and one or more processors coupled to at least one memory. The processor(s) are configured to receive an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having bit length. The processor(s) are also configured to select layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. The processor(s) are further configured to determine the maximum median value according to the weight distribution in the selected layers. The processor(s) are further configured to calculate scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the processor(s) are configured to apply scaling parameters to the weights of the selected layers of the ANN model.

[0012]

[0012] Additional features and advantages of the present disclosure are described below. Those skilled in the art will understand that the present disclosure can be readily used as a basis for modifying or designing other structures to accomplish the same purposes of the present disclosure. Those skilled in the art will also recognize that such equivalent configurations do not deviate from the teachings of the present disclosure as set forth in the appended claims. Novel features that are considered to be unique to the present disclosure will be better understood, along with further purposes and advantages, with respect to both their organization and operation, by examining the following descriptions in relation to the appended drawings. However, it should be clearly understood that each of the drawings is provided for illustrative and explanatory purposes only and is not intended to define the limits of the present disclosure. [Brief explanation of the drawing]

[0013]

[0013] The features, properties, and advantages of this disclosure will become more apparent from the detailed description below when read in conjunction with the drawings, which throughout them use similar reference numerals to identify corresponding elements. [Figure 1]

[0014] This disclosure provides several exemplary implementations of neural networks using a system-on-a-chip (SOC) including a general-purpose processor. [Figure 2ABC]

[0015] This figure shows neural networks in various aspects of the present disclosure. [Figure 2D]

[0016] This figure shows exemplary deep convolutional networks (DCNs) according to various aspects of the present disclosure. [Figure 3]

[0017] This block diagram shows exemplary deep convolutional networks (DCNs) according to various aspects of the present disclosure. [Figure 4]

[0018] This block diagram shows exemplary software architectures that allow for the modularization of artificial intelligence (AI) functions, according to various aspects of this disclosure. [Figure 5]

[0019] FIG. is a diagram illustrating an example of pseudocode for determining a scaling factor according to various aspects of the present disclosure. [Figure 6A]

[0020] FIG. is a block diagram showing an example of an artificial neural network (ANN) model according to various aspects of the present disclosure and a corresponding conversion from a 16-bit brain floating point (bfloat16) format using a scaling factor to an Institute of Electrical and Electronics Engineers (IEEE) 16-bit floating point (FP16) format. [Figure 6B] FIG. is a block diagram showing an example of an artificial neural network (ANN) model according to various aspects of the present disclosure and a corresponding conversion from a 16-bit brain floating point (bfloat16) format using a scaling factor to an Institute of Electrical and Electronics Engineers (IEEE) 16-bit floating point (FP16) format. [Figure 7]

[0021] FIG. is a flowchart showing an exemplary method implemented on a processor for converting a pre-trained model from a bfloat16 format to an FP16 format using various scaling factors according to various aspects of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0022] The following detailed description of the invention with reference to the accompanying drawings describes various configurations and does not represent the only configuration in which the described concepts can be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some examples, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0015]

[0023] Those skilled in the art will understand, based on the teachings, that the scope of this disclosure is intended to encompass any aspect of the disclosure, whether implemented independently of or in combination with any other aspect of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. In addition, the scope of this disclosure is intended to encompass any such apparatus or method practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described. It will be understood that any aspect of the disclosed disclosure may be embodied by one or more elements of the claims.

[0016]

[0024] The word "exemplary" is used to mean "serving as an example, case, or illustration." Any aspect described as "exemplary" is not necessarily interpreted as being preferable or more advantageous than other aspects.

[0017]

[0025] While specific embodiments are described, many variations and substitutions of these embodiments fall within the scope of this disclosure. While some advantages and benefits of preferred embodiments are stated, the scope of this disclosure is not limited to any particular advantage, use, or purpose. Rather, the embodiments of this disclosure are broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated as examples in the figures and the following description of preferred embodiments. The embodiments and drawings for carrying out the invention are not limiting but merely illustrative of this disclosure, and the scope of this disclosure is defined by the appended claims and equivalents.

[0018]

[0026] Brain floating-point format is a custom 16-bit floating-point format (may be called bfloat16 or BF16). The bfloat16 format includes 1 sign bit, 8 exponents, and 7 mantissas. This differs from the industry-standard Institute of Electrical and Electronics Engineers (IEEE) 16-bit floating-point (FP16) format, which was not designed with deep learning applications in mind. The IEEE FP16 format includes 1 sign bit, 5 exponents, and 10 mantissas.

[0019]

[0027] Bfloat16 is a truncated version of IEEE 32-bit single-precision floating-point number developed by Google Brain, which reduces the fractional part (e.g., mantissa) to 7 bits while keeping the exponent at 8 bits. Bfloat16 aims to reduce model training time and overall size. However, due to the small number of bits in the fractional part (e.g., mantissa), models tend to use the exponent to represent weights. As a result, small changes in the exponent can cause the absolute values ​​of weights to become too high. That is, models with higher exponents, and therefore higher values, may not fit into hardware (e.g., memory) that supports IEEE FP16, thereby leading to data overflow and degrading model accuracy. Furthermore, even if the weights are within the IEEE FP16 range, intermediate activation values ​​may also fall outside the IEEE FP16 range due to the higher values ​​of the weights.

[0020]

[0028] Some conventional methods attempt to scale down all weights to predetermined hardcoded values ​​to fit the IEEE FP16 range. However, such methods rely on empirical selection of scaling factors until a solution works, and may fail for different inputs not encountered or for other datasets. Additionally, these conventional methods only work on certain architectures and may struggle to handle nonlinearities other than rectified linear units.

[0021]

[0029] Accordingly, in order to address these and other challenges, aspects of this disclosure focus on converting a pre-trained model from the bfloat16 format to the IEEE FP16 format. For dense layers with non-scale-invariant activation values ​​or non-zero biases, weight scaling may be applied to the architecture including pre-activation, or by applying scaling factors to the non-zero bias.

[0022]

[0030] Certain aspects of the subject matter described herein can be implemented to achieve one or more of the following potential benefits. In some examples, the techniques described may be usefully converted from bfloat16 format to IEEE FP16 format without retraining. Thus, aspects of this disclosure may avoid the tedious and time-consuming task of retraining artificial neural network (ANN) models. Additionally, the techniques described may result in a converted model that preserves the accuracy (e.g., improved precision) of the original ANN model while allowing the use of conventional accelerators. Furthermore, the disclosed techniques may also reduce inference time because they allow the ANN model to operate over a smaller value range.

[0023]

[0031] Furthermore, text-based video editing can enhance camera capabilities. For example, aspects of this disclosure may enable content editing of videos captured using a camera. For instance, the captured video may be edited before being shared. Additionally, user personalization may be performed to inject personal effects into the video.

[0024]

[0032] Certain embodiments and techniques described herein can be at least partially implemented using artificial intelligence (AI) programs, such as programs that include machine learning (ML) or artificial neural network (ANN) models. An exemplary ML model may include a mathematical representation or define computational power for making inferences from input data based on patterns or relationships identified within the input data. As used herein, the term “inference” may include one or more judgments, predictions, decisions, or values ​​that may represent the output of an ML model. Computational power may be defined with respect to specific parameters of the ML model, such as weights and biases. Weights may represent a relationship between specific input data and a specific output of the ML model, while biases may represent an offset that indicates the starting point of the output of the ML model. An exemplary ML model operating on input data may start with an initial output based on biases and then update its output based on a combination of input data and weights.

[0025]

[0033] ML models can be characterized in terms of the type of learning that produces a specific type of trained model that performs a specific type of task. For example, various types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. ML models can be used to perform various tasks such as classification or regression, where classification refers to determining one or more discrete output values ​​from a given set of output values, and regression refers to determining a continuous value that is not restricted by a given output value. Some exemplary ML models configured to perform such tasks include ANNs such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), transformers, diffusion models, regression analysis models (statistical models, etc.), large language models (LLMs), decision tree learning (predictive models, etc.), support vector networks (SVMs), and probabilistic graphical models (Bayesian networks, etc.).

[0026]

[0034] Figure 1 shows an exemplary implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU, configured to convert a pre-trained model from one operating with floating-point data in a first format to a second format without retraining the model. Variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., neural networks with weights), delays, frequency bin information, and task information may be stored in memory blocks associated with the neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with the graphics processing unit (GPU) 104, memory blocks associated with the digital signal processor (DSP) 106, memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 or from memory block 118.

[0027]

[0035] The SOC100 may also include a connectivity block 110 which may include a GPU 104, a DSP 106, fifth-generation (5G) connectivity, fourth-generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and additional processing blocks adapted to specific functions, such as a multimedia processor 112 which can detect and recognize gestures. In one implementation, the NPU 108 is implemented in the CPU 102, DSP 106, and / or GPU 104. The SOC100 may also include a navigation module 120 which may include a sensor processor 114, image signal processors (ISPs) 116, and / or a global positioning system.

[0028]

[0036] The SOC100 may be based on the ARM instruction set. In one aspect of this disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having bit length. The instructions loaded into the general-purpose processor 102 may also include code for selecting the layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. The instructions loaded into the general-purpose processor 102 may further include code for determining the maximum median value according to the weight distribution in the selected layers. The instructions loaded into the general-purpose processor 102 may additionally include code for calculating scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the instructions loaded into the general-purpose processor 102 may include code for applying the scaling parameters to the weights of the selected layers of the ANN model.

[0029]

[0037] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction within each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck in traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems relied heavily on human-designed features, perhaps in combination with shallow classifiers. A shallow classifier could be, for example, a two-class linear classifier where a weighted sum of feature vector components is compared to a threshold to predict which class an input belongs to. Human-designed features could be templates or kernels adapted to a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures can represent similar features to those that human engineers can design, but can learn through training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans may not have considered.

[0030]

[0038] Deep learning architectures can learn feature hierarchies. When presented with visual data, for example, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at a specific frequency. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0031]

[0039] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, classifying motorized vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0032]

[0040] Neural networks can be designed using various connectivity patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer transmitting information to neurons in higher layers. As mentioned above, a hierarchical representation can be constructed within the consecutive layers of a feedforward network. Neural networks can also have recursive or feedback (also called top-down) connectivity. In recursive connectivity, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns that span two or more chunks of input data delivered to the neural network in a sequence. Connectivity from a neuron in a given layer to a neuron in a lower layer is called feedback (or top-down) connectivity. Networks with many feedback connectivitys can be useful when the recognition of a high-level concept can help identify specific low-level features of the input.

[0033]

[0041] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, neurons in the first layer can transmit their output to all neurons in the second layer, thereby each neuron in the second layer receiving input from all neurons in the first layer. Figure 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, neurons in the first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of a locally connected neural network 204 can be configured such that each neuron in the layer has the same or similar connectivity pattern but can have different connection strengths (e.g., 210, 212, 214, and 216). Since higher-layer neurons within a given region can receive inputs that, through training, are tuned to the characteristics of a limited portion of the total input to the network, the connection patterns of local connections can create spatially distinct receptive fields within the higher layers.

[0034]

[0042] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful.

[0035]

[0043] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of DCN200, designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. In this example, DCN200 can be trained to identify traffic signs and the numbers written on them. Of course, DCN200 can be trained for other tasks, such as identifying lane markings or traffic signals.

[0036]

[0044] DCN200 can be trained using supervised learning. During training, DCN200 may be presented with images such as image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel in convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, four different convolutional kernels were applied to image 226 in convolutional layer 232, since four different feature maps are generated in the first set of feature maps 218. Convolutional kernels are also sometimes called filters or convolutional filters.

[0037]

[0045] A first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218; that is, the size of the second set of feature maps 220, such as 14x14, is smaller than the size of the first set of feature maps 218, such as 28x28. This smaller size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0038]

[0046] In the example in Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may contain a number corresponding to a possible feature of image 226, such as "label", "60", and "100". A softmax function (not shown) can be used to convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN200 may be the probability that image 226 contains one or more features.

[0039]

[0047] In this example, the probabilities in output 222 for "label" and "60" are higher than the probabilities for other outputs in output 222 such as "30", "40", "50", "70", "80", "90", and "100". Before training, the outputs 222 generated by DCN200 are likely to be inaccurate. Therefore, an error can be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., "label" and "60"). The weights of DCN200 can then be adjusted so that the output 222 of DCN200 is closer to the target output.

[0040]

[0048] To adjust the weights, the learning algorithm may compute a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient may directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient may depend on the weight values ​​and the computed error gradients from the upper layers. The weights can then be adjusted to minimize the error. This method of weight adjustment is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0041]

[0049] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, DCN200 may be presented with new images, and a forward pass through DCN200 may yield an output 222 that can be considered DCN200's inference or prediction.

[0042]

[0050] Optimization algorithms can be used during the training process to adjust weights and biases as needed to reduce or minimize the loss function, which should improve the model's performance. There are various optimization algorithms that can be used in conjunction with backpropagation techniques or other training techniques. Some early examples include optimization algorithms based on gradient descent and optimization algorithms based on stochastic gradient descent. Stochastic gradient descent techniques can be used to adjust weights / biases to minimize or reduce the loss function. Mini-batch gradient descent techniques, a variation of gradient descent, may involve updating weights / biases using small batches of training data rather than the entire dataset. Momentum techniques can accelerate the optimization process by adding momentum terms to update or influence specific weights / biases.

[0043]

[0051] Adaptive learning rate techniques can adjust the learning rate of an optimization algorithm based on one or more characteristics of the training data. Batch normalization techniques can be used to normalize the input to the model, stabilizing the training process and potentially improving the model's performance. "Dropout" techniques can be used to randomly drop out some artificial neurons from the model during the training process, for example, to reduce overfitting and potentially improve the model's generalization. "Early termination" techniques can be used to terminate the ongoing training process early, for example, when the model's performance on the validation dataset begins to decline.

[0044]

[0052] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. Because RBMs can learn probability distributions without information about which class each input should belong to, they are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lower RBM of a DBN can be trained unsupervised and can function as a feature extractor, while the upper RBM can be trained supervised (on a combined distribution of inputs from previous layers and target classes) and can function as a classifier.

[0045]

[0053] DCNs are convolutional networks composed of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output targets are known in a large number of cases, and the network weights are modified using gradient descent.

[0046]

[0054] A DCN can be a feedforward network. In addition, as mentioned above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN may be significantly less than that of a similarly sized neural network, for example, one that includes recursive or feedback connections.

[0047]

[0055] The processing of each layer of a convolutional network can be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input can be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension that captures color information. The output of the convolutional connections can be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed using rectification, max(0,x), and other nonlinearities. Values ​​from adjacent neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization, corresponding to whitening, can also be applied through lateral inhibition between neurons in the feature map.

[0048]

[0056] Figure 3 is a block diagram of a deep convolutional network (DCN) 350. The DCN 350 may include several different types of layers based on coupling and weight sharing. As shown in Figure 3, the DCN 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0049]

[0057] Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks 354A and 354B may be included in the DCN350 according to design preferences.

[0050]

[0058] The convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolutional filters. For example, the normalization layer 358 may provide whitening or side suppression. The maximal pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0051]

[0059] For example, a parallel filter bank of a deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 (e.g., Figure 1) to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN350 may have access to other processing blocks that may reside on the SOC 100, such as the sensor processor 114 and the navigation module 120, which are dedicated to sensors and navigation, respectively.

[0052]

[0060] DCN350 may also include one or more fully connected layers 362 (FC1 and FC2). DCN350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of DCN350 there are weights (not shown) that will be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as input to one of the subsequent layers (e.g., 356, 358, 360, 362, 364) in DCN350 to learn a hierarchical feature representation from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied in the first of the convolutional blocks 354A. The output of DCN350 is a classification score 366 for the input data 352. A classification score of 366 can be a set of probabilities, where each probability is the probability of the input data containing a feature from the set of features.

[0053]

[0061] Figure 4 is a block diagram showing an exemplary software architecture 400 in which artificial intelligence (AI) functions can be modularized. Using architecture 400, an application can be designed that enables various processing blocks of the SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) to support floating-point format conversion for an AI application 402, according to aspects of this disclosure. Architecture 400 may be included in a computing device such as a smartphone.

[0054]

[0062] The AI ​​application 402 may be configured to invoke functions defined in user space 404, for example, which can provide scene detection and recognition indicating the location where a computing device, including architecture 400, is currently operating. The AI ​​application 402 may configure the microphone and camera differently depending on whether the scene to be recognized is an office, auditorium, restaurant, or an outdoor setting such as a lake. The AI ​​application 402 may make requests to compiled program code associated with a library defined in the AI ​​Functional Application Programming Interface (API) 406. These requests may ultimately rely on the output of a deep neural network configured to provide inference responses based on video and positioning data, for example.

[0055]

[0063] The AI ​​application 402 may also have further access to a runtime engine 408, which may be compiled code for a runtime framework. The AI ​​application 402 may request inference from the runtime engine 408, for example, at specific time intervals or triggered by events detected by the user interface of the AI ​​application 402. The runtime engine 408 may then signal an operating system in the operating system (OS) space 410, such as a kernel 412 running on the SOC 420, when it is required to provide an inference response. In some examples, kernel 412 may be a LINUX kernel. The operating system may then cause a continuous relaxation of quantization on the CPU 422, DSP 424, GPU 426, NPU 428, or any combination thereof. The CPU 422 may be directly accessed by the operating system, while the other processing blocks may be accessed through drivers such as drivers 414, 416, or 418, respectively, for the DSP 424, GPU 426, or NPU 428. In an exemplary example, the deep neural network may be configured to operate on a combination of processing blocks such as CPU422, DSP424, and GPU426, or it may operate on NPU428.

[0056]

[0064] As described above, aspects of this disclosure relate to converting a pre-trained model from one that operates with floating-point data in a first format to one that operates with floating-point data in a second format, without retraining the model. For example, the first format may be the bfloat16 format, and the second format may be the IEEE FP16 format.

[0057]

[0065] In various embodiments, the ANN architecture and weights can be mathematically modeled to determine sensitive layers or blocks (e.g., blocks that may be more susceptible to overflow errors). For example, a mathematical model may be applied to determine which blocks (or layers) of an ANN can be scaled and which blocks of the ANN can be retained without scaling. Different scaling factors may be calculated for different blocks and weights. In some embodiments, worst-case analysis may be used to further reduce overflow errors. For example, the worst-case range of scaling factors can be determined without using a dataset.

[0058]

[0066] Figure 5 illustrates an example of pseudocode 500 for determining scaling factors according to an aspect of the present disclosure. Referring to Figure 5, the exemplary pseudocode 500 may determine scaling factors for parameters (e.g., weights) in various blocks of an ANN model (e.g., a self-attention block or a cross-attention block). Rather than determining a uniform scaling factor (e.g., a fixed scaling factor) that applies to multiple blocks (or layers) of the ANN, different scaling factors may be determined for different blocks (or layers) of the ANN. For example, different scaling factors may be applied to layers including (but not limited to) dense layers, self-attention layers, or normalization layers. In some embodiments, the ANN model may include, for example, a convolutional neural network (CNN) (e.g., DCN350 shown in Figure 3), a recurrent neural network (RNN), a transformer neural network, or other neural networks.

[0059]

[0067] Given the sheath norm weights, value projection weights, output projection weights, the dimension of the ANN input, and the maximum representable value (for example, with respect to a specific hardware device such as a hardware accelerator), a scaling factor can be determined. As shown in line 502, an intermediate value v can be calculated. The intermediate value v is obtained by the sheath norm weights G and the transpose of the value projection weights V (GV Tand the square root of the input dimension

[0060]

Number

[0061] can be given by performing matrix multiplication on the product of the square root of the input dimension. At line 504, the maximum value of the intermediate value (v max y ) can be determined. Next, at line 506, the maximum output value (o max ) can be determined. The maximum output value (o max ) can be calculated by multiplying the intermediate value v by the transpose of the maximum value of the output projection weight O. At line 508, the scaling factor (f v ) of the value projection weight can be determined by dividing the maximum intermediate value v max by the maximum representable value m. For example, in the case of IEEE FP16, the maximum representable value can be 65004. Additionally, at line 510, the scaling factor (f o ) of the output projection weight can be determined by dividing the maximum output value o max by the maximum representable value m. Since the maximum output value can vary from block to block, the scaling factor can also vary accordingly. Therefore, aspects of the present disclosure can determine the scaling factor based on the distribution of weights within a block rather than based on a dataset (e.g., a validation dataset). That is, the scaling factors (f v , f o ) are based on the weights of various layers (e.g., layer norm weights, value projection weights, output projection weights) rather than on a specific value of the input.

[0062]

[0068] According to aspects of the present disclosure, weight scaling can be applied such that the ANN model weights take on lower absolute values while maintaining mathematical equivalence across blocks in the ANN model. Lower absolute weights can reduce overflow and, in some aspects, can make the intermediate activation values (e.g., intermediate layer outputs) less susceptible to IEEE FP16 overflow.

[0063]

[0069] As explained, different scaling strategies can be employed for different architectures. The scaling coefficients at different layers of an ANN model can be mathematically estimated using the weights of each layer. In other words, different layers of an ANN model can be adapted in different ways based on their mathematical properties. For example, in the case of dense layers with or without rectifier linear unit (ReLU) activation (e.g., fully connected layers), the weights can be scaled as follows:

[0064]

number

[0065] Here, W is the weight, x is the input, b is the bias, and α is the scaling factor.

[0066]

[0070] If the dense layer has no bias, the scaling factor may be applied as shown in Equation 1. On the other hand, if the dense layer has a bias, the scaling factor may be applied as shown in Equation 3. That is, since the bias can be scaled along with the weights, the output is scaled in a similar manner. In some embodiments, Equation 3 may only be applicable if the activation function is linear on both sides (negative and positive) of zero.

[0067]

[0071] The activation in Equation 3 is the ReLU function, but this disclosure is not limited thereto. Rather, scaling factors can be applied even if ReLU is not the activation function in the ANN model, as long as a nonlinear function is applied in “pre-activation”. That is, scaling factors can be applied when a nonlinear activation is included before (e.g., prior to) a linear or convolutional neural network (CNN) layer. For example, in a stable diffusion XL (SDXL) model, the layers may be arranged in the order of group normalization, sigmoid linear unit (SiLU) nonlinearity, and convolutional layer. The SiLU function can be expressed as x*σ(x), where σ(x) is the logistic sigmoid function. In this example, the weights and biases of the convolution can be scaled because the next block in the SDXL model has layers in the same order. Group normalization, which can reset the scaling factors, is applied first. Therefore, the output of the current block may remain unchanged.

[0068]

[0072] If a block (or layer) has residual connections, a scaling factor may be applied to both the input and output so that they are on the same scale and can be added together as shown below.

[0069]

number

[0070]

[0073] Therefore, with respect to layers with residual connections, the input can be scaled by scaling the weights of the previous layer or by introducing scaling operators between nodes.

[0071]

[0074] Additionally, if the ANN model has a normalization layer (e.g., layer normalization (LayerNorm) or root mean square layer normalization (RMSNorm)), the scaling of the activation values ​​can be reset. That is, the scaling coefficient applied in the previous layer has no effect. This is because if weight scaling is applied to a layer and that layer is followed by a normalization layer (e.g., LayerNorm or RMSNorm), the output of the normalization layer will be the same as the output of normalization without scaling. Therefore, weight scaling can be applied to further layers (e.g., residuals), or rescaling can be applied to the output by adapting the normalization layer, for example, as follows:

[0072]

number

[0073]

[0075] In some embodiments, scaling operators may be employed alternatively or additionally. For example, an element-wise multiplication node / operation may be added to the points, with a scaling factor as a parameter.

[0074]

number

[0075] Here,

[0076]

number

[0077] This indicates the tensor product.

[0078]

[0076] Scaling operators can be inserted, for example, between blocks or layers of an ANN model to descaling / rescaling the currently maintained scaling in the layer activation values. For example, when using RMSNorm with residual connections, the scaling operators can rescale the input.

[0079]

[0077] Since each block has a different weight, additional analysis of the ANN model may be performed to determine the scaling factor. Using a single large scaling factor may reduce the accuracy of the ANN model, for example, because the accelerator may not be able to represent the scaled value.

[0080]

[0078] In various embodiments, determining the scaling coefficient may involve accessing the extreme values ​​of the input (e.g., minimum (min) and maximum (max)) which are the outputs of the previous layer. Thus, an analysis of the extreme values ​​of all preceding layers can be performed.

[0081]

[0079] In the analysis described, if the model has a normalization layer (e.g., LayerNorm or RMSNorm), only the extrema of the normalized tensor (e.g., the weight vector to which normalization has been applied) is used, rather than the extrema of the preceding layer.

[0082]

[0080] In order to find the extrema of the result of a matrix multiplication (MatMul) operation, the input may be set such that the input vector is parallel to the longest basis (e.g., the longest column vector) of the weight matrix. The length of the input vector used may be fixed due to the normalization layer. The maximum length is at most before multiplication with the weights.

[0083]

number

[0084] And it can be specified as follows:

[0085]

number

[0086] Here, a i is the i-th dimension of the input vector, and g i is the i-th dimension of the LayerNorm weight vector G, and b i is the i-th dimension of the LayerNorm bias.

[0087]

[0081] The maximum absolute value can be calculated by analyzing the block weights of the ANN model. For example, the maximum absolute value can be calculated as follows:

[0088]

number

[0089] Here, Lin represents the linear operator and the length of the vector.

[0090]

number

[0091] This can be due to normalization, A T is the transpose of the attention matrix A, and Rn is the RMSNorm function defined in Equation 9. Next,

[0092]

number

[0093] The largest absolute value of a matrix multiplication, matrix GA T It can be the longest basis.

[0094]

[0082] The elements of the result of matrix multiplication are the result of the dot product of the two vectors. That is,

[0095]

number

[0096] Here, the maximum absolute value of cosθ is 1. Next, the maximum possible result is:

[0097]

number

[0098] It can be the maximum value. As mentioned above,

[0099]

number

[0100] The maximum absolute value in

[0101]

number

[0102] Therefore, the maximum absolute value of the result of the MatMul operation following RMSNorm can be expressed as follows:

[0103]

number

[0104] Here,

[0105]

number

[0106] is a matrix GA T This is the length of the longest base.

[0107]

[0083] After obtaining the maximum absolute value, the scaling coefficient of the linear sheath weight matrix can be determined as follows.

[0108]

number

[0109] Here, ceil(x) is the ceiling function that calculates the smallest integer greater than or equal to x, and the maximum absolute value is

[0110]

number

[0111] It can be derived that the maximum number of representable values ​​in IEEE FP16 is, for example, 65504.

[0112]

[0084] Therefore, different scaling factors can be determined for each block or layer of the ANN model. For example, if the ANN model includes a transformer model, the same transformer decoder block may be repeated multiple times within the model. However, since the maximum value may differ for each transformer decoder block, the scaling factor in each of such transformer decoder blocks may differ for each transformer decoder block, unlike conventional methods.

[0113]

[0085] Following the first linear layer after layer normalization, there may be further layers in which IEEE FP16 overflows may occur. For example, in the case of an attention module, the output may be expressed as follows:

[0114]

number

[0115] Here, z is the normalized output from RMSNorm (in some aspects z may represent the normalized output of LayerNorm as shown in Equation 8), K is the key projection matrix, Q is the query projection matrix, V is the value projection matrix, and O is the output projection matrix. The last output projection matrix O MatMul may not be compatible with V MatMul because it has been reshaped for multi-head attention. Additionally, the maximum value of zV can be calculated, for example, as described in Equations 10-14. To determine the maximum value of softmax(zV) (e.g., the maximum possible output), the maximum possible value of each column of zV can be determined. Each column of zV is GA T The norm of the corresponding column of the matrix

[0116]

number

[0117] It can have the maximum value obtained by multiplying by . That is,

[0118]

number

[0119]

[0086] The value of softmax can have a maximum value of 1. Then, softmax() zV can produce the maximum possible row only if all columns of the projection weight matrix V are the same, which is very unlikely but possible in the worst case. That is,

[0120]

number

[0121]

[0087] In the above example, the first row of the result of softmax()zV may be the maximum value. In this row, the maximum value of the result of (softmax()zV)O) can be determined using the norm of this vector which has the maximum norm of the output weight matrix O. That is,

[0122]

number

[0123]

[0088] Here too, the maximum value can be determined as follows.

[0124]

number

[0125]

[0089] Next, the maximum value can be used to determine the scaling factor in the output projection matrix multiplication, as shown in Equation 15.

[0126]

[0090] By applying the techniques described to selectively scale blocks or layers of the ANN model, the ANN model can be converted, for example, from bfloat16 to IEEE FP16 without retraining the model. Thus, in various embodiments, the ANN model can operate using hardware that supports IEEE FP16 instead of bfloat16 (e.g., a hardware accelerator).

[0127]

[0091] Figures 6A and 6B are block diagrams showing exemplary ANN model 600 and the corresponding transformation from bfloat16 to IEEE FP16 using scaling factors, according to various aspects of the present disclosure. Referring to Figure 6A, exemplary ANN model 600 may be received as input. Exemplary ANN model 600 may include a transformer neural network, as shown in the non-limiting example of Figure 6A. Transformer neural networks may include, for example, transformer architectures such as bi-directional encoder representations from transformers (BERT), robustly optimized BERT approach (RoBERTa), XLNet, Transformer-XL, and generative pre-trained transformer (GPT) family transformer (e.g., GPT-2).

[0128]

[0092] An exemplary ANN model 600 may include a plurality of blocks 604a to z (collectively referred to as block 604). Each of the blocks (e.g., 604a to z) may include a self-attention layer and a feedforward layer. In some embodiments, the blocks (e.g., 604a to z) may also include cross-attention blocks.

[0129]

[0093] The ANN model 600 can be trained to operate with values ​​in bfloat16 format. That is, the weights of blocks (e.g., 604a to z) can be pre-trained to operate on received embeddings (e.g., 602a and 602b) according to the bfloat16 format.

[0130]

[0094] Referring to Figure 6B, an exemplary block (e.g., 604b) is shown. The exemplary block (e.g., 604b) includes a self-attention layer 622, a cross-attention layer 624, and a feedforward layer 626. Block 628 provides a set of example operations to be performed in each block (e.g., 604a-z) of the received ANN model 600, which has been pre-trained according to the IEEE FP16 format (e.g., without scaling).

[0131]

[0095] As shown in block 630, for example, a scaling factor α may be applied to a selected block (e.g., 604b) or layer (e.g., 622) of the ANN model 600, depending on the type of layer. However, the value of the scaling factor α may be determined based on the topology and weight distribution in each layer, as shown in equations 1 to 22, for example. For example, in block 630, as in block 628, the ANN (e.g., 600) includes an RMS normalized layer (632). As shown in 634, the scaling factor α is applied to the value-projected weights. The scaling may be propagated through the network so that the output is also scaled (636). In another example, in 638, the ANN (e.g., 600) may include a dense layer with ReLU activation. In this example, the ReLU does not have a bias term. Thus, the scaling parameter α may be applied in 640, in which case the weights W1 may be scaled by the scaling factor α as specified in equations 1 and 2. However, if the dense layer contains bias, the scaling factor may be applied to scale not only the bias but also the weights (e.g., W1), as specified in Equation 3. Thus, different scaling factors may be applied to different layers (e.g., 622) of the ANN model 600.

[0132]

[0096] In some embodiments, different scaling parameters may be applied to layers of the same type. Since the scaling parameters may be based on the weight distribution of the layer, and the weight distributions of even layers of the same type may be different, different values ​​of the scaling parameters may be determined for each of such layers.

[0133]

[0097] Figure 7 is a flowchart illustrating an exemplary processor-implemented method 700 for converting a pre-trained model from bfloat16 to IEEE FP16 using various scaling factors, according to various aspects of the present disclosure. The processor-implemented method 700 may be performed by one or more processors, such as CPU102, GPU104, and / or NPU108.

[0134]

[0098] As shown in Figure 7, in block 702, at least one processor receives an artificial neural network (ANN) model as input. The ANN model is trained to operate with values ​​in 16-bit brain floating-point (bfloat16) format. For example, as shown in Figure 6A, an exemplary ANN model 600 may be received as input. The exemplary ANN model 600 may include, for example, a transformer neural network. The transformer neural network may include, for example, a stable spreading model such as the SDXL model, or a transformer architecture such as Bidirectional Encoder Representation from Transformers (BERT), Robust Optimized BERT Method (RoBERTa), XLNet, Transformer-XL, and the Generative Pre-trained Transformer (GPT) family of transformers (e.g., GPT-2).

[0135]

[0099] In block 704, at least one processor selects a layer of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. For example, as shown in Figure 6B, a layer in the block (e.g., 604b) may be selected to apply scaling factors based on the layer type and topology. In one example, according to block 630, scaling factors may be applied to the self-attention layer 622. For example, if the layer includes an RMS normalization layer (e.g., 632), weight scaling may be applied.

[0136]

[0100] In block 706, at least one processor determines the maximum median value according to the weight distribution in the selected layer of the ANN model. As explained with reference to Figure 5, the median value v is the transpose of the layer norm weights G and the value projection weights V (GV T ) and the square root of the input dimension

[0137]

number

[0138] It can be obtained by multiplying by matrix multiplication. At line 504, the maximum value of the intermediate (v max ) can be determined. Next, on line 506, the maximum output value (o max ) can be determined. Maximum output value (o max ) can be calculated by multiplying the intermediate value v by the transpose of the maximum value of the output projection weight O.

[0139]

[0101] In block 708, at least one processor calculates a scaling parameter based on the ratio of the maximum median value to the maximum representable value. For example, the scaling coefficient α is defined as in equation 15,

[0140]

number

[0141] This can be determined as the ceiling of the ratio between the maximum intermediate value of the ANN layer and the maximum number of representable values, derived as follows:

[0142]

[0102] In block 710, at least one processor applies a scaling parameter to the weights of a selected layer of the ANN model. For example, as illustrated with reference to Figure 6B, in 634, the scaling coefficient α may be applied to the RMS normalized layer. For example, the value projection weights may be divided by the scaling coefficient α. The weight scaling can then be propagated through the network so that the output is also scaled as a result of the scaling in 634 (e.g., 636). Additionally, as described above, for dense layers with or without rectifier linear unit (ReLU) activation (e.g., fully connected layers), if the dense layer has no bias, the scaling coefficient α may be applied to the weights as specified by Equation 1. On the other hand, if the dense layer has a bias, the scaling coefficient α may be applied to the weights and bias as specified by Equation 3.

[0143]

[0103] Implementation examples are provided in the following numbered clauses. 1. A device, At least one memory, At least one processor coupled to at least one memory, It is equipped with at least one processor, An artificial neural network (ANN) model, which is trained to operate on a value in a first floating-point format having a bit length, takes an ANN model as input, Based on the layer type and topology of the ANN model, select the layers of the ANN model to which the scaling parameters will be applied. The maximum median value is determined according to the weight distribution in the selected layers of the ANN model. The scaling parameter is calculated based on the ratio of the maximum median value to the maximum representable value. Apply scaling parameters to the weights of the selected layers of the ANN model. A device configured in such a way. 2. The apparatus of Clause 1, wherein the first floating-point format is the 16-bit brain floating-point (bfloat16) format. 3. The apparatus of clause 1 or 2, wherein at least one processor is further configured to apply a scaling parameter to the dense layer, regardless of whether the bias value of the dense layer is zero or non-zero. 4. At least one processor is further configured to apply a scaling operator between layers of the ANN model, wherein the scaling operator performs element-wise multiplication of a scaling parameter with the activation output from a preceding layer, according to any device of clauses 1 to 3. 5. Any apparatus of any of the above clauses 1 to 4, wherein at least one processor is further configured to apply a scaling parameter to a layer having a nonlinear activation function preceding a linear or convolutional layer. 6. Any apparatus according to any of the clauses 1 to 5, wherein at least one processor is further configured to apply the scaling parameters of the layers preceding the normalization layer inversely to the normalization layer. 7. Any apparatus of clauses 1 to 6, wherein at least one processor is further configured to determine the maximum intermediate value based on the longest basis of matrix multiplication of a set of layer norm weights and a set of intermediate activation values. 8. A processor-implemented method that is executed by at least one processor, The system accepts an artificial neural network (ANN) model as input, which is trained to operate on values ​​in a first floating-point format having a bit length, Based on the layer type and topology of the ANN model, select the layers of the ANN model to which scaling parameters will be applied, Determining the maximum median value according to the weight distribution in the selected layers of the ANN model, Calculating the scaling parameter based on the ratio of the maximum median value to the maximum representable value, Applying scaling parameters to the weights of selected layers in the ANN model, A method implemented by the processor, including the following. 9. How the processor of Clause 8 implements the first floating-point format being the 16-bit brain floating-point (bfloat16) format. 10. A method implemented in the processor of Clause 8 or 9, wherein the scaling parameter is applied to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero. 11. A method implemented in any of the processors of Clauses 8 to 10, further comprising applying a scaling operator between layers of an ANN model, wherein the scaling operator performs element-wise multiplication of the scaling parameter with the activation output from the preceding layer. 12. A method implemented in a processor according to any of the clauses 8 to 11, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer. 13. A method implemented in any of the processors described in clauses 8 through 12, in which the scaling parameters of the layers preceding the normalization layer are inversely applied to the normalization layer. 14. A method to be implemented in any of the processors in clauses 8 to 13, wherein the maximum intermediate value is determined based on the longest basis of the matrix multiplication of the set of sheath norm weights and the set of intermediate activation values. 15. A non-temporary computer-readable medium on which program code is recorded, wherein the program code is executed by a processor, Program code for accepting an artificial neural network (ANN) model as input, which is trained to operate on a first floating-point format value having a bit length, Program code for selecting the layers of an ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. Program code for determining the maximum median value according to the weight distribution in a selected layer of an ANN model, Program code for calculating scaling parameters based on the ratio of the maximum median value to the maximum representable value, Program code for applying scaling parameters to the weights of selected layers in an ANN model, Non-temporary computer-readable media, including [specific examples of such media]. 16. Non-temporary computer-readable media of Clause 15, wherein the first floating-point format is the 16-bit brain floating-point (bfloat16) format. 17. A non-temporary computer-readable medium of Clause 15 or 16, which includes program code for applying scaling parameters to a dense layer, regardless of whether the bias value of the dense layer is zero or non-zero. 18. A non-temporary computer-readable medium of any of clauses 15 to 17, in which the program code includes program code for applying a scaling operator between layers of an ANN model, wherein the scaling operator performs element-wise multiplication of a scaling parameter with the activation output from a preceding layer. 19. A non-transient computer-readable medium of any of clauses 15 to 18, in which the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer. 20. A non-temporary computer-readable medium of any of clauses 15 to 19, in which the program code includes program code for applying the scaling parameters of a preceding layer to the normalization layer inversely. 21. A non-temporary computer-readable medium of any of Clauses 15 to 20, which includes program code for determining the maximum intermediate value based on the longest basis of matrix multiplication of a set of sheath norm weights and a set of intermediate activation values. 22. Apparatus, An artificial neural network (ANN) model, trained to operate on a first floating-point format value having a bit length, and means for receiving the ANN model as input, A means for selecting the layers of an ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. A means for determining the maximum median value according to the weight distribution in a selected layer of an ANN model, A means for calculating a scaling parameter based on the ratio of the maximum median value to the maximum representable value, A means of applying scaling parameters to the weights of selected layers of an ANN model, A device equipped with the following features. 23. The apparatus of Clause 22, wherein the first floating-point format is the 16-bit brain floating-point (bfloat16) format. 24. The apparatus of clause 22 or 23, further comprising means for applying a scaling parameter to a dense layer, regardless of whether the bias value of the dense layer is zero or non-zero. 25. The apparatus further comprises means for applying a scaling operator between layers of an ANN model, wherein the scaling operator performs element-wise multiplication of a scaling parameter with the activation output from a preceding layer, according to any of the provisions of 22 to 24. 26. An apparatus according to any of the clauses 22 to 25, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer. 27. Any apparatus of any of the clauses 22 to 26, further comprising means for inversely applying the scaling parameters of a layer preceding the normalization layer to the normalization layer. 28. An apparatus of any of the clauses 22 to 27, further comprising means for determining the maximum intermediate value based on the longest basis of matrix multiplication of a set of sheath norm weights and a set of intermediate activation values.

[0144]

[0104] In one embodiment, the receiving means, selecting means, and / or applying means may be a CPU 102 GPU 104, a program memory associated with the CPU 102 or GPU 104, a fully connected layer 362, an NPU 428, and / or a routing connection processing unit 216, all configured to perform the functions described. In another configuration, the means described above may be any module or any device configured to perform the functions enumerated by the means described above.

[0145]

[0105] Various operations of the above-described method may be performed by any preferred means capable of performing the corresponding function. The means may include, but are not limited to, various hardware and / or software components and / or modules, including, circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding relative means-plus-function components of similar numbering.

[0146]

[0106] When used, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. Additionally, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and furthermore, “determining” may include resolving, selecting, choosing, and establishing.

[0147]

[0107] When used, the phrase referring to “at least one of” an enumeration of items refers to any combination of those items that includes a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc.

[0148]

[0108] Various exemplary logic blocks, modules, and circuits described in this disclosure may be implemented or run using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices (PLDs), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0149]

[0109] Steps of the methods or algorithms described in this disclosure may be embodied in hardware directly, in software modules executed by a processor, or in a combination of the two. Software modules may reside in any form of storage medium known in the art. Some examples of storage mediums that may be used include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. Software modules may consist of one or more instructions and may be distributed across several different code segments, between different programs, and across multiple storage mediums. Storage mediums may be coupled to a processor so that the processor can read information from and write information to the storage medium. Alternatively, storage mediums may be integrated with the processor.

[0150]

[0110] The disclosed methods include one or more steps or actions for achieving the described method. The steps and / or actions of those methods may be interchanged with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, may be modified without departing from the claims.

[0151]

[0111] The functions described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the exemplary hardware configuration may include a processing system within the device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application of the processing system and the overall design constraints. The bus may link various circuits to each other, including processors, machine-readable media, and bus interfaces. The bus interface may, among other things, be used to connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In some embodiments, a user interface (e.g., a keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, but these circuits are well known in the art and are therefore not described further.

[0152]

[0112] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or dedicated processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software is broadly interpreted to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. Machine-readable medium may include, for example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0153]

[0113] In hardware implementations, machine-readable media may be part of a processing system separate from the processor. However, as will be readily apparent to those skilled in the art, machine-readable media or any part thereof may be outside the processing system. For example, machine-readable media may include transmission lines, data-modulated carriers, and / or computer products separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively, or additionally, machine-readable media or any part thereof may be integrated into the processor, as may be the case with caches and / or general-purpose register files. The various components discussed may be described as having a specific location, such as local components, but they may also be configured in various ways, such as several components configured as part of a distributed computing system.

[0154]

[0114] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functions and external memory providing at least a portion of a machine-readable medium, all linked together with other auxiliary circuits through an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphic processors for implementing the described neuron model and neural system model. Another alternative is that the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, auxiliary circuits, and at least a portion of a machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the functions described for the processing system depending on the specific application and the overall design constraints imposed on the entire system.

[0155]

[0115] A machine-readable medium may comprise several software modules. A software module contains instructions that, when executed by a processor, cause the processing system to perform various functions. A software module may include a transmit module and a receive module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. While a software module is being executed, the processor may load several instructions into a cache to speed up access. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to the functions of a software module below, it will be understood that such functions are implemented by the processor when executing instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of processors, computers, machines, or other systems that implement such aspects.

[0156]

[0116] When implemented in software, functions may be stored on or transmitted via computer-readable media as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. Storage media may be any available media that can be accessed by a computer. Such computer-readable media may include, but not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. When used, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc, where disk typically reproduces data magnetically and disc optically using a laser. Thus, in some embodiments, computer-readable medium may include non-temporary computer-readable medium (e.g., tangible medium). In addition, in other embodiments, the computer-readable medium may include a temporary computer-readable medium (e.g., a signal). The above combinations should also be included within the scope of the computer-readable medium.

[0157]

[0117] Accordingly, some embodiments may comprise a computer program product that performs the operations described. For example, such a computer program product may comprise a computer-readable medium storing (and / or encoding) instructions, the instructions being executable by one or more processors to perform the operations described. In some embodiments, the computer program product may comprise packaging material.

[0158]

[0118] Furthermore, it should be understood that modules and / or other suitable means for performing the described methods and techniques may be downloaded and / or acquired by user terminals and / or base stations as appropriate. For example, such devices may be coupled to a server to facilitate the transfer of means for performing the described methods. Alternatively, the various described methods may be provided via storage means so that user terminals and / or base stations can acquire the various methods by coupling or providing storage means (e.g., physical storage media such as RAM, ROM, compact disks (CDs), or floppy disks) to their devices. Moreover, any other suitable techniques for providing the described methods and techniques to a device may be utilized.

[0159]

[0119] It should be understood that the claims are not limited to the exact configurations and components illustrated above. Various modifications, changes, and variations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the claims.

Claims

1. It is a device, At least one memory, At least one processor coupled to the at least one memory, The at least one processor is An artificial neural network (ANN) model, which is trained to operate on a first floating-point format value having a bit length, takes an ANN model as input, Based on the layer type and topology of the ANN model, select the layers of the ANN model to which the scaling parameters will be applied. The maximum median value is determined according to the weight distribution in the selected layer of the ANN model. The scaling parameter is calculated based on the ratio of the maximum intermediate value to the maximum representable value. Apply the scaling parameter to the weights of the selected layer of the ANN model. A device configured in such a way.

2. The apparatus according to claim 1, wherein the first floating-point format is a 16-bit brain floating-point (bfloat16) format.

3. The apparatus according to claim 1, wherein the at least one processor is further configured to apply the scaling parameter to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

4. The apparatus according to claim 1, wherein the at least one processor is further configured to apply a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter by the activation output from a preceding layer.

5. The apparatus according to claim 1, wherein the at least one processor is further configured to apply the scaling parameter to a layer having a nonlinear activation function preceding a linear layer or a convolutional layer.

6. The apparatus according to claim 1, wherein the at least one processor is further configured to apply the scaling parameters of a layer preceding the normalization layer inversely to the normalization layer.

7. The apparatus according to claim 1, wherein the at least one processor is further configured to determine the maximum intermediate value based on the longest basis of a matrix multiplication of a set of layer norm weights and a set of intermediate activation values.

8. A processor-implemented method that is executed by at least one processor, The system accepts an artificial neural network (ANN) model as input, which is trained to operate on a first floating-point format value having a bit length, Based on the layer type and topology of the ANN model, select the layers of the ANN model to which the scaling parameters will be applied. The maximum median value is determined according to the weight distribution in the selected layer of the ANN model, The scaling parameter is calculated based on the ratio of the maximum intermediate value to the maximum representable value, Applying the scaling parameter to the weights of the selected layer of the ANN model, A method implemented by the processor, including the following.

9. A method implemented in a processor according to claim 8, wherein the first floating-point format is a 16-bit brain floating-point (bfloat16) format.

10. A method implemented in a processor according to claim 8, wherein the scaling parameter is applied to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

11. A method implemented in a processor according to claim 8, further comprising applying a scaling operator between layers of the ANN model, wherein the scaling operator performs element-wise multiplication of the scaling parameter by the activation output from a preceding layer.

12. The method implemented in the processor according to claim 8, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear layer or a convolutional layer.

13. A method implemented in a processor according to claim 8, wherein the scaling parameters of a layer preceding the normalization layer are inversely applied to the normalization layer.

14. The method implemented in the processor according to claim 8, wherein the maximum intermediate value is determined based on the longest basis of the matrix multiplication of a set of sheath norm weights and a set of intermediate activation values.

15. It is a device, An artificial neural network (ANN) model, which is trained to operate on a first floating-point format value having a bit length, and means for receiving the ANN model as input, A means for selecting the layers of the ANN model to which scaling parameters will be applied, based on the layer type and topology of the ANN model. A means for determining the maximum median value according to the weight distribution in the selected layer of the ANN model, A means for calculating the scaling parameter based on the ratio of the maximum intermediate value to the maximum representable value, means for applying the scaling parameter to the weights of the selected layer of the ANN model, A device equipped with the following features.

16. The apparatus according to claim 15, wherein the first floating-point format is a 16-bit brain floating-point (bfloat16) format.

17. The apparatus according to claim 15, further comprising means for applying the scaling parameter to the dense layer, regardless of whether the bias value of the dense layer is zero or non-zero.

18. The apparatus according to claim 15, further comprising means for applying a scaling operator between layers of the ANN model, wherein the scaling operator performs element-wise multiplication of the scaling parameter and the activation output from the preceding layer.

19. The apparatus according to claim 15, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear layer or a convolutional layer.

20. The apparatus according to claim 15, further comprising means for inversely applying the scaling parameters of a layer preceding the normalization layer to the normalization layer.

Images