Method and apparatus for converting pre-training model data format without retraining

Abstract

A processor-implemented method for converting a pre-trained model from a first floating point format to a second floating point format includes receiving an artificial neural network (ANN) model as an input. The ANN model is trained to operate on values in a first floating point format. A layer of the ANN model in which scaling parameters are to be applied is selected based on a layer type and topology of the ANN model. A maximum intermediate value is determined from a weight distribution in the selected layer of the ANN model. A scaling parameter is calculated based on a ratio of the maximum intermediate value and the maximum representable value. The scaling parameter is applied to the weights of the selected layers of the ANN model.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Indian Patent Application No. 202441010214, filed February 14, 2024, entitled “METHODS AND APPARATUS FOR CONVERTING PRE-TRAINED MODEL DATA FORMAT WITHOUT RETRAINING”, and PCT Application No. PCT / US2023 / 034975, filed October 11, 2023, entitled “METHODS AND APPARATUS FOR CONVERTING PRE-TRAINED MODEL DATA FORMAT WITHOUT RETRAINING”, the latter claiming the benefit of PCT Application No. PCT / US2023 / 034975, filed July 14, 2023, entitled “CONVERTING PRE-TRAINED MODEL DATA FORMAT WITHOUT RETRAINING”. The benefit of Indian patent application No. 202341047622, ​​entitled “RETRAINING (converting the format of pre-trained model data without retraining),” the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] The aspects of this disclosure generally relate to data processing, and more specifically to floating-point data in machine learning models. More particularly, the aspects relate to converting a pre-trained model from floating-point data operations in a first format to floating-point data operations in a second format without retraining the model. Background Technology

[0004] Artificial neural networks can comprise interconnected groups of artificial neurons (e.g., neuron models). Artificial neural networks can be computing devices or represented as methods to be performed by computing devices. Convolutional neural networks (CNNs) are a type of feedforward artificial neural network. A CNN can comprise an ensemble of neurons, where each neuron has a receptive field and collectively constructs the input space. CNNs, such as deep convolutional neural networks (DCNs), have numerous applications. Specifically, these neural network architectures are used in various technologies such as image recognition, speech recognition, acoustic scene classification, keyword retrieval, autonomous driving, and other classification tasks.

[0005] The use of floating-point values ​​allows for computations with sufficient precision in many neural network applications. This is especially true for applications such as autonomous driving. Representing floating-point values ​​in computations is very challenging given device size and memory constraints. Floating-point values ​​are typically represented using scientific or engineering notation. Given a location format (e.g., 16-bit or 32-bit), the number of bits can be divided to represent different parts of a floating-point value in scientific notation. For example, one bit can represent the sign of the floating-point value. One set of bits can represent the mantissa (also known as the significant bits), while another set can represent the exponent.

[0006] In computing, half-precision is a format used to represent floating-point values. Half-precision can be described as a floating-point bit format that uses sixteen bits in computer memory. However, half-precision formats offer lower precision. The industry standard for half-precision formats is the Institute of Electrical and Electronics Engineers (IEEE) sixteen-bit floating-point (FP16) format. The IEEE FP16 format consumes less memory and enables faster computations than higher-precision formats such as single-precision (thirty-two-bit format) or double-precision (sixty-four-bit format), but offers lower precision and a smaller range.

[0007] To achieve greater accuracy in neural network applications, some conventional methods have attempted to use non-standard FP16 formats (e.g., formats other than IEEE FP16). However, when pre-trained models are designed to have such non-standard FP16 formats, leveraging inference accelerators that are typically designed to support standard data formats can be challenging, and in some cases, impossible. Summary of the Invention

[0008] This disclosure is set forth in the independent claims. Some aspects of this disclosure are described in the dependent claims.

[0009] In some aspects of this disclosure, a processor-implemented method includes receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having a bit length. The processor-implemented method further includes selecting layers of the ANN model to which scaling parameters are to be applied based on the layer type and topology of the ANN model. The processor-implemented method further includes determining a maximum median value based on the weight distribution in the selected layers. The processor-implemented method additionally includes calculating the scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the processor-implemented method includes applying the scaling parameters to the weights of the selected layers of the ANN model.

[0010] Various aspects of this disclosure relate to an apparatus including components for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having a bit length. The apparatus also includes components for selecting layers of the ANN model to which scaling parameters are to be applied, based on the layer type and topology of the ANN model. The apparatus further includes components for determining the maximum median value based on the weight distribution in the selected layers. The apparatus additionally includes components for calculating the scaling parameters based on the ratio of the maximum median value to the maximum representable value. Furthermore, the apparatus includes components for applying the scaling parameters to the weights of the selected layers of the ANN model.

[0011] In some aspects of this disclosure, a non-transitory computer-readable medium having program code recorded thereon is disclosed. This program code is executed by a processor and includes program code for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having a bit length. The program code also includes program code for selecting layers of the ANN model to which scaling parameters are to be applied, based on the layer type and topology of the ANN model. The program code also includes program code for determining the maximum intermediate value based on the weight distribution in the selected layers. The program code additionally includes program code for calculating the scaling parameters based on the ratio of the maximum intermediate value to the maximum representable value. The program code also includes program code for applying the scaling parameters to the weights of the selected layers of the ANN model.

[0012] Various aspects of this disclosure relate to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processors are configured to receive an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having a bit length. The processors are also configured to select layers of the ANN model to which scaling parameters are to be applied, based on the layer type and topology of the ANN model. The processors are additionally configured to determine a maximum intermediate value based on the weight distribution in the selected layers. The processors are further configured to compute scaling parameters based on the ratio of the maximum intermediate value to the maximum representable value. Furthermore, the processors are configured to apply the scaling parameters to the weights of the selected layers of the ANN model.

[0013] Additional features and advantages of this disclosure will be described below. Those skilled in the art will understand that this disclosure can be readily used as the basis for modifying or designing other structures for implementing the same purposes as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of this disclosure as set forth in the appended claims. Novel features considered characteristic of this disclosure, in both their organization and manner of operation, along with further objects and advantages, will be better understood when considered in conjunction with the accompanying drawings. However, it is to be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to be a definition of a limitation of this disclosure. Attached Figure Description

[0014] The features, substance, and advantages of this disclosure will become more apparent when understood in conjunction with the accompanying drawings, in which the same reference numerals are consistently used for identification.

[0015] Figure 1 Examples of specific implementations of neural networks using a system-on-a-chip (SoC) (including a general-purpose processor) according to certain aspects of this disclosure are illustrated.

[0016] Figure 2A , Figure 2B and Figure 2C These are illustrations of neural networks according to various aspects of this disclosure.

[0017] Figure 2D This is a diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0018] Figure 3 This is a block diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0019] Figure 4 This is a block diagram illustrating exemplary software architectures that enable modularization of artificial intelligence (AI) functions according to various aspects of this disclosure.

[0020] Figure 5 This is a diagram illustrating example pseudocode for determining scaling factors according to various aspects of this disclosure.

[0021] Figure 6A and Figure 6B This is a block diagram illustrating example artificial neural network (ANN) models according to various aspects of this disclosure and the corresponding conversion from 16-bit brain floating-point (bfloat16) format to IEEE 16-bit floating-point (FP16) format using a scaling factor.

[0022] Figure 7This is a flowchart illustrating an example processor implementation of a method for converting a pre-trained model from bfloat16 format to FP16 format using varying scaling factors, according to various aspects of this disclosure. Detailed Implementation

[0023] The detailed description following, taken in conjunction with the accompanying drawings, is intended as a description of various configurations and not as representing only configurations in which the described concepts can be practiced. To provide a comprehensive understanding of the various concepts, the detailed description includes specific details. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0024] Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects described. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the various aspects of this disclosure described. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0025] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” need not be construed as superior to or better than other aspects.

[0026] While specific aspects have been described, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or purpose. Rather, aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0027] The brain-based floating-point format is a custom 16-bit floating-point format (also known as bfloat16 or BF16). The bfloat16 format includes one sign bit, eight exponent bits, and seven mantissa bits. This differs from the industry-standard Institute of Electrical and Electronics Engineers (IEEE) 16-bit floating-point (FP16) format, which was not designed for deep learning applications. The IEEE FP16 format includes one sign bit, five exponent bits, and ten mantissa bits.

[0028] Bfloat16 is a truncated version of IEEE 32-bit single-precision floating-point, developed by Google Brain by reducing the fractional part (e.g., mantissa) to seven bits while keeping the exponent at eight bits. Bfloat16 aims to reduce training time and the overall size of the model. However, due to the smaller number of bits in the fractional part (e.g., mantissa), models tend to use the exponent bits to represent weights. Therefore, a small change in the exponent bits can lead to excessively high absolute values ​​of the weights. That is, a model with a high exponent and therefore high values ​​may not be suitable for hardware (e.g., memory) that supports IEEE FP16, resulting in data overflow, which can reduce model accuracy. Furthermore, even if the weights are within the IEEE FP16 range, intermediate activation values ​​may exceed the IEEE FP16 range, again due to the high weight values.

[0029] Some conventional methods attempt to scale down all weights with predefined hard-decoded values ​​to fit the IEEE FP16 range. However, such methods rely on empirical selection of scaling factors until the solution works, and may fail for different inputs that have not been seen before, or for other datasets. Additionally, these conventional methods may only work on a few architectures and may struggle to handle nonlinearities other than modified linear units.

[0030] Therefore, to address these and other challenges, aspects of this disclosure relate to converting pre-trained models from bfloat16 format to IEEE FP16 format. For dense layers with non-scale-invariant activations or non-zero biases, weight scaling can be applied to architectures including pre-activations, or by applying a scaling factor to the non-zero bias.

[0031] Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. In some examples, the described techniques can advantageously convert pre-trained models from bfloat16 format to IEEE FP16 format without retraining. Therefore, aspects of this disclosure avoid the arduous and time-consuming task of retraining artificial neural network (ANN) models. Additionally, the described techniques can produce converted models that retain the accuracy of the original ANN model (e.g., increased precision) while enabling the use of conventional accelerators. Furthermore, because the disclosed techniques enable ANN models to operate with a smaller range of values, inference time can also be reduced.

[0032] Furthermore, text-based video editing can enhance camera features. For example, aspects of this disclosure enable content editing of videos captured using a camera. For instance, captured videos can be edited before sharing. Additionally, user personalization can be performed to inject personal effects into the video.

[0033] Some aspects and techniques described herein can be implemented, at least in part, using artificial intelligence (AI) programs, such as those including machine learning (ML) or artificial neural network (ANN) models. Example ML models may include mathematical representations or definitions of computational capabilities for inferring from input data based on patterns or relationships identified in the input data. As used herein, the term "inference" may include one or more of decision, prediction, determination, or value, which may represent the output of the ML model. Computational capabilities may be defined based on certain parameters of the ML model, such as weights and biases. Weights may indicate a relationship between certain input data and certain outputs of the ML model, and biases may represent an offset that may indicate the starting point of the ML model's output. An example ML model operating on input data may begin with an initial output based on the biases and then update its output based on a combination of input data and weights.

[0034] ML models can be characterized by a learning type that generates a specific type of learning model that performs a particular type of task. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and so on. ML models can be used to perform various tasks, such as classification or regression, where classification refers to determining one or more discrete output values ​​from a predefined set of output values, and regression refers to determining continuous values ​​that are not constrained by predefined output values. Some example ML models configured to perform such tasks include ANNs, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), transformers, diffusion models, regression analysis models (such as statistical models), large language models (LLMs), decision tree learning (such as predictive models), support vector networks (SVMs), and probabilistic graphical models (such as Bayesian networks), etc.

[0035] Figure 1 An example implementation of a System-on-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU, configured to convert a pre-trained model from floating-point data operations in a first format to floating-point data operations in a second format without retraining the model. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a weighted neural network), latency, frequency window (bin) information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or may be loaded from memory block 118.

[0036] SOC 100 may also include additional processing blocks tailored for specific functions, such as GPU 104, DSP 106, connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one specific implementation, NPU 108 is implemented within CPU 102, DSP 106, and / or GPU 104. SOC 100 may also include sensor processor 114, image signal processor (ISP) 116, and / or navigation module 120, which may include a global positioning system.

[0037] The SOC 100 may be based on the ARM instruction set. In one aspect of this disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in a first floating-point format having a bit length. The instructions loaded into the general-purpose processor 102 may also include code for selecting layers of the ANN model to which scaling parameters are to be applied, based on the layer type and topology of the ANN model. The instructions loaded into the general-purpose processor 102 may also include code for determining the maximum intermediate value based on the weight distribution in the selected layers. The instructions loaded into the general-purpose processor 102 may additionally include code for calculating the scaling parameters based on the ratio of the maximum intermediate value to the maximum representable value. Furthermore, the instructions loaded into the general-purpose processor 102 may include code for applying the scaling parameters to the weights of the selected layers of the ANN model.

[0038] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0039] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0040] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0041] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0042] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is shown. In the fully connected neural network 202, neurons in the first layer can pass their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2BAn example of a locally connected neural network 204 is shown. In the locally connected neural network 204, neurons in the first layer can connect to a finite number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0043] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is shown. Convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0044] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as an in-vehicle camera) is provided. The DCN 200 in this example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0045] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and forward passes can then be computed to produce output 222. The DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

[0046] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0047] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 can be the probability that image 226 includes one or more features.

[0048] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0049] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0050] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, DCN 200 can present a new picture, and the forward pass of DCN 200 can produce an output that can be considered an inference or prediction of DCN 200.

[0051] During training, optimization algorithms can be used to adjust weights and biases as needed to reduce or minimize the loss function, which improves model performance. Various optimization algorithms exist that can be used with backpropagation or other training techniques. Some initial examples include gradient descent-based optimization algorithms and stochastic gradient descent-based optimization algorithms. Stochastic gradient descent can be used to adjust weights / biases to minimize or otherwise reduce the loss function. Mini-batch gradient descent, a variant of gradient descent, involves updating weights / biases using mini-batch training data instead of the entire dataset. Momentum techniques can accelerate the optimization process by adding momentum terms to update or otherwise influence certain weights / biases.

[0052] Adaptive learning rate techniques adjust the learning rate of an optimization algorithm that is associated with one or more characteristics of the training data. Batch normalization can be used to normalize the input to a model in order to stabilize the training process and potentially improve the model's performance. "Dropout" techniques can be used to randomly drop some artificial neurons from the model during training, for example, to reduce overfitting and potentially improve the model's generalization. "Early stopping" techniques can be used to stop the ongoing training process early, such as when the performance of a model using a validation dataset begins to degrade.

[0053] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0054] DCN is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0055] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0056] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the convolutional connections can be viewed as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as correction, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0057] Figure 3 This is a block diagram illustrating a Deep Convolutional Network (DCN) 350. A DCN 350 can include multiple layers of different types based on connectivity and weight sharing. Figure 3 As shown, DCN 350 includes convolutional blocks 354A and 354B. Each convolutional block in convolutional blocks 354A and 354B can be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0058] Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks 354A and 354B may be included in the DCN 350 according to design preferences.

[0059] Convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate feature maps. Normalization layer 358 can normalize the output of the convolutional filters. For example, normalization layer 358 can provide whitening or lateral suppression. Max pooling layer 360 can provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0060] Parallel filter banks of deep convolutional networks can be loaded onto an SOC 100 (e.g., Figure 1 The CPU 102 or GPU 104 of the SOC 100 can be used to achieve high performance and low power consumption. In an alternative implementation, a parallel filter bank can be loaded onto the DSP 106 or ISP 116 of the SOC 100. Additionally, the DCN 350 can access other processing blocks that may exist on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0061] The DCN 350 may also include one or more fully connected layers 362 (FC1 and FC2). The DCN 350 may also include logistic regression (LR) layers 364. Weights (not shown) to be updated are located between each of the layers 356, 358, 360, 362, and 364 of the DCN 350. The output of each layer (e.g., 356, 358, 360, 362, and 364) can be used as input to the next layer in the DCN 350 to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in convolutional block 354A. The output of the DCN 350 is a classification score 366 of the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data includes a feature from the feature set.

[0062] Figure 4 This is a block diagram illustrating an exemplary software architecture 400 that allows for modularization of artificial intelligence (AI) functionality. According to various aspects of this disclosure, architecture 400 can be used to design various processing blocks of an SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) to support floating-point format conversion for AI applications 402. Architecture 400 can, for example, be included in a computing device such as a smartphone.

[0063] AI application 402 can be configured to invoke functions defined in user space 404, which may, for example, provide the detection and recognition of a scene indicating the current location of the computing device (including architecture 400). For example, AI application 402 may configure microphones and cameras differently depending on whether the recognized scene is an office, lecture hall, restaurant, or outdoor environment such as a lake. AI application 402 may make requests to compiled program code associated with libraries defined in the AI ​​Function Application Programming Interface (API) 406. This request may ultimately rely on the output of a deep neural network configured to provide inferred responses based on, for example, video and location data.

[0064] The runtime engine 408 (which may be compiled code of the runtime framework) may further be accessible to the AI ​​application 402. The AI ​​application 402 may cause the runtime engine 408 to request inferences, for example, at specific time intervals or triggered by events detected by the user interface of the AI ​​application 402. Upon causing the runtime engine 408 to provide an inference response, the runtime engine may then signal to the operating system (OS) space 410 running on the SOC 420, such as kernel 412. In some examples, kernel 412 may be a LINUX kernel. The operating system may then enable sequential quantization relaxation to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be directly accessible by the operating system, while other processing blocks may be accessed via drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to run on a combination of processing blocks such as CPU 422, DSP 424 and GPU 426, or on NPU 428.

[0065] As described, aspects of this disclosure relate to converting a pre-trained model from floating-point data operations in a first format to floating-point data operations in a second format without retraining the model. For example, the first format may be bfloat16 format, and the second format may be the Institute of Electrical and Electronics Engineers (IEEE) FP16 format.

[0066] In various ways, the ANN architecture and weights can be mathematically modeled to identify sensitive layers or blocks (e.g., blocks that may be more susceptible to overflow errors). For example, mathematical models can be applied to determine certain scalable blocks (or layers) of the ANN and those blocks that can be retained without scaling. Different scaling factors can be computed for different blocks and weights. In some aspects, worst-case analysis can be employed to further reduce overflow errors. For example, the worst-case range of scaling factors can be determined without using a dataset.

[0067] Figure 5 This is a diagram illustrating example pseudocode 500 for determining a scaling factor according to various aspects of this disclosure. Reference Figure 5 Example pseudocode 500 determines scaling factors for parameters (e.g., weights) in various blocks (e.g., self-attention blocks or cross-attention blocks) of an ANN model. Instead of determining a uniform scaling factor (e.g., a fixed scaling factor) applied to multiple blocks (or layers) of the ANN, different scaling factors can be determined for different blocks (or layers) of the ANN. For example, different scaling factors can be applied to layers that include (but are not limited to) dense layers, self-attention layers, or normalized layers. For example, in some aspects, an ANN model may include a convolutional neural network (CNN) (e.g., Figure 3 The diagram shows DCN 350, recurrent neural networks (RNN), transformer neural networks, or other neural networks.

[0068] Given the layer norm weights, value projection weights, output projection weights, the dimension of the ANN input, and the maximum representable value (e.g., for a specific hardware device such as a hardware accelerator), the scaling factor can be determined. As shown at line 502, intermediate values ​​can be computed. Median Can be determined by layer norm weights Sum projection weights Matrix multiplication of transpose ( Multiply by the square root of the input dimension ( The maximum value of the median can be determined at line 504. At line 506, the maximum output value can then be determined. Maximum output value () ) can be achieved by using intermediate values Multiply by the output projection weight The maximum value is calculated by transposing the value. At line 508, the scaling factor of the projection weight is used. This can be achieved by using the largest median value. Divide by the largest representable value To determine this. For example, for IEEE FP16, the maximum representable value could be 65004. Additionally, at line 510, the scaling factor for the projected weights is output. This can be achieved by using the maximum output value. Divide by the largest representable value The scaling factor is determined based on the weight distribution within the block, rather than on the dataset (e.g., a validation dataset). Because the maximum output value may differ for each block, the scaling factor may also differ. Therefore, aspects of this disclosure can determine the scaling factor based on the weight distribution within the block rather than on the dataset (e.g., a validation dataset). That is, the scaling factor ( It is not based on a specific value of the input, but on the weights of various layers (e.g., layer norm weights, value projection weights, output projection weights).

[0069] According to various aspects of this disclosure, weight scaling can be applied so that the weights of the ANN model have lower absolute values ​​while still maintaining the mathematical equivalence across blocks in the ANN model. Lower absolute values ​​of the weights can reduce overflow and, in some respects, can prevent intermediate activation values ​​(e.g., intermediate layer outputs) from being affected by IEEE FP16 overflow.

[0070] As described, different scaling strategies can be used for different architectures. The scaling factors for different layers of an ANN model can be mathematically derived using the weights of the corresponding layers. That is, different layers of an ANN model can be adapted in different ways based on mathematical properties. For example, for dense layers with Rectified Linear Unit (ReLU) activations or no activations (e.g., fully connected layers), the weights can be scaled as follows:

[0071] (1)

[0072] (2)

[0073] (3)

[0074] in It's weight. It is input. It is a bias, and It is the scaling factor.

[0075] If the dense layer has no bias, a scaling factor can be applied as shown in Equation 1. On the other hand, if the dense layer has a bias, a scaling factor can be applied as shown in Equation 3. That is, the bias can be scaled along with the weights, so that the output is scaled in a similar manner. In some respects, Equation 3 can only be applied if the activation function is linear on both sides of zero (negative and positive).

[0076] Although the activation in Equation 3 is a ReLU function, this disclosure is not limited thereto. Instead, if a nonlinear function is applied along with "pre-activation," a scaling factor can be applied even if ReLU is not the activation function in the ANN model. That is, a scaling factor can be applied when a nonlinear activation is included before a linear layer or a convolutional neural network (CNN) layer (e.g., the nonlinear activation precedes a linear layer or a CNN layer). For example, in a stable diffusion XL (SDXL) model, layers can be arranged in the following order: group normalization, sigmoid linear unit (SiLU) nonlinearity, and convolutional layers. The SiLU function can be expressed as... ,in It is a logistic sigmoid function. In this example, the weights and biases of the convolution can be scaled because the next block in the SDXL model has the same layer order. Group normalization is applied first, which resets the scaling factor. Therefore, the output of the current block remains unchanged.

[0077] When a block (or layer) has residual connections, a scaling factor can be applied to both the input and output so that the input and output are on the same scale and can be added together, as shown below:

[0078] (4)

[0079] Therefore, for layers with residual connections, the input can be scaled by scaling the weights of previous layers or by introducing scaling operators between nodes.

[0080] Additionally, if the ANN model has a normalization layer (e.g., LayerNorm or RMSNorm), the scaling of the activations can be reset. That is, the scaling factor applied in previous layers has no effect. This is because when a layer has weight scaling applied and is followed by a normalization layer (e.g., LayerNorm or RMSNorm), the output of the normalization layer is the same as the output of a normalized layer without scaling. Therefore, weight scaling can be applied to other layers (e.g., for residuals), or rescaling can be applied to the output by adapting the normalization layer, for example, as follows:

[0081] (5)

[0082] (6)

[0083] In some respects, scaling operators may be used alternatively or additionally. For example, element-wise multiplication nodes / operations may be added at points that have a scaling factor as a parameter.

[0084] (7)

[0085] in Represents the tensor product.

[0086] For example, scaling operators can be inserted between blocks or layers of an ANN model to unscale / rescale the scaling currently held in layer activations. For instance, when using RMSnorm and residual connections, scaling operators can rescale the input.

[0087] Additional analysis can be performed on the ANN model to determine the scaling factor, since each block has different weights. For example, using a single large scaling factor may reduce the accuracy of the ANN model because accelerators cannot represent scaling values.

[0088] In various ways, determining the scaling factor can involve accessing the extreme values ​​of the inputs (e.g., the minimum and maximum values), which are the outputs of the previous layer. Therefore, analysis of the extreme values ​​of all previous layers can be performed.

[0089] In the described analysis, when the model has normalized layers (e.g., LayerNorm or RMSNorm), only the extrema of the normalized tensor (e.g., the weight vector that has been normalized) are used, rather than the extrema of the previous layers.

[0090] To find the extrema of the result of a matrix multiplication (MatMul) operation, the input can be configured such that the input vector is parallel to the longest basis of the weight matrix (e.g., a column vector with the longest length). Due to the normalization layer, the length of the input vector used can be fixed. The maximum length before multiplication with the weights is at most [value missing]. And can be specified as follows:

[0091] (8)

[0092] (9)

[0093] in It is the i-th dimension of the input vector. It is the weight vector of LayerNorm. The i-th dimension, and It is the i-th dimension of the bias of LayerNorm.

[0094] The maximum absolute value can be calculated by analyzing the weights of the blocks in the ANN model. For example, the maximum absolute value can be calculated as follows:

[0095] (10)

[0096] (11)

[0097] (12)

[0098] in Represents linear operators, vectors The length may be one due to normalization. It is an attention matrix The transpose of, and It is the RMSNorm function defined in Equation 9. Then, it has The maximum absolute value of matrix multiplication can be the matrix The longest base.

[0099] The result of matrix multiplication will be the result of the dot product of the two vectors:

[0100] (13)

[0101] in The maximum absolute value is one. Then, the maximum possible result could be... The maximum value. As described, The maximum absolute value is Therefore, the maximum absolute value in the result of the MatMul operation following RMSNorm can be expressed as:

[0102] (14)

[0103] in It is a matrix The length of the longest basis.

[0104] After obtaining the maximum absolute value, the scaling factor of the linear layer weight matrix can be determined as follows:

[0105] (15)

[0106] Where ceil(x) is the upper bound function for calculating the smallest integer greater than or equal to x, and its maximum absolute value can be derived as follows: Furthermore, the maximum number of representations that IEEE FP16 can represent is, for example, 65504.

[0107] Therefore, different scaling factors can be determined for each block or layer of the ANN model. For example, when the ANN model includes a transformer model, the same transformer decoder block can be repeated multiple times in the model. However, because the maximum value may be different for each transformer decoder block within the transformer decoder block, the scaling factor for each such transformer decoder block may also be different, which differs from conventional methods.

[0108] After the first linear layer after layer normalization, there may be additional layers where IEEE FP16 overflow may occur. For example, for the attention module, the output can be expressed as:

[0109] (16)

[0110] (17)

[0111] in It is the normalized output from RMSNorm (in some respects, This can represent the normalized output of LayerNorm shown in Equation 8. It is the key projection matrix. It queries the projection matrix. It is a value projection matrix, and This is the output projection matrix. The final output projection matrix. MatMul may not be related to The MatMul combination is undergoing reshaping to target bullish attention. Additionally, The maximum value can be calculated, for example, as described in equation 10-14. To determine... The maximum value (e.g., the maximum possible output) can be determined. The maximum possible value for each column. Each column can have a maximum value, which is the GA. T The norm of the corresponding column of the matrix multiplied by :

[0112] (18)

[0113] (19)

[0114] The value of softmax can have a maximum value of one. Then, It is possible to project only the weight matrix The maximum possible number of rows is generated when all columns are identical (which is highly unlikely, but possible in the worst case):

[0115] (20)

[0116] In the example above, The first row of the results can be the maximum value. For that row, ( The maximum value of the result can be determined using the norm of the vector and the maximum norm of the output weight matrix O:

[0117] (twenty one)

[0118] Similarly, the maximum value can be determined as:

[0119] .(twenty two)

[0120]

[0121] Then, as described in Equation 15, the maximum value can be used to determine the scaling factor for the output projection matrix multiplication.

[0122] Therefore, by applying the described techniques for selectively scaling blocks or layers of an ANN model, an ANN model can be converted, for example, from bfloat16 to IEEE FP16 without retraining the model. Thus, in various respects, an ANN model can operate using hardware that supports IEEE FP16 instead of, for example, bfloat16 (e.g., hardware accelerators).

[0123] Figure 6A and Figure 6B This is a block diagram illustrating an example ANN model 600 according to various aspects of this disclosure and the corresponding transformation from bfloat16 to IEEE FP16 using a scaling factor. Reference Figure 6A It can accept 600 example ANN models as input. Figure 6A As shown in the non-limiting example, the example ANN model 600 may include a transformer neural network. For example, the transformer neural network may include transformer architectures such as (but not limited to) bidirectional encoder representations from transformers (BERT), robustly optimized BERT methods (RoBERTa), XLNet, Transformer-XL, and generative pre-trained transformer (GPT) families (e.g., GPT-2).

[0124] Example ANN model 600 may include multiple blocks 604a-z (collectively referred to as block 604). Each block in a block (e.g., 604a-z) may include a self-attention layer and a feedforward layer. In some aspects, blocks (e.g., 604a-z) may also include cross-attention blocks.

[0125] ANN model 600 can be trained to operate on values ​​in bfloat16 format. That is, weights of blocks (e.g., 604a-z) can be pre-trained in bfloat16 format to operate on received embeddings (e.g., 602a and 602b).

[0126] refer to Figure 6B An example block (e.g., 604b) is shown. The example block (e.g., 604b) includes a self-attention layer 622, a cross-attention layer 624, and a feedforward layer 626. Box 628 provides a set of example operations performed in each block (e.g., 604a-z) of the received ANN model 600, which is pre-trained according to the IEEE FP16 format (e.g., without scaling).

[0127] As shown in box 630, scaling factor Depending on the type of layer, it can be applied to selected blocks (e.g., 604b) or layers (e.g., 622) of the ANN model 600. However, the scaling factor... The value can be determined based on the topology and weight distribution in the corresponding layer, as described, for example, in Equations 1-22. For example, in box 630, compared to box 628, the ANN (e.g., 600) includes an RMS normalized layer (632). As shown at 634, the scaling factor... The scaling is applied to the value projection weights. Scaling propagates through the network, causing the output to also be scaled (636). In another example, at 638, the ANN (e.g., 600) may include a dense layer with ReLU activations. In this example, ReLU does not have a bias term. Therefore, the scaling parameter... It can be applied at position 640, where the weights Scaling factor Scaling, as specified in Equations 1 and 2. However, if the dense layer includes biases, a scaling factor can be applied to scale the weights (e.g., ) and biases, as specified in Equation 3. Therefore, different scaling factors can be applied to different layers of the ANN model 600 (e.g., 622).

[0128] In some respects, different scaling parameters can be applied to layers of the same type. Because scaling parameters can be based on the weight distribution of the layers, and even-numbered layers of the same type can have different weight distributions, different scaling parameter values ​​can be determined for each layer in such a layering system.

[0129] Figure 7 This is a flowchart illustrating an example processor implementation of method 700 for converting a pre-trained model from bfloat16 to IEEE FP16 using varying scaling factors, according to various aspects of this disclosure. For example, the processor-implemented method 700 may be executed by one or more processors, such as CPU 102, GPU 104, and / or NPU 108.

[0130] like Figure 7 As shown, at box 702, at least one processor receives an artificial neural network (ANN) model as input. The ANN model is trained to operate on values ​​in 16-bit brain floating-point (bfloat16) format. For example, as Figure 6A As shown, an example ANN model 600 can be received as input. For example, the example ANN model 600 may include a transformer neural network. For example, the transformer neural network may include a stable diffusion model (such as the SDXL model) or a transformer architecture (such as a bidirectional encoder representation from a transformer (BERT), a robustly optimized BERT method (RoBERTa), XLNet, Transformer-XL, and a generative pre-trained transformer (GPT) family (e.g., GPT-2)).

[0131] At box 704, at least one processor selects the layers of the ANN model to which scaling parameters are to be applied, based on the layer type and topology of the ANN model. For example, as Figure 6B As shown, layers in a block (e.g., 604b) can be selected to apply scaling factors based on layer type and topology. In one example, a scaling factor can be applied to a self-attention layer 622 according to box 630. For example, weight scaling can be applied when the layer includes an RMS normalized layer (e.g., 632).

[0132] At box 706, at least one processor determines the maximum median value based on the weight distribution in the selected layer of the ANN model. (See reference...) Figure 5 The described median value Can be determined by layer norm weights Sum projection weights Matrix multiplication of transpose ( Multiply by the square root of the input dimension ( The maximum value of the median can be determined at line 504. At line 506, the maximum output value can then be determined. Maximum output value () ) can be achieved by using intermediate values Multiply by the maximum value of the output projection weights The transpose is used for calculation.

[0133] At box 708, at least one processor calculates a scaling parameter based on the ratio of the maximum intermediate value to the maximum representable value. For example, the scaling factor, as defined in Equation 15. The largest intermediate value that can be determined as a layer of an ANN (derived as) The upper limit of the ratio of the largest representable number to the maximum number of numbers.

[0134] At box 710, at least one processor applies scaling parameters to the weights of the selected layers in the ANN model. For example, as referenced... Figure 6B As described, at 634, the scaling factor This can be applied to RMS normalization layers. For example, value projection weights can be divided by a scaling factor. Furthermore, weight scaling propagates through the network, causing the output to also be scaled due to scaling at 634 (e.g., 636). Additionally, as described, for dense layers with ReLU activations or no activations (e.g., fully connected layers), if the dense layer has no bias, a scaling factor α can be applied to the weights as specified in Equation 1. On the other hand, if the dense layer does have a bias, a scaling factor α can be applied to both the weights and the bias, as specified in Equation 3.

[0135] Specific implementation examples are provided in the following numbered clauses.

[0136] 1. An apparatus, the apparatus comprising:

[0137] At least one memory; and

[0138] At least one processor, coupled to the at least one memory, is configured to:

[0139] The system receives an artificial neural network (ANN) model as input, which is trained to operate on values ​​in a first floating-point format having a bit length.

[0140] The layer in the ANN model to which the scaling parameter is to be applied is selected based on the layer type and topology of the ANN model.

[0141] The maximum median value is determined based on the weight distribution in the selected layer of the ANN model;

[0142] The scaling parameter is calculated based on the ratio of the maximum median value to the maximum representable value; and

[0143] The scaling parameter is applied to the weights of the selected layers in the ANN model.

[0144] 2. The apparatus according to Clause 1, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

[0145] 3. The apparatus according to clause 1 or 2, wherein the at least one processor is further configured to apply the scaling parameter to the dense layer, regardless of whether the bias value of the dense layer is zero or non-zero.

[0146] 4. The apparatus according to any one of clauses 1 to 3, wherein the at least one processor is further configured to apply a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

[0147] 5. The apparatus according to any one of clauses 1 to 4, wherein the at least one processor is further configured to apply the scaling parameter to a layer having a nonlinear activation function preceding a linear or convolutional layer.

[0148] 6. The apparatus according to any one of clauses 1 to 5, wherein the at least one processor is further configured to apply the scaling parameters of the preceding layer of the normalization layer in reverse to the normalization layer.

[0149] 7. The apparatus according to any one of clauses 1 to 6, wherein the at least one processor is further configured to determine the maximum intermediate value based on the longest basis of matrix multiplication of the layer norm weight set and the intermediate activation value set.

[0150] 8. A processor-implemented method executed by at least one processor, the processor-implemented method comprising:

[0151] The system receives an artificial neural network (ANN) model as input, which is trained to operate on values ​​in a first floating-point format having a bit length.

[0152] The layer in the ANN model to which the scaling parameter is to be applied is selected based on the layer type and topology of the ANN model.

[0153] The maximum median value is determined based on the weight distribution in the selected layer of the ANN model;

[0154] The scaling parameter is calculated based on the ratio of the maximum median value to the maximum representable value; and

[0155] The scaling parameter is applied to the weights of the selected layers in the ANN model.

[0156] 9. The processor-implemented method according to Clause 8, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

[0157] 10. The processor implementation method according to clause 8 or 9, wherein the scaling parameter is applied to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

[0158] 11. The processor-implemented method according to any one of Clauses 8 to 10, the processor-implemented method further comprising applying a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

[0159] 12. The method implemented by the processor according to any one of clauses 8 to 11, wherein the scaling parameter is applied to a layer having a non-linear activation function preceding a linear or convolutional layer.

[0160] 13. The processor implementation of any one of Clauses 8 to 12, wherein the scaling parameters of the preceding layer of the normalization layer are applied in reverse to the normalization layer.

[0161] 14. The method implemented by the processor according to any one of Clauses 8 to 13, wherein the maximum intermediate value is determined based on the longest basis of matrix multiplication of the layer norm weight set and the intermediate activation value set.

[0162] 15. A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising:

[0163] Program code for receiving an artificial neural network (ANN) model as input, the ANN model being trained to operate on values ​​in a first floating-point format having a bit length;

[0164] Program code for selecting the layers of the ANN model to which scaling parameters are applied, based on the layer type and topology of the ANN model;

[0165] Program code for determining the maximum intermediate value based on the weight distribution in the selected layer of the ANN model;

[0166] Program code for calculating the scaling parameter based on the ratio of the maximum intermediate value to the maximum representable value; and

[0167] Program code for applying the scaling parameters to the weights of selected layers in the ANN model.

[0168] 16. The non-transitory computer-readable medium as described in Clause 15, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

[0169] 17. The non-transitory computer-readable medium according to Clause 15 or 16, wherein the program code includes program code for applying the scaling parameter to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

[0170] 18. A non-transitory computer-readable medium according to any one of Clauses 15 to 17, wherein the program code includes program code for applying a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

[0171] 19. A nontransitory computer-readable medium according to any one of clauses 15 to 18, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer.

[0172] 20. A non-transitory computer-readable medium according to any one of Clauses 15 to 19, wherein the program code includes program code for applying the scaling parameters of the preceding layer of the normalization layer in reverse to the normalization layer.

[0173] 21. The non-transitory computer-readable medium according to any one of Clauses 15 to 20, wherein the program code includes program code for determining the maximum intermediate value based on the longest basis of matrix multiplication of the set of layer norm weights and the set of intermediate activation values.

[0174] 22. An apparatus comprising:

[0175] A component for receiving an artificial neural network (ANN) model as input, the ANN model being trained to operate on values ​​in a first floating-point format having a bit length;

[0176] Components for selecting the layers of the ANN model to which scaling parameters are applied, based on the layer type and topology of the ANN model;

[0177] A component used to determine the maximum intermediate value based on the weight distribution in the selected layer of the ANN model;

[0178] A component for calculating the scaling parameter based on the ratio of the maximum intermediate value to the maximum representable value; and

[0179] A component for applying the scaling parameters to the weights of selected layers in the ANN model.

[0180] 23. The apparatus according to Clause 22, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

[0181] 24. The apparatus according to clause 22 or 23, the apparatus further comprising a component for applying the scaling parameter to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

[0182] 25. The apparatus according to any one of clauses 22 to 24, the apparatus further comprising means for applying a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

[0183] 26. The apparatus according to any one of clauses 22 to 25, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer.

[0184] 27. The apparatus according to any one of clauses 22 to 26, the apparatus further comprising means for applying the scaling parameters of the preceding layer of the normalization layer in reverse to the normalization layer.

[0185] 28. The apparatus according to any one of clauses 22 to 27, the apparatus further comprising means for determining the maximum intermediate value based on the longest basis of matrix multiplication of the set of layer norm weights and the set of intermediate activation values.

[0186] In one aspect, the receiving component, selection component, and / or application component may be a CPU 102, a GPU 104, a program memory associated with the CPU 102 or GPU 104, a fully connected layer 362, an NPU 428, and / or a routing connection processing unit 216 configured to perform the described functions. In another configuration, the aforementioned components may be any module or any device configured to perform the functions described by the aforementioned components.

[0187] The various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components with similar numbering plus functional components.

[0188] As used, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., searching in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, choosing, building, etc.

[0189] As used, the phrase "at least one of the items in the list" refers to any combination of these items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0190] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic component, discrete hardware component, or any combination thereof designed to perform the described functions. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0191] The steps or algorithms of the methods described in this disclosure may be directly embodied in hardware, a software module executed by a processor, or a combination of both. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction or multiple instructions and may be distributed across several different code segments, across different programs, and across multiple storage media. The storage medium may be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integral with the processor.

[0192] The disclosed method includes one or more steps or actions for implementing the described method. The steps and / or actions of the method may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0193] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. This processing system may utilize a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link various circuits together, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. In some respects, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0194] A processor may be responsible for managing the bus and general-purpose processing, including executing software stored on a machine-readable medium. A processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or others. By way of example, a machine-readable medium may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. A machine-readable medium may be embodied in a computer program product. A computer program product may include packaging material.

[0195] In a hardware implementation, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media, or any portion thereof, can be external to the processing system. By way of example, machine-readable media may include transmit lines, carrier waves modulated by data, and / or computer components separate from the device, all accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or a general-purpose register file. Although the various components discussed may be described as having a specific location, such as local components, they may also be configured in various ways, such as certain components being configured as part of a distributed computing system.

[0196] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functionality and external memory providing at least a portion of machine-readable medium, all of which are linked together with other supporting circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron and nervous system models. As another alternative, the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, bus interface, user interface, supporting circuitry, and at least a portion of machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality of the processing system depends on the specific application and the overall design constraints imposed on the system as a whole.

[0197] Machine-readable media may include multiple software modules. These software modules include instructions that, when executed by a processor, enable the processing system to perform various functions. Software modules may include send and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module may be loaded from a hard disk drive into RAM. During the execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a general-purpose register file for processor execution. When the functionality of a software module is referred to below, it will be understood that such functionality is implemented by the processor when executing the instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0198] If implemented in software, the functions may be stored as one or more instructions or codes on or transmitted through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. A storage medium can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and is accessible to a computer. Additionally, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then such coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. The disks and optical discs used include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs. ® Optical discs, where magnetic disks typically reproduce data magnetically, and optical discs reproduce data optically using lasers. Therefore, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0199] Therefore, some aspects may include a computer program product for performing the presented operations. For example, such a computer program product may include a computer-readable medium on which instructions are stored (and / or encoded) that can be executed by one or more processors to perform the described operations. In some aspects, the computer program product may include packaging material.

[0200] Furthermore, it should be understood that modules and / or other suitable components for performing the described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such devices can be coupled to a server to facilitate the delivery of components for performing the described methods. Alternatively, the various methods described can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that the user terminal and / or base station can obtain the various methods once the storage component is coupled to or provided to the device. In addition, any other suitable techniques suitable for providing the described methods and techniques to the device may be utilized.

[0201] It should be understood that the claims are not limited to the precise configurations and components illustrated above. Various modifications, variations, and alterations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. An apparatus, the apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The system receives an artificial neural network (ANN) model as input, which is trained to operate on values ​​in a first floating-point format having a bit length. The layer in the ANN model to which the scaling parameter is to be applied is selected based on the layer type and topology of the ANN model. The maximum median value is determined based on the weight distribution in the selected layer of the ANN model; The scaling parameter is calculated based on the ratio of the maximum intermediate value to the maximum representable value; as well as The scaling parameter is applied to the weights of the selected layers in the ANN model.

2. The apparatus of claim 1, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

3. The apparatus of claim 1, wherein the at least one processor is further configured to apply the scaling parameter to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

4. The apparatus of claim 1, wherein the at least one processor is further configured to apply a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

5. The apparatus of claim 1, wherein the at least one processor is further configured to apply the scaling parameter to a layer having a nonlinear activation function preceding a linear or convolutional layer.

6. The apparatus of claim 1, wherein the at least one processor is further configured to apply the scaling parameters of the preceding layer of the normalization layer in reverse to the normalization layer.

7. The apparatus of claim 1, wherein the at least one processor is further configured to determine the maximum intermediate value based on the longest basis of matrix multiplication of the layer norm weight set and the intermediate activation value set.

8. A processor-implemented method executed by at least one processor, the processor-implemented method comprising: The system receives an artificial neural network (ANN) model as input, which is trained to operate on values ​​in a first floating-point format having a bit length. The layer in the ANN model to which the scaling parameter is to be applied is selected based on the layer type and topology of the ANN model. The maximum median value is determined based on the weight distribution in the selected layer of the ANN model; The scaling parameter is calculated based on the ratio of the maximum intermediate value to the maximum representable value; as well as The scaling parameter is applied to the weights of the selected layers in the ANN model.

9. The processor-implemented method according to claim 8, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

10. The processor-implemented method of claim 8, wherein the scaling parameter is applied to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

11. The processor-implemented method of claim 8, further comprising applying a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

12. The processor-implemented method of claim 8, wherein the scaling parameter is applied to a layer having a non-linear activation function preceding a linear or convolutional layer.

13. The processor-implemented method of claim 8, wherein the scaling parameters of the layer preceding the normalization layer are applied in reverse to the normalization layer.

14. The processor-implemented method of claim 8, wherein the maximum intermediate value is determined by the longest basis of matrix multiplication of the layer norm weight set and the intermediate activation value set.

15. An apparatus comprising: A component for receiving an artificial neural network (ANN) model as input, the ANN model being trained to operate on values ​​in a first floating-point format having a bit length; Components for selecting the layers of the ANN model to which scaling parameters are applied, based on the layer type and topology of the ANN model; A component used to determine the maximum intermediate value based on the weight distribution in the selected layer of the ANN model; A component for calculating the scaling parameter based on the ratio of the maximum intermediate value to the maximum representable value; and A component for applying the scaling parameters to the weights of selected layers in the ANN model.

16. The apparatus of claim 15, wherein the first floating-point format is a 16-bit brain-float (bfloat16) format.

17. The apparatus of claim 15, further comprising a component for applying the scaling parameter to the dense layer regardless of whether the bias value of the dense layer is zero or non-zero.

18. The apparatus of claim 15, further comprising a component for applying a scaling operator between layers of the ANN model, the scaling operator performing element-wise multiplication of the scaling parameter and the activation output from the previous layer.

19. The apparatus of claim 15, wherein the scaling parameter is applied to a layer having a nonlinear activation function preceding a linear or convolutional layer.

20. The apparatus of claim 15, further comprising means for applying the scaling parameters of the preceding layer of the normalization layer in reverse to the normalization layer.

Images