IMPROVING THE ACCURACY OF MACHINE LEARNING OPERATIONS BY COMPENSATING FOR LOWER ACCURACY DUE TO SCALE SHIFTING

Abstract

A technical solution is presented to improve the accuracy of machine learning operations by compensating for lower accuracy through scale shifting.An example non-transitory computer-readable medium includes instructions that, when executed, cause a machine to identify at least a first accuracy data type and a second accuracy data type associated with the execution of a machine learning model, the first accuracy data type having a first data accuracy that is greater than a second data accuracy of the second accuracy data type, determine at least one scaling factor to be applied to first weights of the machine learning model, the first weights based on the first accuracy data type, and convert the first weights to second weights based on a multiplication of the first weights and the at least one scaling factor, the second weights based on the second accuracy data type.

Description

AREA OF REVELATION

[0001] This disclosure relates generally to operations (e.g., dot product operations) in machine learning (ML), and more particularly to improving the accuracy of ML operations by compensating for lower accuracy with scale shift. BACKGROUND

[0002] Operations (e.g., dot product operations) performed in bulk are an element of artificial intelligence (AI), machine learning (ML), and / or general scientific computing. Hardware vendors promote the acceleration of these operations by providing new instructions (e.g., Advanced Vector Extensions (AVX), Advanced Matrix Extensions (AMX)) and / or specialized functional units, and software vendors implement the new instructions and / or specialized units to perform further algorithmic changes, data layout changes, backup operations to increase cache efficiency, and other computing-related performance. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is an illustration of an example electronic system including an example accelerator compiler to configure an example accelerator circuit based on an acceleration operation to be performed by the accelerator circuit. Fig. 2 is a block diagram of an example implementation of the accelerator compiler of Fig. 1. Fig. 3 is an illustration of an exemplary conventional convolution operation performed by the exemplary acceleration circuit of the Fig. 1 and / or 2 can be executed. Fig. 4 shows a bitwise binary representation of an FP32 value. Fig. Figure 5 shows an exemplary enumeration process for selecting a scale factor as used by the exemplary scale factor determination circuit of Fig. 2 is carried out. Fig. Figure 6 shows an example of a comparative accuracy diagram for example operations with FP32 values, BF16 values, and scaled BF16 values. Fig. Figure 7 shows an example accuracy percentage table for example operations using FP32 values, BF16 values, a first technique with scaled BF16 values, and a second technique with scaled BF16 values. Fig. 8-11 are flowcharts illustrating example machine-readable instructions and / or example operations that may be performed by example processor circuits to implement the acceleration circuits of Fig. 1 and / or 2 to be implemented. Fig. 12 is a block diagram of an example processing platform having processor circuitry structured to execute the example machine-readable instructions and / or the example operations of Fig. 8-11 to activate the acceleration circuit of Fig. 1 and / or 2 to be implemented. Fig. 13 is a block diagram of an exemplary implementation of the processor circuit of Fig. 12. Fig. 14 is a block diagram of another exemplary implementation of the processor circuit of Fig. 12. Fig. 15 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., software compliant with the example machine-readable instructions of Fig. 8-11) to client devices that interact with end users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed, for example, to retailers and / or to other end users such as direct customers).

[0003] In general, the same reference numbers are used throughout the drawings and the accompanying written description to refer to identical or similar parts. The figures are not to scale. Instead, the thickness of the layers or regions may be exaggerated in the drawings. Although the figures show layers and regions with clear lines and boundaries, some or all of these lines and / or boundaries may be idealized. In reality, the boundaries and / or lines may be unobservable, superimposed, and / or irregular.

[0004] Notwithstanding the foregoing, in the case of a semiconductor device, "above" does not refer to ground, but rather to a bulk region of a base semiconductor substrate (e.g., a semiconductor wafer) on which the components of an integrated circuit are formed. A first device of an integrated circuit is "above" a second device if the first device is farther from the bulk region of the semiconductor substrate than the second device, as used herein.

[0005] Unless otherwise specified, the term "above" describes the relationship between two parts relative to the Earth. A first part is above a second part if the second part has at least one part between the Earth and the first part. Likewise, a first part is "below" a second part if the first part is closer to the Earth than the second part. As previously mentioned, a first part can be above or below a second part, with one or more of the following characteristics: other parts between them, with no other parts between them, with the first and second parts touching, or without the first and second parts in direct contact with each other.

[0006] Unless explicitly stated otherwise, descriptors such as "first," "second," "third," etc., are used herein without implying or otherwise indicating any meaning of priority, physical order, arrangement in a list, and / or order, but are used merely as labels and / or arbitrary names to distinguish elements for easier understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while the same element may be referred to in a claim with a different descriptor, such as "second" or "third." In such cases, such descriptors should serve merely to uniquely identify elements that might otherwise, for example, share the same name.

[0007] As used herein, "approximately" and "about" change their themes / values to acknowledge the potential existence of variations that occur in real-world applications. For example, the terms "approximately" and "about" may change dimensions that are not exact in practice due to manufacturing tolerances and / or other imperfections, as those familiar with the subject matter will appreciate. For example, "approximately" and "about" may indicate that these dimensions may be within a + / - 10% tolerance range unless otherwise noted in the description below. As used herein, "essentially real-time" refers to near-instantaneous occurrence, although in the real world there may be delays due to computing time, transmission, etc. Unless otherwise noted, the term "essentially real-time" refers to real-time + / - 1 second.

[0008] As used herein, the term “processor circuit” is defined to include (i) one or more special-purpose electrical circuits structured to perform specific operations and one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors) and / or (ii) one or more general-purpose electrical semiconductor-based circuits programmable with instructions to perform specific operations and one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors).Examples of processor circuits include programmable microprocessors, FPGAs (Field Programmable Gate Arrays) that can instantiate instructions, CPUs (Central Processor Units), GPUs (Graphics Processor Units), DSPs (Digital Signal Processors), XPUs or microcontrollers, and integrated circuits such as ASICs (Application Specific Integrated Circuits). An XPU, for example, can be implemented by a heterogeneous computing system that includes multiple types of processor circuits (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or a combination thereof) and application programming interface(s) (API(s)), allowing the processor to assign computational task(s) to the one(s) best suited to perform the computational task(s). DETAILED DESCRIPTION

[0009] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables machines (e.g., computers, logic circuits, etc.) to use a model to process input data to produce an output based on patterns and / or associations previously learned by the model through a training process. For example, the model may be trained on data to recognize patterns and / or associations and follow such patterns and / or associations when processing input data, so that other inputs result in outputs consistent with the recognized patterns and / or associations.

[0010] There are many different types of machine learning models and / or machine learning architectures. Some examples disclosed herein use a convolutional neural network (CNN) model. Using a CNN model enables weight sharing (e.g., reducing the number of weights that must be learned by the model), thereby reducing model training time and computational costs. In general, the machine learning models / architectures suitable for use in the examples disclosed herein are neural networks (NN), deep neural networks (DNN), and / or recurrent neural networks (RNN). However, additionally or alternatively, other types of machine learning models such as support vector machines (SVM), long-term short memory (LSTM), gated recurrent units (GRU), etc. may also be used.

[0011] In general, implementing an ML / AI system involves two phases: a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train a model to operate according to patterns and / or associations, for example, based on training data. Generally, the model has internal parameters that direct how input data is transformed into output data, e.g., through a series of nodes and connections within the model to transform input data into output data. Additionally, hyperparameters are used as part of the training process to control how learning is performed (e.g., a learning rate, a number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0012] Depending on the nature of the ML / AI model and / or the expected output, different types of training can be performed. For example, supervised training uses inputs and corresponding expected (e.g., labeled) outputs to select parameters (e.g., by repeating the combinations of selected parameters) for the ML / AI model that reduce model error. As used here, labeling refers to an expected output of the machine learning model (e.g., a classification, an expected output value, etc.). Alternatively, unsupervised training (e.g., in deep learning, a subset of machine learning, etc.) involves inferring patterns from the inputs to select parameters for the ML / AI model (e.g., without the benefit of expected (e.g., labeled) outputs).

[0013] In examples disclosed herein, ML / AI models are trained using stochastic gradient descent. However, in addition or alternatively, any other training algorithm may be used. In the examples described herein, training is performed until an acceptable amount of error is reached. In the examples disclosed herein, training may be performed in the electronic system 102 (e.g., for the ML model(s) 124). Training is performed using hyperparameters that control how learning is performed (e.g., a learning rate, a number of layers to be used in the machine learning model, etc.). In the examples disclosed herein, hyperparameters are used that control the accuracy of the values used as operands. Such hyperparameters are selected, for example, manually and / or through statistical (random) sampling.In some examples, retraining may be performed. Such retraining may be performed if an accuracy metric fails to meet a threshold.

[0014] Training is performed using training data. In the examples disclosed herein, the training data may be from a data store (e.g., from the example data store 270 used in conjunction with Fig. 2). Since this is a supervised training, the training data are labeled. The labeling is performed by an accelerator compiler (e.g., accelerator compiler 104A-C, which is used in conjunction with Fig. 1) is applied to the training data. In some examples, the training data is preprocessed, e.g., using an interface (e.g., interface circuit 114 used in conjunction with Fig. 1). In some examples, the accelerator compiler 104A-C of Fig. 1 the training data into a first part of data for training the machine learning model(s) 124 and a second part of data for validating the machine learning model(s) 124 of Fig. 1.

[0015] Once training is complete, the model is deployed for use as an executable construct that processes an input and provides an output based on the network of nodes and connections defined in the model. The model is stored in a data store (e.g., example data store 270 of Fig. 2). The model may then be executed by an exemplary model execution circuit 250 (used in conjunction with Fig. 2). In some examples, the platform on which the model is running may have certain limitations regarding the precision and / or accuracy of the operands.

[0016] After training, the deployed model can be run in an inference phase to process data. In the inference phase, data to be analyzed (e.g., live data) is input to the model, and the model is executed to produce an output. This inference phase can be thought of as the AI "thinking" to produce the output based on what was learned from training (e.g., by running the model to apply the learned patterns and / or associations to the live data). In some examples, the input data undergoes preprocessing before being used as input to the machine learning model. Furthermore, in some examples, after being generated by the AI model, the output data can undergo postprocessing to transform the output into a useful result (e.g., a display of data, an instruction to be executed by a machine, etc.).

[0017] In some examples, the output of the deployed model can be captured and provided as feedback. By analyzing the feedback, an accuracy of the deployed model can be determined. If the feedback indicates that the accuracy of the deployed model is lower than a threshold or other criterion, training of an updated model can be triggered using the feedback, an updated training dataset, hyperparameters, etc., to generate an updated deployed model.

[0018] There are many different ways an operand and / or value (e.g., integer, scalar, fraction, etc.) can be represented on a given computing device, with each representation characterized by a different amount of storage space and / or different levels of precision. For example, a simple floating-point format (hereafter referred to as the "FP32" format) is most commonly used in machine learning (ML) and artificial intelligence (AI) applications, particularly in deep neural networks (DNNs). The FP32 format is characterized by 1 sign bit, 8 exponent bits, and 23 mantissa bits, which enables high precision due to the larger-than-average available bit memory. However, the FP32 format is characterized by a larger storage footprint because each value represented in this format comprises a total of 32 bits.Therefore, in situations where large amounts of values and / or data need to be processed and / or stored in a memory of a specific computer, a remote server, etc., a reduction in machine performance can be observed.

[0019] The Bfloatl6 floating-point format (referred to here as the "BF16" format), which can also be used for ML and AI calculations, is characterized by 1 sign bit, 8 exponent bits, and 7 mantissa bits. This results in lower precision compared to the FP32 format due to less bit storage. However, since the BF16 format requires 16 fewer mantissa bits for memory storage, it also features a smaller memory footprint, which is an advantage for better (e.g., more efficient) performance of a computing device and / or less resource-intensive computation.

[0020] For many types of calculations (e.g., scientific calculations), it is perfectly acceptable to work with lower-precision values (such as BF16 values) in favor of a better (e.g., more efficient and / or less resource-intensive) calculation. For example, an advertising engine may achieve a lower click-through rate (CTR) due to a small inaccuracy in classification (e.g., a lower number of users clicking on an ad rather than skimming it). In some cases, the inaccuracies in classification may be related to a reduction in the number of floating-point bits (e.g., mantissa bits) used to describe the value, with the lower number of bits indicating a lower degree of precision (e.g.,similar to the number of digits listed after a decimal point for fractional values), and the lower degree of precision results in a frequency of classification errors that is considered acceptable in the advertising industry.

[0021] However, for many other types of calculations (e.g., cumulative dot product operations), accuracy cannot be compromised for faster and / or better performance. In particular, when artificial intelligence (AI) and / or high-performance computing (HPC) techniques are used in fields requiring high-precision operation, such as finance, robotics, pharmaceuticals, radiology, etc., formats for representing low-precision operands and / or values, such as BF16, are not preferred over formats for representing high-precision values that require more storage space, such as FP32. In these examples, higher accuracy is preferred over computational efficiency.

[0022] Therefore, increasing the accuracy provided by a typically less accurate value representation format (e.g., BF16) while maintaining the less resource-intensive characteristics of such a format is the desired approach to increasing the efficiency of data processing and / or operation of electronic systems (e.g., exemplary electronic system 102, exemplary external electronic systems 130 used in conjunction with Fig. 1 will be explained in more detail).

[0023] Current approaches to using lower-fidelity data representation formats focus on frequently retraining the machine learning (ML) and / or artificial intelligence (AI) model in which they are used, using extensive software testing to determine that any loss of accuracy resulting from switching to lower-fidelity value representations (e.g., BF16) is within an acceptable range. Other approaches use higher-fidelity value representations (e.g., FP32) in the training phase of the ML / AI model and lower-fidelity value representations (e.g., BF16) in the inference phase of the ML / AI model. These approaches also employ extensive software testing to ensure that the associated loss of accuracy is routinely within the acceptable range.

[0024] The extensive software retraining and / or testing employed and / or required by these approaches often results in unsatisfactory, delayed, and / or unclear recommendations (e.g., particularly during the ML / AI model's inference phase). Furthermore, the frequent repetition of this precision and accuracy testing required to ensure optimal model results is resource- and computationally intensive and / or challenging, especially in cases where the test and / or training datasets are very large and / or change frequently. This means that the software testing required to ensure that model results are within an acceptable accuracy range can result in lengthening validation cycles.Furthermore, the decision between using low- or high-fidelity data representation formats becomes increasingly complex when retraining and / or retraining the model improves accuracy, but using lower-fidelity data representation formats decreases accuracy. Furthermore, the approaches described here are only applicable in a limited number of situations due to a foreseeable and / or observed risk of making an incorrect cutoff decision, particularly when the triage performed by a model using a lower-fidelity data representation format (e.g., BF16) results in an incorrect confidence value. In short, current approaches often complicate the process of deploying ML / AI models by complicating the training phase and / or the post-training testing / inference phase.

[0025] Exemplary methods for using lower-precision data representation formats while maintaining high accuracy of ML / AI model results focus on scaling values and / or operands from high-precision data representation formats (e.g., FP32) to lower-precision data representation formats (e.g., BF16) using a weighting factor, and then scaling back (e.g., from BF16 to FP32 format) their resulting value (e.g., after the operation and / or computation) by the same order of magnitude (e.g., the same weighting factor). Such a method reduces the amount of information stored and / or contained in the lower bits of the mantissa of the value representation format (e.g., with the BF16 representation format, which is characterized by 16 fewer mantissa bits), effectively reducing the storage requirements and / or computational demands associated with each value.For operations performed by the ML / AI model, such as dot product operations, where frequent accumulation of values occurs on a large scale, the cumulative information stored in the lower-order bits of the FP32 representation of the mantissa is effectively captured via a scaling and / or weighting factor applied to the values upon conversion to their BF16 representation, thereby reducing the incidence of truncation errors and / or information / precision losses associated with converting these value formats. This means that, for example, rescaling (e.g., descaling) the values represented by BF16 effectively corrects any potential truncation error observed by eliminating the 16 lower-order mantissa bits by removing the same mantissa bits after the completion of the given computation and / or operation (e.g.,dot product operation) back to their original FP32 format, effectively eliminating any bias, error, and / or the need for extensive software testing to ensure model conformance within an acceptable range of accuracy of the results.

[0026] Furthermore, eliminating precision loss as a barrier to using lower-precision data representation formats such as BF16 enables a massive reduction in computational cost, effort, and / or resource expenditures (e.g., by reducing overall memory requirements). While the examples disclosed herein use FP32 and BF16 data representation formats to describe scaling and descaling between higher- and lower-precision data representation formats, any other type of data representation format can be used to perform bit-width reductions.

[0027] Fig. 1 is an illustration of an example computing environment 100 with an example electronics system 102 including an example accelerator compiler 104A-C for configuring an ML / AI accelerator to perform scaling operations such as convolution operations, matrix multiplication operations (e.g., MatMul), etc., to achieve improved efficiency and performance of the accelerator. In some examples, the accelerator compiler 104A-C receives output from a machine learning framework (e.g., a NN framework) and compiles the output for implementation on the accelerator based on the scaling operation to be performed and / or otherwise performed by the accelerator.

[0028] The electronic system 102 of the Fig. 1 includes an example central processing unit (CPU) 106, a first example acceleration circuit (ACCELERATING CIRCUIT A) 108, a second example acceleration circuit (ACCELERATING CIRCUIT B) 110, an example general-purpose processing circuit 112, an example interface circuit 114, an example bus 116, an example power source 118, and an example data storage 120. In this example, the data storage 120 includes example configuration data (CONFIGURATION DATA) 122 and example machine learning model(s) (ML MODEL(S)) 124. In the example shown in Fig. 1, an example user interface 126, an example network 128, and an example external electronic systems 130 are also depicted.

[0029] In some examples, the electronic system 102 is a system on a chip (SoC) comprised of one or more integrated circuits (ICs) (e.g., compact ICs) that contain components of a computer or other electronic system in a compact format. For example, the electronic system 102 may be implemented with a combination of one or more programmable processors, hardware logic, and / or hardware peripherals and / or interfaces. Additionally or alternatively, the example electronic system 102 may be Fig. 1 may include memory, one or more input / output ports, and / or secondary storage. For example, the electronic system 102 includes the accelerator compiler 104A-C, the CPU 106, the first acceleration circuit 108, the second acceleration circuit 110, the general processing circuit 112, the interface circuit 114, the bus 116, the power source 118, the data storage 120, the memory, the I / O port(s), and / or the secondary storage, all located on the same substrate (e.g., silicon substrate, semiconductor-based substrate, etc.). In some examples, the electronic system 102 includes digital, analog, mixed-signal, radio frequency (RF), or other signal processing functions.

[0030] In the Fig. 1, the first acceleration circuit 108 is an artificial intelligence (AI) accelerator. For example, the first acceleration circuit 108 may be implemented by a hardware accelerator configured to accelerate AI tasks or workloads, such as NNs (e.g., artificial neural networks (ANNs)), machine vision, machine learning, etc. In some examples, the first acceleration circuit 108 may implement an ML / AI accelerator (e.g., a sparse hardware accelerator). In some examples, the first acceleration circuit 108 may implement a vision processing unit (VPU) to perform machine or computer vision computing tasks, train and / or execute a physical neural network, and / or train and / or execute a neural network.In some examples, the first acceleration circuit 108 may train and / or execute a convolutional neural network (CNN), a deep neural network (DNN), an ANN, a recurrent neural network (RNN), etc., and / or a combination thereof.

[0031] In the Fig. 1, the second acceleration circuit 110 is a graphics processing unit (GPU). For example, the second acceleration circuit 110 may be a GPU that generates computer graphics, performs general-purpose computations, etc. In some examples, the second acceleration circuit 110 is another instance of the first acceleration circuit 108. In some such examples, the electronic system 102 may provide portions of AI / ML workloads that are executed in parallel by the first acceleration circuit 108 and the second acceleration circuit 110.

[0032] The multipurpose processing circuit 112 of the example in Fig. 1 is a programmable processor, such as a CPU or a GPU. Alternatively, one or more of the first acceleration circuits 108, the second acceleration circuits 110, and / or the general processing circuits 112 may be another type of hardware, such as a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), and / or a field-programmable logic device (FPLD) (e.g., a field-programmable gate array (FPGA)).

[0033] In the example shown by Fig. 1, interface circuitry 114 is hardware that can implement one or more interfaces (e.g., computer interfaces, network interfaces, etc.). Interface circuitry 114 can be, for example, hardware, software, and / or firmware that implements a communication device (e.g., a network interface card (NIC), an intelligent NIC, a gateway, a switch, etc.) such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface to facilitate the exchange of data with external machines (e.g., computing devices of any type) over network 128.In some examples, communication occurs via a Bluetooth® connection, an Ethernet connection, a Digital Subscriber Line (DSL) connection, a Wireless Fidelity (Wi-Fi) connection, a telephone connection, a coaxial cable system, a satellite system, a wireless line-of-site system, a cellular phone system, an optical connection (e.g., a fiber optic connection), etc. For example, the interface circuit 114 may be implemented by any type of interface standard, such as a Bluetooth® interface, an Ethernet interface, a Wi-Fi interface, a Universal Serial Bus (USB) interface, a Near Field Communication (NFC) interface, and / or a Peripheral Component Interconnect Express (PCIe) interface.

[0034] The electronic system 102 includes the power source 118 to power the hardware of the electronic system 102. In some examples, the power source 118 may implement a power grid. The power source 118 may, for example, be an alternating current to direct current (AC / DC) power supply. In some examples, the power source 118 may be connected to a power grid infrastructure such as an AC grid (e.g., a 110-volt AC grid, a 220-volt AC grid, etc.). Additionally or alternatively, the power source 118 may be implemented by a battery. The power source 118 may, for example, be a limited-energy device such as a lithium-ion battery or other rechargeable battery or power source. In some examples, the power source 118 may be connected via a power adapter or converter (e.g., an AC / DC power converter), a power outlet (e.g.,a 110V AC outlet, a 220V AC outlet, etc.), a portable energy storage device (e.g., a portable power bank, a portable power cell, etc.), etc.

[0035] The electronic system 102 of the Fig. 1 includes data storage 120 for recording data (e.g., configuration data 122, ML model(s) 124, etc.). Data storage 120 of this example may be implemented by volatile memory (e.g., synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), etc.) and / or non-volatile memory (e.g., flash memory). Data storage 120 may additionally or alternatively be implemented by one or more double data rate (DDR) memories, such as DDR, DDR2, DDR3, DDR4, mobile DDR (mDDR), etc. The data storage 120 may additionally or alternatively be implemented by one or more mass storage devices such as hard disk drive(s) (HDD(s)), compact disc drive(s) (CD), DVD drive(s) (Digital Versatile Disc), solid state disk drive(s) (SSD), etc.While the data store 120 is depicted as a single data store in the example shown, the data store 120 may be implemented using any number and / or type(s) of data stores. Furthermore, the data stored in the data store 120 may be in any data format, such as binary data, comma-separated data, tab-separated data, Structured Query Language (SQL) structures, an executable file, etc.

[0036] In the Fig. 1, the electronic system 102 is in communication with the user interface 126. For example, the user interface 126 may be implemented by a graphical user interface (GUI), an application user interface, etc., which may be presented to a user on a display device connected to and / or otherwise communicating with the electronic system 102. In some of these examples, a user (e.g., a developer, an IT administrator, a customer, etc.) controls the electronic system 102, configures, trains, and / or executes the ML model(s) 124, etc., via the user interface 126. Alternatively, the electronic system 102 may also include and / or otherwise implement the user interface 126.

[0037] In the Fig. 1, the accelerator compiler 104A-C, the CPU 106, the first acceleration circuit 108, the second acceleration circuit 110, the general-purpose processing circuit 112, the interface circuit 114, the power source 118, and the data memory 120 communicate with each other via the bus 116. For example, the bus 116 may be implemented by at least one of the following bus types: Inter-Integrated Circuit (I2C) bus, Serial Peripheral Interface (SPI) bus, Peripheral Component Interconnect (PCI) bus, or PCIe bus. Additionally or alternatively, the bus 116 may be implemented by any other type of data processing or electrical bus.

[0038] In the example shown by Fig. 1, the network 128 is the Internet. However, the network 128 of this example may be implemented using one or more suitable wired and / or wireless networks, including, for example, one or more data buses, one or more local area networks (LANs), one or more wireless LANs, one or more cellular networks, one or more private networks, one or more public networks, etc. In some examples, the network 128 enables the electronic system 102 to communicate with one or more external electronic systems 130.

[0039] In the example shown by Fig. 1, the external electronic systems 130 include one or more electronic devices (e.g., computers) on which the ML model(s) 124 are to be executed and / or otherwise implement them. In this example, the external electronic systems 130 include an example desktop computer 132, an example mobile device (e.g., a smartphone, an internet-enabled smartphone, etc.) 134, an example laptop computer 136, an example tablet (e.g., a tablet computer, an internet-enabled tablet computer, etc.) 138, and an example server 140. In some examples, fewer or more than the Fig. 1. Additionally or alternatively, the external electronic systems 130 may include, correspond to, and / or otherwise be representative of any other type and / or quantity of computing devices.

[0040] In some examples, one or more of the external electronic systems 130 execute one or more of the ML models 124 to process a computational effort (e.g., an AI / ML workload). For example, the mobile device 134 may be implemented as a mobile phone with one or more processors (e.g., a CPU, a GPU, a VPU, an AI or NN-specific processor, etc.) on a single SoC to process an AI / ML workload with one or more ML models 124. In some examples, the desktop computer 132, the laptop computer 136, the tablet computer, and / or the server 140 may be implemented as electronic device(s) (e.g., computers) having one or more processors (e.g., a CPU, a GPU, a VPU, an AI / NN-specific processor, etc.) on one or more SoCs to process AI / ML workload(s) using one or more ML models 124.In some examples, server 140 may implement one or more servers (e.g., physical servers, virtualized servers, etc., and / or a combination thereof) that may implement a data facility, a cloud service (e.g., a public or private cloud provider, a cloud-based repository, etc.), etc., to process AI / ML workload(s) using one or more ML models 124.

[0041] In the example shown by Fig. 1, the electronic system 102 includes a first accelerator compiler 104A (e.g., a first instance of the accelerator compiler 104A-C), a second accelerator compiler 104B (e.g., a second instance of the accelerator compiler 104A-C), and a third accelerator compiler 104C (e.g., a third instance of the accelerator compiler 104A-C) (collectively referred to herein as accelerator compiler 104A-C unless otherwise noted). In this example, the first accelerator compiler 104A is implemented by the CPU 106 (e.g., by hardware, software, and / or firmware of the CPU 106).

[0042] In the example shown by Fig. 1, the second accelerator compiler 104B is implemented by the general-purpose processing circuit 112 (e.g., by hardware, software, and / or firmware of the general-purpose processing circuit 112). In this example, the third accelerator compiler 104C is external to the CPU 106. For example, the third accelerator compiler 104C may be implemented by hardware, software, and / or firmware of the electronic system 102. In some of these examples, the third accelerator compiler 104C may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, GPUs, DSPs, ASICs, PLDs, and / or FPLDs.

[0043] In some examples, the first accelerator compiler 104A, the second accelerator compiler 104B, the third accelerator compiler 104C, and / or portions thereof may be virtualized by being implemented with one or more containers, one or more virtual resources (e.g., virtualizations of compute, memory, network, storage, etc., physical hardware resources), one or more virtual machines, etc. In some examples, one or more of the first accelerator compiler 104A, the second accelerator compiler 104B, the third accelerator compiler 104C, and / or portions thereof may be implemented by different resources of the electronic system 102. Alternatively, the electronic system 102 may not include the first accelerator compiler 104A, the second accelerator compiler 104B, and / or the third accelerator compiler 104C.

[0044] In the Fig. 1, the accelerator compiler 104A-C may compile an AI / ML framework based on the configuration data 122 for implementation on one or more of the accelerator circuits 108, 110. In some examples, the configuration data 122 may include AI / ML configuration data (e.g., register configurations, activation data, activation sparsity data, weight data, weight sparsity data, hyperparameters, etc.), a convolution operation to be performed (e.g., a 2D convolution, a depth-first convolution, a grouped convolution, a dilated convolution, etc.), a non-convolution operation (e.g., an element-wise addition operation), and / or a combination thereof. In some examples, the accelerator compiler 104A-C may compile the AI / ML framework to produce an executable construct that may be executed by the acceleration circuit(s) 108, 110.

[0045] In the example shown by Fig. 1, the accelerator compiler 104A-C may instruct, direct, and / or otherwise invoke one or more of the accelerator circuits 108, 110 to execute one or more of the ML models 124. The ML models 124 may, for example, implement AI / ML models. AI, including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables machines (e.g., computers, logic circuits, etc.) to use a model to process input data to produce an output based on patterns and / or associations previously learned by the model via a training process. For example, the machine learning model(s) 124 may be trained with data to recognize patterns and / or associations and to follow such patterns and / or associations when processing input data, such that other inputs result in outputs consistent with the recognized patterns and / or associations.

[0046] There are many different types of machine learning models and / or machine learning architectures. In some examples, the accelerator compiler 104A-C generates the machine learning model(s) 124 as neural network model(s). The accelerator compiler 104A-C may call the interface circuitry 114 to communicate the machine learning model(s) 124 to one or more of the external electronic systems 130. Using a neural network model enables the accelerator circuitry 108, 110 to execute an AI / ML workload. Machine learning models / architectures suitable for use in the examples disclosed herein generally include recurrent neural networks. However, other types of machine learning models may also be used, such as supervised learning ANN models, clustering models, classification models, etc., and / or a combination thereof.Examples of supervised learning ANN models include two-layer radial basis neural networks (RBN), learning vector quantization (LVQ) neural networks for classification, etc. Examples of clustering models include k-means clustering, hierarchical clustering, mean-shift clustering, density-based clustering, etc. Examples of classification models may include logistic regression, support vector machine (SVM) or network, Naive Bayes, etc. In some examples, the accelerator compiler 104A-C may compile and / or otherwise generate one or more of the machine learning models 124 as lightweight machine learning models.

[0047] In general, implementing an ML / AI system involves two phases: a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train the machine learning model(s) 124 to operate in accordance with patterns and / or associations based, for example, on training data. In general, the machine learning model(s) 124 includes internal parameters (e.g., the configuration data 122) that dictate how input data is transformed into output data, e.g., through a series of nodes and connections within the machine learning model(s) 124 to transform input data into output data. Additionally, hyperparameters (e.g., the configuration data 122) are used as part of the training process to control how learning is performed (e.g.,a learning rate, a number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0048] Depending on the nature of the ML / AI model and / or the expected output, different types of training may be performed. For example, the accelerator compiler 104A-C may invoke supervised training to use inputs and corresponding expected (e.g., labeled) outputs to select parameters (e.g., by iterating over combinations of selected parameters) for the machine learning model(s) 124 that reduce model error. As used herein, the term "labeling" refers to an expected output of the machine learning model (e.g., a classification, an expected output value, etc.). Alternatively, the accelerator compiler 104A-C may perform unsupervised training (e.g., in deep learning, a subset of machine learning, etc.) in which patterns are inferred from the inputs to select parameters for the machine learning model(s) 124 (e.g., without the benefit of expected (e.g., labeled) outputs).

[0049] In some examples, the accelerator compiler 104A-C trains the machine learning model(s) 124 using unsupervised clustering of operational observation data. However, the accelerator compiler 104A-C may additionally or alternatively use any other training algorithm, such as stochastic gradient descent, simulated annealing, particle swarm optimization, evolutionary algorithms, genetic algorithms, nonlinear conjugate gradients, etc.

[0050] In some examples, the accelerator compiler 104A-C may train the machine learning model(s) 124 until the error amount no longer decreases. In some examples, the accelerator compiler 104A-C may train the machine learning model(s) 124 locally on the electronic system 102 and / or remotely on an external electronic system (e.g., one of the external electronic systems 130) communicatively coupled to the electronic system 102. In some examples, the accelerator compiler 104A-C trains the machine learning model(s) 124 using hyperparameters that control how the learning is performed (e.g., a learning rate, a number of layers used in the machine learning model, etc.). In some examples, the accelerator compiler 104A-C may use hyperparameters that control model performance and training speed, e.g., B. the learning rate and regularization parameters.The accelerator compiler 104A-C may select such hyperparameters, for example, through trial and error, to achieve optimal model performance. In some examples, the accelerator compiler 104A-C uses Bayesian hyperparameter optimization to determine an optimal and / or otherwise improved or more efficient network architecture to avoid model overfitting and improve the general applicability of the machine learning model(s) 124. Alternatively, the accelerator compiler 104A-C may use any other type of optimization. In some examples, the accelerator compiler 104A-C may perform retraining. The accelerator compiler 104A-C may perform such retraining in response to override by a user of the electronic system 102, receipt of new training data, etc.

[0051] In some examples, the accelerator compiler 104A-C facilitates training of the machine learning model(s) 124 using training data. In some examples, the accelerator compiler 104A-C uses training data obtained from locally generated data. In some examples, the accelerator compiler 104A-C uses training data obtained from externally generated data. In some examples where supervised training is used, the accelerator compiler 104A-C may label the training data. The label is applied to the training data manually by a user or by an automatic data preprocessing system. In some examples, the accelerator compiler 104A-C may preprocess the training data, for example, by using an interface (e.g., interface circuitry 114).In some examples, the accelerator compiler 104A-C divides the training data into a first portion of data for training the machine learning model(s) 124 and a second portion of data for validating the machine learning model(s) 124.

[0052] After training is complete, the accelerator compiler 104A-C may provide the machine learning model(s) 124 for use as an executable construct that processes an input and provides an output based on the network of nodes and connections defined in the machine learning model(s) 124. The accelerator compiler 104A-C may store the machine learning model(s) 124 in the data storage 120. In some examples, the accelerator compiler 104A-C may call the interface circuitry 114 to transfer the machine learning model(s) 124 to one or more of the external electronic systems 130. In some examples, in response to transmitting the machine learning model(s) 124 to the external electronic system(s) 130, the external electronic system 130 may execute the machine learning model(s) 124 to execute AI / ML workloads with at least improved efficiency or performance.

[0053] After training, the deployed machine learning models 124 may be operated in an inference phase to process data. In the inference phase, the data to be analyzed (e.g., live data) is input to the machine learning model(s) 124, and the machine learning model(s) 124 is / are executed to produce an output. This inference phase can be thought of as the AI "thinking" to generate the output based on what it learned during training (e.g., by executing the machine learning model(s) 124 to apply the learned patterns and / or associations to the live data). In some examples, the input data undergoes preprocessing before being used as input to the machine learning model(s) 124.Furthermore, in some examples, after the output data is generated by the machine learning model(s) 124, it may be subjected to post-processing to transform the output into a useful result (e.g., a display of data, a detection and / or identification of an object, an instruction to be executed by a machine, etc.).

[0054] In some examples, the output of the deployed machine learning model(s) 124 may be captured and provided as feedback. By analyzing the feedback, the accuracy of the deployed machine learning model(s) 124 may be determined. If the feedback indicates that the accuracy of the deployed model is lower than a threshold or other criterion, training of an updated model may be triggered using the feedback, an updated training dataset, hyperparameters, etc., to generate an updated deployed model.

[0055] In some examples, the accelerator compiler 104A-C configures one or more of the accelerator circuits 108, 110 to perform a convolution operation, such as a 2D convolution operation. The accelerator circuits 108, 110 may, for example, implement a CNN. In some examples, CNNs ingest and / or otherwise process images as tensors, which are matrices of numbers with additional dimensions. For example, a CNN may receive an input image represented by 3D tensors, where first and second dimensions correspond to a width and height of a matrix and a third dimension corresponds to a depth of the matrix. For example, the width and height of the matrix may correspond to a width and height of an input image, and the depth of the matrix may correspond to a color depth (e.g., a color plane) or a color encoding of the image (e.g., a red-green-blue (RGB) encoding).

[0056] A typical CNN may also receive an input and transform it through a series of hidden layers. For example, a CNN may have multiple convolutional layers, pooling layers, and / or fully connected layers. In some examples, a CNN may have multiple triple layers, including a convolutional layer, a pooling layer, and a fully connected layer. In some examples, a CNN may have multiple pairs of convolutional and pooling layers, resulting in one or more fully connected layers. In some examples, a CNN may include 20 layers, 30 layers, etc.

[0057] In some examples, the acceleration circuit 108, 110 may execute a convolutional layer to apply a convolutional function or operation to map images from a (previous) input layer to the next layer in a CNN. In some examples, the convolution may be three-dimensional (3-D), as each input layer may have multiple input features (e.g., input channels) associated with an input image. The acceleration circuits 108, 110 may execute the convolutional layer to perform convolution by forming a regional filter window in each individual input channel and generating output data or activations by computing a product of (1) a filter weight associated with the regional filter window and (2) the input data covered by the regional filter window.For example, the acceleration circuits 108, 110 may determine an output feature of an input image by using the convolution filter to sample a plurality of input channels containing a plurality of regional filter windows.

[0058] In some examples, the acceleration circuits 108, 110 may perform a pooling layer to extract information from a set of activations in each output channel. The pooling layer may perform a max pooling operation corresponding to a max pooling layer or an average pooling operation corresponding to an average pooling layer. In some examples, the max pooling operation may include selecting a maximum value of activations within a pooling window. In some examples, the average pooling operation may include calculating an average value of the activations within the pooling window.

[0059] In some examples, the acceleration circuit 108, 110 may execute a fully connected layer to obtain the data computed by the convolutional layer(s) and / or the pooling layer(s) and / or classify the data into one or more classes. In some examples, the fully connected layer may determine whether the classified data corresponds to a particular image feature of the input image. For example, the acceleration circuit 108, 110 may execute the fully connected layer to determine whether the classified data corresponds to a simple image feature (e.g., a horizontal line) or a more complex image feature such as an animal (e.g., a cat).

[0060] In some examples, the accelerator compiler 104A-C may configure one or more of the accelerator circuits 108, 110 to perform non-2D convolution operations as 2D convolution operations. For example, the accelerator compiler 104A-C may configure the accelerator circuit(s) 108, 110 to implement a depth-first convolution operation, an element-wise addition operation, a grouped convolution operation, a dilated convolution operation, a user-defined operation (e.g., a user-defined convolution, a user-defined acceleration operation, etc.), etc., as a 2D convolution operation. In some such examples, the accelerator compiler 104A-C may instruct the one(s) of the acceleration circuits 108, 110 to generate data internally rather than receiving the data from the accelerator compiler 104A-C, the configuration data 122, etc.For example, the accelerator compiler 104A-C may instruct the first acceleration resource to generate activation sparsity data, weight sparsity data, or weighting data based on the acceleration operation to be performed by the first acceleration circuit 108. In some such examples, the accelerator compiler 104A-C may instruct the one or more of the acceleration circuits 108, 110 to execute the ML model(s) 124 based on the data generated by the acceleration circuit(s) 108, 110, which may be based on a convolution operation to be performed by the acceleration circuit(s) 108, 110.

[0061] Fig. 2 is a block diagram of an example accelerator compiler 200. In some examples, the accelerator compiler 200 may be Fig. 2 one or more of the accelerator compilers 104A-C of Fig. 1 to perform scale shifting of operands to increase computational and / or resource efficiency along with operational precision while using lower precision operand values. The accelerator compiler 104A-C of Fig. 2 may be instantiated by processor circuitry such as a central processing unit that executes instructions (e.g., create an instance of, bring into existence for an arbitrary period of time, materialize, implement, etc.). Additionally or alternatively, the accelerator compiler 104A-C of Fig. 1 can be instantiated (e.g., create an instance, bring it into existence for an arbitrary period of time, materialize it, implement it, etc.) by an ASIC or an FPGA that is structured to perform operations corresponding to the instructions. It goes without saying that some or all of the circuitry of Fig. 2 at the same or different times. For example, some or all of the circuits may be instantiated in one or more threads that execute concurrently on the hardware and / or in series on the hardware. Furthermore, in some examples, some or all of the circuits may be Fig. 2 be implemented by microprocessor circuits that execute instructions to implement one or more virtual machines and / or containers.

[0062] In the Fig. 2, the accelerator compiler 200 may configure a hardware accelerator, such as the example interface circuit 210, the example data accuracy selection circuit 220, the example scale factor technique selection circuit 230, the example scale factor determination circuit 240, the example model execution circuit 250, the example executable file generation circuit 260, an example bus 280, and an example data store 270, which further includes an example machine learning (ML) model 272, example operator values 274, example scale factors 276, and example executable file(s) 278.

[0063] In operation, the example interface circuit 210 receives any number of operators (e.g., the operator values 274) to perform a machine learning (ML) operation (e.g., a dot product operation). In the examples disclosed herein, the interface circuit 210 may obtain the operator values 274 from the example data store 270. In the examples disclosed herein, this source may be any type of database, internet source, etc. Furthermore, in the examples disclosed herein, the interface circuit 210 may receive data (e.g., the operator values 274) and / or transmit it to a network or other portions of the electronic system 102 of Fig. 1, such as the acceleration circuits 108, 110 of Fig. 1. In some examples, the interface circuit 210 may also be the interface and / or cable from a laptop to a GPU and / or FPGA to configure an operation such as loading an image. In some examples, the example interface circuit 210 is instantiated by a processor circuit that executes the instructions of the interface circuit 210 and / or is configured to perform operations as described in the flowcharts of Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0064] In some examples, the interface circuit 210 includes means for retrieving any number of operators (e.g., the operator values 274) to perform an ML operation (e.g., a dot product operation). For example, the means for retrieving any number of operators (e.g., the operator values 274) to perform an ML operation (e.g., a dot product operation) may be implemented by the interface circuit 210. In some examples, the interface circuit 210 may be implemented by a processor circuit such as the example processor circuit 1212 of Fig. 12. For example, the interface circuit 210 may be instantiated by the exemplary microprocessor 1300 of Fig. 13, which executes machine-executable instructions as described at least in block 1102 of Fig. 11. In some examples, the interface circuit 210 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the interface circuit 210 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the interface circuit 210 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0065] The example data accuracy selection circuit 220 determines both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed. For example, the data accuracy selection circuit 220 may specify an output format of FP32 and a scaled format of BF16 for each of the operands and / or values to increase computational efficiency. In some examples, the example data accuracy selection circuit 220 is instantiated by a processor circuit that executes instructions of the data accuracy selection circuit 220 and / or is configured to perform operations as shown in the flowcharts of Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0066] In some examples, the data accuracy selection circuit 220 includes means for determining both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed. For example, the means for determining both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed may be implemented by a data accuracy selection circuit 220. In some examples, the data accuracy selection circuit 220 may be implemented by a processor circuit such as the example processor circuit 1212 of Fig. 12. For example, the data accuracy selection circuit 220 may be instantiated by the exemplary microprocessor 1300 of Fig. 13, which executes machine-executable instructions as described at least in blocks 802, 1104 of Fig. 8 and / or 11. In some examples, the data accuracy selection circuit 220 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the data accuracy selection circuit 220 may be implemented by any other combination of hardware, software, and / or firmware. For example, the data accuracy selection circuit 220 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0067] The exemplary scale factor technique selection circuit 230 preprocesses the data to select a scaling technique to be used by the scale factor determination circuit 240. That is, the scale factor technique selection circuit 230 may consider, for example, limitations on acceptable accuracy, data volumes, specific operations to be performed, etc. Equation 1 below shows an example of the dot product of two vectors x i and w i (where all values are represented in FP32 format). Equation 2 similarly shows a BF16 representation x' i , w' i of the two vectors from equation 1 for the same dot product operation. yfp32=∑i=0N−1xiwi ybf16=∑i=0N−1x′iw′i

[0068] The resulting value difference between the individual dot product operations can accordingly be represented as in Equation 3. The scale factor technique selection circuit 230, in conjunction with the scale factor determination circuit 240, selects scale factors that aim to reduce the value of the resulting difference (Δ) that represents a potential loss of accuracy when converting the values between high and low precision data representation formats. Δ=fyp32−ybf16=∑i=0N−1xiwi−∑i=0N−1x′iw′i=∑i=0N−1(x′i+Δx i)(w′i+Δwi)−∑i=0N−1x′iw′i=∑i=0N−1(x′iΔwi+w′iΔxi+ΔxiΔwi)

[0069] To reduce the value of the total dot product value difference (Δ), the scale factor technique selection circuit 230 determines a minimization of the sum of x' i Δ wi + w' i Δ xiby using a weighting and / or scaling factor to account for any rounding and / or truncation errors. In the first term x' i Δ wi : are the inputs x' i into the ML / AI model during the inference phase is likely to be dynamic, while the associated weight values w' i are static values. Since a single scaling factor Δ wi is applied to all weight values, the effect of this factor in the first term is, however, in the products x' i Δ wi constant. If the weights are rescaled accordingly after the dot product is completed, the value of Δ wi can therefore be greatly reduced, which reduces its contribution to the total loss (e.g., difference). The second term w' i Δ xiis difficult to reduce as effectively as the first, since the scaling factor technique selection circuit 230 is unable to accurately predict each of the unknown inputs to the ML / AI model prior to the inference phase. However, by training the model with weights and / or scaling factors by the scaling factor technique selection circuit 230 and / or the exemplary scaling factor determination circuit 240, the weights can be regulated to smaller values (e.g., by penalizing large w i values), effectively reducing the overall difference value (e.g., indicating less loss of accuracy). In some examples, the example scale factor technique selection circuit 230 is instantiated by a processor circuit that executes the instructions of the scale factor technique selection circuit 230 and / or is configured to perform operations as described in the flowcharts of the Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0070] In some examples, the scale factor technique selection circuit 230 includes means for selecting a scaling technique for the scale factor determination circuit 240 that is used to perform an ML operation (e.g., a dot product operation) while maintaining high accuracy. For example, the means for selecting a scaling technique for the scale factor determination circuit 240 that is used to perform an ML operation (e.g., a dot product operation) while maintaining high accuracy may be implemented by the scale factor technique selection circuit 230. In some examples, the scale factor technique selection circuit 230 may be implemented by a processor circuit such as the example processor circuit 1212 of Fig. 12. For example, the scale factor technique selection circuit 230 may be instantiated by the exemplary microprocessor 1300 of Fig. 13 which executes machine-executable instructions as described at least in blocks 814, 902, 904, 908 and / or 1002-1012 of Fig. 8, Fig. 9 and / or 10. In some examples, the scale factor technique selection circuit 230 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the scale factor technique selection circuit 230 may be implemented by any other combination of hardware, software, and / or firmware. For example, the scale factor technique selection circuit 230 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without the execution of software or firmware, but other structures are also suitable.

[0071] The exemplary scale factor determination circuit 240 determines a scale factor to be used in converting and / or deconverting the high-precision data representation format (e.g., FP32) to the low-precision data representation format (e.g., BF16) based on the preprocessing performed by the scale factor technique selection circuit 230. In the examples disclosed herein, the scaling and / or descaling is performed by the scale factor determination circuit 240 by first calculating a target value associated with each of the FP32-represented operands, wherein the target value is obtained by masking a number of lower-order bits to reduce an associated storage requirement. In the examples disclosed herein, the lower 16 mantissa bits (e.g., the least significant bits) of each of the FP32-represented values would be masked (e.g.,set to zero) to determine each target value, as shown in example equation 4 below (where w. FP32 represents an original FP32-represented value and t FP32 its corresponding target value with the masked lower 16 mantissa bits). wFP32=0x3f937ffftFP32=0x3f9e0000

[0072] Once the respective target value has been determined by the scale factor technique selection circuit 230, a scale factor (s) is determined using Equation 5 as shown below with minimal to no loss of accuracy associated with the target value represented by BF16. s∗w=(tFP32wFP32)∗wFP32=tBF16

[0073] In the examples disclosed herein, the exponent and / or sign bits associated with each BF16 and / or FP32 are unimportant when the scale factor determination circuit 240 selects a scale factor, since each associated target value t FP32 the same sign and / or exponent as its corresponding weighting value w FP32 Therefore, the reduction of information loss, the reduction of bias voltage, and / or the reduction of truncation error related only to the lower-order mantissa bits is performed by the scale factor determination circuit 240.

[0074] Furthermore, in the examples disclosed herein, if the value w of an individual element, operand, and / or value of the scaling factor determination circuit 240 is known in advance and always remains unchanged, it is possible to find the optimal scaling factor therefor by enumerating all FP32 values (e.g., values that can be represented as discrete values on a particular computing device) between 1.0 and 2.0. By enumerating such values, the scaling factor determination circuit 240 can select a scaling factor with the least loss of accuracy for all weight values. That is, the scaling factor s can be selected by the scaling factor determination circuit 240 in these examples if abs((w * s) FP32 - (f * S) BF16). In these examples, when the value of w is known in advance to the scale factor determination circuit 240 and / or is dynamic, the mantissa bits of the resulting value (e.g., after the operation) are taken into account by the scale factor determination circuit 240. For example, Equation 6 below shows two FP32 values, which are presented below in their bitwise format. w=(−1)b31w∗2(b30wb29w…b23w)−127∗(1.b22wb21w…b0w)2s=(−1)b31s∗2(b30sb29s…b23s)−127∗(1.b22sb21s…b0s)2

[0075] If w and s from equation 6 are multiplied together, taking into account the multiplication of the values of the mantissa bits (ie (1.b22wb21w…b0w)2and(1.b22sb21s…b0s)2 ) a value of 47 (if the result is less than 2) or 48 bits (if the result is greater than or equal to 2). To return to the original FP32 value representation format, the least significant 23 bits or 24 bits in the mantissa must be omitted. When converting the FP32 result to the BF16 value representation format, the lowest 16 mantissa bits would need to be further truncated (e.g., by the scale factor determination circuit 240) to conform to the standards of the particular value representation format. That is, the value of the omitted mantissa bits (e.g., in integer form) is proportional to the loss of precision that occurs when converting the scaled FP32 value to the BF16 format (e.g., by the scale factor determination circuit 240), as shown in Equation 7 below. Therefore, the scale factor determination circuit 240 selects when enumerating all FP32 values (e.g.as discretely representable values on a particular computing device) between 1.0 and 2.0, when the input value(s) w is / are unknown, selects a scaling factor with the smallest associated decay value and, accordingly, the smallest associated loss of accuracy. Δdropped=yi&(1<<39)=((bconst_1b22wb21w…b0w)2∗(bconst_1b22sb22s…b0s)2)%239=(wm∗sm)%239

[0076] In some examples, the example scale factor determination circuit 240 is instantiated by a processor circuit that executes instructions of the scale factor determination circuit 240 and / or is configured to perform operations as illustrated by the flowcharts of Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0077] In some examples, the scale factor determination circuit 240 includes means for determining a scale factor to be used in converting and / or deconverting the high-precision data representation format (e.g., FP32) to the low-precision data representation format (e.g., BF16). For example, the means for determining a scale factor to be used in converting and / or deconverting the high-precision data representation format (e.g., FP32) to the low-precision data representation format (e.g., BF16) may be implemented by the scale factor determination circuit 240. In some examples, the scale factor determination circuit 240 may be implemented by a processor circuit such as the exemplary processor circuit 1212 of Fig. 12. For example, the scale factor determination circuit 240 may be instantiated by the exemplary microprocessor 1300 of Fig. 13 which executes machine-executable instructions as described at least in blocks 804, 812, 906, 910 and / or 912 of Fig. 8 and / or 9. In some examples, the scale factor determination circuit 240 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the scale factor determination circuit 240 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the scale factor determination circuit 240 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0078] The example model execution circuit 250 applies the weight and / or scale factor determined by the scale factor determination circuit 250 to all values used by the ML / AI model. In some examples, the example model execution circuit 250 is instantiated by a processor circuit that executes instructions of the model execution circuit 250 and / or is configured to perform operations as described in the flowcharts of Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0079] In some examples, the model execution circuit 250 includes means for applying the weighting and / or scaling factor determined by the scaling factor determination circuit 250 to all values used by the ML / AI model. For example, the means for applying the weighting and / or scaling factor determined by the scaling factor determination circuit 250 to all values used by the ML / AI model may be implemented by the model execution circuit 250. In some examples, the model execution circuit 250 may be implemented by a processor circuit such as the exemplary processor circuit 1212 of Fig. 12. For example, the model execution circuit 250 may be instantiated by the exemplary microprocessor 1300 of Fig. 13 which executes machine-executable instructions as described at least in blocks 806-810 and / or 1106-1114 of Fig. 8 and / or 11. In some examples, the model execution circuit 250 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the model execution circuit 250 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the model execution circuit 250 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0080] The example executable generation circuit 260 outputs an executable file that an accelerator (e.g., GPU, FPGA, etc.) can execute and / or instantiate to perform an ML / AI workload. In some examples, the example executable generation circuit 260 is instantiated by a processor circuit that executes instructions of the executable generation circuit 260 and / or is configured to perform operations as described in the flowcharts of the Fig. 8, Fig. 9, Fig. 10 and / or 11 are shown.

[0081] In some examples, the executable file generation circuit 260 includes means for outputting an executable file that an accelerator (e.g., GPU, FPGA, etc.) can execute and / or instantiate to perform an ML / AI workload. For example, the means for outputting an executable file that an accelerator (e.g., GPU, FPGA, etc.) can execute and / or instantiate to perform an ML / AI workload can be implemented by the mode execution circuit 260. In some examples, the model execution circuit 260 can be implemented by a processor circuit such as the example processor circuit 1212 of Fig. 12. For example, the model execution circuit 260 may be instantiated by the exemplary microprocessor 1300 of Fig. 13 which executes machine-executable instructions as described at least in blocks 816 and / or 914 of Fig. 8 and / or 9. In some examples, the model execution circuit 260 may be instantiated by a hardware logic circuit implemented by an ASIC, an XPU, or the FPGA circuit 1400 of Fig. 14, which is structured to perform operations corresponding to the machine-readable instructions. Additionally or alternatively, the model execution circuit 260 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the model execution circuit 260 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, an XPU, a comparator, an operational amplifier, a logic circuit, etc.) structured to execute some or all of the machine-readable instructions and / or to perform some or all of the operations corresponding to the machine-readable instructions without executing software or firmware, but other structures are also suitable.

[0082] In the Fig. 2, the interface circuit 210, the data accuracy selection circuit 220, the scale factor technique selection circuit 230, the scale factor determination circuit 240, the model execution circuit 250, the executable file generation circuit 260, and the data storage 270 containing the machine learning (ML) model(s) 272, the operator values 274, the scale factor 276, and the executable file(s) 278 are interconnected via bus 280. Bus 280 may be implemented, for example, by an I2C (Inter-Integrated Circuit) bus, an SPI (Serial Peripheral Interface) bus, a PCI (Peripheral Component Interconnect) bus, or a PCIe or PCIE (Peripheral Component Interconnect Express) bus. Additionally or alternatively, bus 280 may be implemented by any other type of data processing or electrical bus.

[0083] In the examples disclosed herein, the example data store 270 may be any type of data store, database, etc., that includes the example machine learning (ML) model(s) 272 and the example operator values 274 as obtained from the interface circuit 210, the example scaling factors 276 as determined by the scaling factor technique selection circuit 230 and the scaling factor determination circuit 240, and the example executable file(s) 278 as used and / or generated by the model execution circuit 250 and the executable file generation circuit 260.

[0084] While an example implementation of the accelerator compiler 104A-C of Fig. 1 in Fig. 2, one or more of the Fig. 2 may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Furthermore, the example interface circuit 210, the example data accuracy selection circuit 220, the example scale factor technique selection circuit 230, the example scale factor determination circuit 240, the example model execution circuit 250, the example executable file generation circuit 260, and / or more generally, the example accelerator compiler 104A-C of Fig. 1 may be implemented by hardware alone or by hardware in combination with software and / or firmware. For example, any of the example interface circuitry 210, the example data accuracy selection circuitry 220, the example scale factor technique selection circuitry 230, the example scale factor determination circuitry 240, the example model execution circuitry 250, the example executable file generation circuitry 260, and / or more generally, the example accelerator compiler 104A-C of Fig. 1 may be implemented by processor circuits, analog circuits, digital circuits, logic circuits, programmable processors, programmable microcontrollers, graphics processing units (GPUs), digital signal processor(s) (DSP), application-specific integrated circuit(s) (ASIC), programmable logic devices (PLD), and / or field-programmable logic devices (FPLD) such as field-programmable gate arrays (FPGAs). Furthermore, the example accelerator compiler 104A-C of Fig. 1 one or more elements, processes and / or devices in addition to or instead of the Fig. 2 and / or may include more than one or all of the elements, processes and devices shown.

[0085] Fig. 3 is an illustration of an exemplary conventional convolution operation 300 performed by the first acceleration circuit 108 of Fig. 1, the second acceleration circuit 110 of Fig. 1 and / or the accelerator compiler 200 of Fig. 2. In some examples, the conventional convolution operation 300 may perform a spatial convolution over one or more images (e.g., an image, a still image of a video, etc.) and / or operations (e.g., dot product operations). In some examples, the accelerator compiler 300 may be configured to operate in a conventional convolution mode, a 2D convolution mode, a three-dimensional (3D) convolution mode, etc., based on the conventional convolution operation performed by the accelerator compiler 200.

[0086] The conventional convolution operation 300 includes applying exemplary filters 302 to an exemplary input tensor 304 to generate an exemplary output tensor 306. In this example, the input tensor 304 is a 3-D object with a size of x i * y i * z i. In this example, there are K filters 302, and each of the filters 402 has a size of f * f * z k Alternatively, any other size may be used to implement one or more of the filters 302. For example, one or more of the filters 402 may have a size of f x * f y * z k where x and y can be different and thus also f x and f y may be different. In this example, the filters 302 are quadratic filters, so that f x equal to f y but the examples described here are not so limited. In this example, the output tensor 306 has a size of x o * y o * z o . In this example, z k = z i and z o= K. In this example, the filters 402 are applied to the input tensor 404 along with a nonlinear activation function to generate the output tensor 306. For example, the accelerator compiler 200 of Fig. 2, receive the input tensor 304 as activation data, receive one of the filters 302 as weight data, and output the output tensor 306. In some such examples, the accelerator compiler 200 may implement the conventional convolution operation 300 in a "dense" manner, while in other examples, the accelerator compiler 200 may implement the conventional convolution operation 300 using sparsity.

[0087] Advantageously, the accelerator compiler 300 may perform the convolution operation 300 based on sparse data to reduce the number of computations. For example, the accelerator compiler 200 may obtain and / or generate activation sparsity data and / or weight sparsity data to output the output tensor 306 by applying sparsity techniques.

[0088] Fig. 4 shows a bitwise binary representation 400 of an exemplary FP32 value. In the illustrated bitwise binary representation 400, 1 sign bit 402, 8 exponent bits 404, and 23 fractional (mantissa) bits 406 are shown along with their corresponding bit index 408. In this illustrated example, the leftmost bits (e.g., sign bit 402, exponent bits 404), which are enumerated as higher values according to bit index 408, are not considered for scaling by the scale factor technique selection circuit 230 and / or the scale factor determination circuit 240, as explained in more detail herein. The lowest order bits, as represented by the lower values enumerated by bit index 408 (e.g., mantissa bits 406), are masked and / or truncated by scale factor technique selection circuit 230 and / or scale factor determination circuit 240 when scaling between data representation formats. Fig. 5 shows an exemplary enumeration process for selecting a scale factor as performed by the exemplary scale factor determination circuit 240 of Fig. 2. As in connection with Fig. 2, the input value of a single element, operand and / or value is determined by the exemplary scale factor determination circuit 240 of Fig. 2 is known before training and / or inference and always remains unchanged, then it is possible to find the optimal scaling factor for the value(s) by enumerating all FP32 values (e.g., as discretely representable values on a particular computing device) between 1.0 and 2.0. By enumerating such values, the scaling factor determination circuit (240) of Fig. 2 select a scaling factor with the smallest loss of accuracy. Fig. The example illustrated in Figure 5 shows a scale factor enumeration chart 500 in which a dropped integer value 502 is plotted against an enumerated scale factor 504. In the examples disclosed herein, the dropped integer value 502 represents the value dropped from the value when converting to a less precise data representation format (e.g., integer form) and is proportional to the loss of precision when converting between data representation formats. The scale factor enumeration chart 500 is a periodic variation graph for any input value (e.g., 1, 66666667) that shows a trend of decreasing integer values 502 as the enumerated scale factor 504 changes. The point of minimum drop deviation 506, as shown in the Fig. 5, represents a point of least observed loss of the interview value 502, and accordingly, the point of minimum drop deviation 506 also represents the point of least loss of accuracy. Thus, the enumerated x-axis scaling factor 504 at this coordinate would be determined by the scaling factor determination circuit 240 in Fig. 2 is selected as the scaling factor to be applied to the ML / AI model values

[0089] Fig. 6 shows an example comparative accuracy graph 600 for example dot product operations using FP32 base values 602, converted BF16 values 604, and scaled BF16 values 606. As shown in the comparative accuracy graph 600, dot product operations performed with the scaled BF16 values 606 exhibit the greatest deviation in accuracy compared to the FP32 base values 602. The comparative accuracy chart 600 further shows that the accuracy results with respect to the scaled BF16 values 606 correspond more closely to the FP32 base values than the converted BF16 values 604, indicating a higher degree of accuracy achieved by the scale factor techniques described herein when using lower precision data representation formats to increase computational efficiency and / or reduce resource overhead.

[0090] Fig. 7 shows an example table 700 for the percentage accuracy across a set of tested machine learning (ML) and / or artificial intelligence (AI) models 702 for example operations (e.g., dot product operations) using base FP32 values 602 (from Fig. 6), converted BF16 values 604 (from Fig. 6), a first technique with scaled BF16 values 704 and a second technique with scaled BF16 values 706. As in connection with Fig. 2, the first technique of scaled BF16 values 704 corresponds to the BF16 values obtained from the scale factor technique selection circuit 230 and / or the scale factor determination circuit 240 of Fig. 2 when the input values (w) are known. Similarly, the second technique of scaled BF16 values 706 corresponds to the BF16 values obtained by the scale factor technique selection circuit 230 and / or the scale factor determination circuit 240 of Fig. 2 if the input values (w) are known. The right column of the accuracy table 700 shows the average accuracy 708 for all tested ML / AI models 702. The average accuracy 708 in conjunction with the FP32 base values 602 represents an ideal accuracy value for the tested ML / AI models 702. Both the first technique of scaled BF16 values 704 and the second technique of scaled BF16 values 706 have an associated average accuracy 708 that is greater than that of the converted BF16 values 604 (e.g., they are closest to the average accuracy 708 of the FP32 base values 602), indicating high operational accuracy even by using the lower-accuracy data representation format (e.g., BF16) through the scaling process (e.g., both the first technique of scaling as well as the second scaling technique).

[0091] Flowcharts illustrating exemplary machine-readable instructions that may be executed to configure the processor circuitry to execute the accelerator compiler 200 of Fig. 2 are to be implemented in Fig. 8-11. The machine-readable instructions may be one or more executable programs or portions of an executable program that can be executed by a processor circuit, such as the processor circuit 1212 shown in the example processor platform 1200 described below in connection with Fig. 12 and / or the exemplary processor circuit described below in connection with Fig. 13 and / or 14. The program may be embodied in software stored on one or more non-transitory, computer-readable storage media such as a compact disc (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disc (DVD), a Blu-ray disc, volatile memory (e.g., random access memory (RAM) of any type, etc.), or non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), FLASH memory, a hard disk, an SSD, etc.) connected to processor circuitry in one or more hardware devices, but all of the program and / or portions thereof could alternatively be executed by one or more hardware devices other than the processor circuitry and / or embodied in firmware or dedicated hardware.The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). The client hardware device may be implemented, for example, by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a RAN gateway that enables communication between a server and an endpoint client hardware device). Likewise, the non-transitory computer-readable storage media may comprise one or more media residing in one or more hardware devices. Although the example program is described with reference to the embodiments described in . Fig. 8-11, many other methods may also be used to implement the example accelerator compiler 200. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, an FPGA, an ASIC, a comparator, an operational amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware. The processor circuits may be distributed across various networks and / or locally on one or more hardware devices (e.g., a single-core processor (e.g.,a single-core central processing unit (CPU), a multi-core processor (e.g., a multi-core processor (e.g., a multi-core CPU), etc.) in a single machine, multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, a CPU and / or an FPGA residing in the same chassis (e.g., in the same IC package or in two or more separate chassis, etc.).

[0092] The machine-readable instructions described herein may be stored in one or more of the following formats: compressed format, encrypted format, fragmented format, compiled format, executable format, packaged format, etc. Machine-readable instructions as described herein may be stored as data or a data structure (e.g., as pieces of instructions, code, representations of code, etc.) that can be used to generate, manufacture, and / or produce machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in the cloud, on edge devices, etc.).The machine-readable instructions may require one or more of the following operations: installation, modification, adaptation, updating, combining, supplementing, configuring, decrypting, decompressing, unpacking, distributing, remapping, compiling, etc., to make them directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple pieces that are individually compressed, encrypted, and / or stored on separate computers, which pieces, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that together may form a program such as the one described herein.

[0093] In another example, the machine-readable instructions may be stored in a state where they can be read by processor circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., settings saved, data entered, network addresses recorded, etc.) before the machine-readable instructions and / or the corresponding program(s) can be executed, in whole or in part.Therefore, machine-readable media as used herein may contain machine-readable instructions and / or programs, regardless of the particular format or state of the machine-readable instructions and / or programs when stored or otherwise at rest or in transit.

[0094] The machine-readable instructions described herein may be represented by any past, present, or future command language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0095] As mentioned above, the exemplary processes of Fig. 8-11 may be implemented using executable instructions (e.g., computer- and / or machine-readable instructions) stored on one or more non-transitory computer- and / or machine-readable media, such as optical storage devices, magnetic storage devices, a hard disk, flash memory, read-only memory (ROM), a CD, a DVD, a cache, RAM of any kind, a register, and / or any other storage device or disk in which information is stored for any duration (e.g., for extended periods of time, permanently, for short periods of time, for temporary caching and / or for caching information).As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and non-transitory machine-readable storage medium are expressly defined to include any type of computer-readable storage device and / or storage disk, and exclude the transmission of signals and transmission media. The terms "computer-readable storage medium" and "machine-readable storage medium" are defined herein to include any physical (mechanical and / or electrical) structure for storing information, but exclude the transmission of signals and transmission media.Examples of computer-readable storage devices and machine-readable storage devices include random access memory of any kind, solid-state memory, flash memory, optical disks, magnetic disks, hard disk drives, and / or redundant array of independent disks (RKID). As used herein, the term "device" refers to physical structures such as mechanical and / or electrical equipment, hardware, and / or circuitry that may or may not be configured by and / or fabricated to execute computer-readable instructions, machine-readable instructions, etc.

[0096] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended expressions. Thus, whenever a claim employs any form of "include" or "comprising" (e.g., comprises, contains, comprising, containing, having, etc.) as a preamble or within a claim recitation of any kind, it is understood that additional elements, expressions, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" ("at least") is used as a transitional expression, for example, in a preamble to a claim, it is open-ended in the same way that the expressions "comprising" and "containing" are open-ended. The term "and / or," when used, for example,used in a form such as A, B and / or C refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A with B, (5) A with C, (6) B with C, or (7) A with B and with C. As used herein in the context of describing structures, components, items, objects and / or things, the phrase "at least one of A and B" is intended to refer to implementations that include (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects and / or things, the phrase "at least one of A or B" is intended to refer to implementations that include (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.As used herein in the context of describing the performance or execution of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Likewise, as used herein in the context of describing the performance or execution of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any element of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0097] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not preclude plurality. The term "a" or "an" object, as used herein, refers to one or more numbers of that object. The terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably herein. Moreover, a plurality of means, elements, or method acts, even if listed individually, may be performed by the same entity or object, for example. In addition, although individual features may be included in different examples or claims, they may be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0098] Fig. 8 is a flowchart illustrating example machine-readable instructions and / or example operations 800 that may be executed and / or instantiated by a processor circuit to perform a scale shift of formatted values of the lower-precision data representation to achieve higher accuracy and / or efficiency of computation. The machine-readable instructions and / or operations 800 of Fig. 8 begin at block 802, where the data accuracy selection circuit 220 identifies a first accuracy data type and a second accuracy data type associated with the execution of a machine learning (ML) model by the acceleration hardware (e.g., the accelerator compiler 200). As discussed herein, the data representation formats used by the accelerator compiler 200 may vary depending on hardware and / or execution constraints. In the examples disclosed herein, FP32 and BF16 data representation formats are described, with the former being the high-accuracy data representation format and the latter being the low-accuracy data representation format. However, in other examples, any other high- and low-accuracy data types may be identified by the scale factor determination circuit 240 at block 802.

[0099] In block 804, the model execution circuit 250 determines scaling factors to be applied to first weights having the first accuracy data type of the machine learning (ML) model. The first accuracy data type, in the examples disclosed herein, may be either the high-accuracy or low-accuracy data type, depending on the application. The first weight determined by the model execution circuit 250 is the target value, as described in connection with Fig. 2. In the examples disclosed herein, the target value is determined by the model execution circuit 250 from Fig. 2 calculated and / or determined if the input values (w) are known before the ML model is executed.

[0100] In block 806, the model execution circuit 250 converts the first weights into second weights with the second accuracy data type based on a multiplication of the first weights and the scaling factor(s). As described herein (in connection with Fig. 2) As described in more detail, the second weights may be calculated (e.g., by the scale factor determination circuit 240 and / or the model execution circuit 250) by enumerating all possible scale factors and determining the scale factor that results in the least loss of accuracy when multiplying values.

[0101] At block 808, the model execution circuit 250 executes the machine learning (ML) model based on the second weight(s). In the examples disclosed herein, the same weight may be calculated and / or applied to all values used by the ML model. However, in other examples, a plurality of weights may be calculated and / or applied based on a particular loss of accuracy caused by each value.

[0102] At block 810, the scaling factor technique selection circuit 230 determines whether the accuracy of the machine learning (ML) model has improved beyond a threshold. If the scaling factor technique selection circuit 230 determines that the accuracy of the ML model has improved beyond a threshold, the process proceeds to block 812. However, if the scaling factor technique selection circuit 230 determines that the accuracy of the ML model has not improved beyond a threshold, the process proceeds to block 814.

[0103] In block 812, in response to the determination by the scaling factor technique selection circuit 230 in block 810 that the accuracy of the ML model has improved beyond a threshold, the scaling factor determination circuit 240 adjusts the scaling factor(s) to achieve even higher accuracy. As described in connection with Fig. 2, the scale factor determination circuit 240 adjusts the scale factor(s) by enumerating all scale values to determine the next best value with the least loss of accuracy.

[0104] At block 814, in response to the scaling factor technique selection circuit 230 determining that the accuracy of the ML model has not improved beyond a threshold established in block 810, the scaling factor technique selection circuit 230 selects a different method for determining the scaling factor(s). An example of a first method is enumerating all possible scaling factors and selecting the factor that best reduces the loss of accuracy, e.g., by the scaling factor technique selection circuit 230. If the scaling factor technique selection circuit 230 determines that a different technique should be selected for determining the scaling factor(s), the process returns to block 804. However, if the scaling factor technique selection circuit 230 determines that no other technique should be selected, the process proceeds to block 816.

[0105] At block 816, the executable generation circuitry 260 outputs an executable based on the machine learning (ML) model, the scaling factor(s), and the second weights. In the examples disclosed herein, this executable may represent an optimal set of weights and / or scaling factor(s) used to achieve improved computational performance with reduced resource overhead and high operational accuracy.

[0106] Fig. 9 is a flowchart illustrating example machine-readable instructions and / or example operations 900 that may be executed and / or instantiated by a processor circuit to perform a scale shift for all values involved in operations of an ML / AI model. The machine-readable instructions and / or operations 900 of Fig. 9 begin in block 902, in which the scale factor technique selection circuit 230 selects a scale factor selection technique. The process of selecting a scale factor selection technique by the scale factor selection circuit 230 is described in connection with Fig. 10 further described.

[0107] In block 904, the scale factor technique selection circuit 230 determines whether the scale factor selection technique is based on a scale factor per tensor, as may be indicated in the examples disclosed herein by example data stored in a data store associated with scale factor(s), etc. (e.g., data store 270 of Fig. 2). If the scale factor technique selection circuit 230 determines that the scale factor selection technique is based on a per-tensor scale factor, the process proceeds to block 906. However, if the scale factor technique selection circuit 230 determines that the scale factor selection technique is not based on a per-tensor scale factor, the process proceeds to block 908.

[0108] In block 906, the scale factor determination circuit 240 determines a scale factor to be used per tensor. In the examples disclosed herein, the scale factor determination circuit 240 may make this determination based on hardware limitations of a particular tensor, associated available data representation formats, etc.

[0109] At block 908, after determining at block 90 that the scaling factor technique is not based on a per-tensor scaling factor, the scaling factor technique selection circuit 230 determines whether the scaling factor technique is based on a per-channel scaling factor. If the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is based on a per-channel scaling factor, the process proceeds to block 910. However, if the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is not based on a per-channel scaling factor, the process proceeds to block 912.

[0110] In block 910, the scale factor determination circuit 240 determines a scale factor to be used per channel. Similar to the determination by the scale factor determination circuit 240 in block 906, the scale factors for all tensors within a particular channel may be aggregated, averaged, etc., to determine a single scale factor to be used per channel.

[0111] In block 912, the scaling factor determination circuit (240) outputs the scaling factor(s) to be used for executing the machine learning (ML) model. In the examples disclosed herein, the scaling factor(s) may be stored in an example database (e.g., data store 270 of Fig. 2).

[0112] In block 914, the executable file generation circuitry 260 outputs an executable file based on the machine learning (ML) model and the scaling factor(s). In the examples described herein, both the ML model and the scaling factor(s) may be stored in the example data store 270 of Fig. 2. In the examples described here, the executable file may be specific to the hardware constraints specified with the respective ML model, but in some examples, it can be generalized to different types of ML models.

[0113] Fig. 10 is a flowchart illustrating example machine-readable instructions and / or example operations 1000 that may be executed and / or instantiated by a processor circuit to select a scale factor selection method to be applied to determine a scale factor for all values used for operations by an ML / AI model. The machine-readable instructions and / or operations 1000 of Fig. 10 begin at block 1002, where the scaling factor technique selection circuit 230 determines whether a scaling factor based on reducing the total absolute delta values between the scaled FP32 values and the converted lower-precision data types with a scaling factor is the preferred scaling factor selection. In the examples disclosed herein, the scaling factor technique selection circuit 230 may select a particular scaling factor technique based on predetermined hardware constraints for a particular ML model, a particular workload range, a type of data representation formats provided by a particular ML model and / or set of operands, etc.

[0114] In block 1004, the scaling factor technique selection circuit 230 determines whether to select the scaling factor based on reducing the total absolute delta values between the scaled FP32 values and the converted values of the lower precision data type with a scaling factor for the top n values. In the examples disclosed herein, the top n values may represent a square root of the number of total dimensions of a set of values, indicating a sampling algorithm of n values to which a scaling factor is applied.

[0115] In block 1006, the scaling factor technique selection circuit 230 determines whether the scaling factor to be selected is based on minimizing a weighted absolute total set of delta values between the scaled FP32 values and the converted lower precision data type with the scaling factor for the top n values. Similar to the determination by the scaling factor technique selection circuit 230 in block 1004, the scaling factor technique selection circuit 230 determines in block 1006 whether to reduce the weighted absolute total delta values. In the examples disclosed herein, the weighted absolute total delta values represent a higher precision, which can be achieved by assigning logarithmically smaller weights in descending order. For the lower half of the values (between n / 2 above and n above) the weighting is 1, for half of the values above (between n / 4 above and n / 2 above) the weighting is 2, and so on.So the weight for a top 1 value is (1 + log2 n), which is a technique that increases accuracy without adjusting the scale too much.

[0116] In block 1008, the scale factor technique selection circuit 230 determines whether to select the scale factor based on minimizing the total absolute delta values between the scaled FP32 values and the converted values of the lower precision data type with the scale factor to reduce the variation in the delta values, which is a further variation that provides a greater degree of accuracy.

[0117] In block 1010, the scale factor technique selection circuit 230 selects a scale factor based on a different method than the methods of blocks 1002 through 1008.

[0118] In block 1012, the scale factor technique selection circuit 230 outputs the desired scale factor technique.

[0119] Fig. 11 is a flowchart illustrating example machine-readable instructions and / or example operations 1100 that may be executed and / or instantiated by a processor circuit to perform lower-precision value descaling once an operation performed by an ML / AI model is complete. The machine-readable instructions and / or operations 1100 of Fig. 11 begin in block 1102, in which the interface circuit 210 receives one (or more) operators to perform a machine learning operation. In the examples disclosed herein, the operators may be stored in a database (e.g., in the data store 270 of Fig. 2) or be provided together with a specific ML model and / or operation of choice.

[0120] In block 1104, the data precision selection circuit 220 determines whether the operators are based on a lower precision data type. In the examples disclosed herein, some operators that are available via a database (e.g., data store 270 in Fig. 2), etc., may not be further scaled to a lower precision data type if they are already provided as a low precision data type and / or cannot be further scaled down. If the model execution circuitry 250 determines that the operators are based on a lower precision data type, the process proceeds to block 1106. However, if the model execution circuitry 250 determines that the operators are not based on a lower precision data type, the process proceeds to block 1108.

[0121] In block 1106, the model execution circuitry 250 scales the operators using scaling factor(s). In the examples disclosed herein, the scaling factor(s) may be provided as weighted factor(s) that is / are imputed via an ML model and / or multiplied by each of the operator values.

[0122] In block 1108, the model execution circuit 250 determines the output values of the machine learning (ML) operation based on the scaled operators. In the examples presented here, these output values may be the result of the operation performed on the scaled operators.

[0123] In block 1110, the model execution circuit 250 scales the output values determined by the model execution circuit 250 in block 1108. In the examples described herein, the scaling is removed, for example, by dividing the output values by the same scaling factor (e.g., multiplying by the inverse) by which they were multiplied during scaling, effectively reversing the scaling process.

[0124] In block 1112, the model execution circuit 250 outputs the descaled output values to another logical unit. In the examples disclosed herein, the logical units may be, for example, the external computing devices 130 of Fig. 1 or any other type of computing device and / or logical unit.

[0125] At block 1114, model execution circuitry 250 determines whether to perform another machine learning (ML) operation (e.g., using the same and / or different operators). If model execution circuitry 250 determines that another ML operation should be performed, the process returns to block 1102. However, if model execution circuitry 250 determines that no further ML operation should be performed, the process ends.

[0126] Fig. 12 is a block diagram of an exemplary processor platform 1200 structured to implement the machine-readable instructions and / or operations of Fig. 8-11 for the implementation of the accelerator compiler 200 of Fig. 2. The processor platform 1200 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smartphone, a tablet such as an iPad™), a personal digital assistant (PDA), an internet device, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.), or another portable device or any other type of computing device.

[0127] The processor platform 1200 of the illustrated example includes a processor circuit 1212. The processor circuit 1212 of the illustrated example is hardware. For example, the processor circuit 1212 may be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers of any family or manufacturer. The processor circuit 1212 may be implemented by one or more semiconductor devices (e.g., silicon-based). In this example, the processor circuit 1212 implements the example interface circuit 210, the example data accuracy selection circuit 220, the example scale factor technique selection circuit 230, the example scale factor determination circuit 240, the example model execution circuit 250, and / or the example executable file generation circuit 260.

[0128] The processor circuit 1212 of the illustrated example includes a local memory 1213 (e.g., a cache, registers, etc.). The processor circuit 1212 of the illustrated example is in communication with a main memory, which includes a volatile memory 1214 and a non-volatile memory 1216, via a bus 1218. The volatile memory 1214 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and / or any other type of RAM device. The non-volatile memory 1216 may be implemented by Flash memory and / or any other desired type of memory device. Access to the main memory 1214, 1216 of the illustrated example is controlled by a memory controller 1217.

[0129] The processor platform 1200 of the illustrated example also includes the interface circuit 1220. The interface circuit 1220 may be implemented in hardware in accordance with any interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth® interface, a Near Field Communication (NFC) interface, a Peripheral Component Interconnect (PCI) interface, and / or a Peripheral Component Interconnect Express (PCIe) interface.

[0130] In the illustrated example, one or more input devices 422 are connected to the interface circuitry 1220. The input device(s) 1222 allow a user to input data and / or instructions into the processor circuitry 1212. The input device(s) 1222 may be implemented, for example, by an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a trackpad, a trackball, an isopoint, and / or a speech recognition system.

[0131] One or more output devices 1224 are also connected to the interface circuit 1220 of the illustrated example. The output device(s) 1224 may be implemented, for example, by display devices (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1220 of the illustrated example therefore typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit such as a GPU.

[0132] The interface circuit 1220 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface, to facilitate the exchange of data with external machines (e.g., computing devices of any type) over a network 1226. The communication may occur, for example, via an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a wireless line-of-site system, a cellular phone system, an optical connection, etc.

[0133] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 for storing software and / or data. Examples of such mass storage devices 1228 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray Disc drives, RAID (Redundant Array of Independent Disks) systems, solid-state storage devices such as flash memory devices, and DVD drives.

[0134] The machine-executable instructions 1232 that are replaced by the machine-readable instructions of Fig. 8-11 may be stored in mass storage 1228, volatile memory 1214, non-volatile memory 1216, and / or on a removable, non-volatile, computer-readable storage medium such as a CD or DVD.

[0135] Fig. 13 is a block diagram of an exemplary implementation of the processor circuit 1212 of Fig. 12. In this example, the processor circuit 1212 is Fig. 12 is implemented by a microprocessor 1300. The microprocessor 1300 may be, for example, a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 1300 executes some or all of the machine-readable instructions of the flowcharts of Fig. 8-11 to switch the circuit from Fig. 2 [diagram] effectively instantiate as logic circuits and perform the operations corresponding to these machine-readable instructions. In some of these examples, the accelerator compiler 200 is used by Fig. 2 instantiated by the hardware circuitry of the microprocessor 1300 in combination with the instructions. The microprocessor 1300 may be implemented, for example, by multi-core hardware circuitry such as a CPU, a DSP, a GPU, an XPU, etc. Although it may include any number of example cores 1302 (e.g., 1 core), the microprocessor 1300 of this example is a multi-core semiconductor device with N cores. The cores 1302 of the microprocessor 1300 may operate independently of one another or cooperate to execute machine-readable instructions. For example, machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one or more of the cores 1302 at the same or different times.In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is divided into threads and executed in parallel by two or more cores 1302. The software program may correspond to some or all of the machine-readable instructions and / or operations depicted in the flowchart of FIG. Fig. 8-11 are shown.

[0136] The cores 1302 may communicate via a first example bus 1304. In some examples, the first bus 1304 may be implemented by a communication bus to enable communication in connection with one or more cores 1302. The first bus 1304 may be implemented, for example, by an I2C (Inter-Integrated Circuit) bus, a SPI (Serial Peripheral Interface) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1304 may be implemented by any other type of computer or electrical bus. The cores 1302 may receive data, commands, and / or signals from one or more external devices via an example interface circuit 1306. The cores 1302 may output data, instructions, and / or signals to one or more external devices via the interface circuit 1306. Although the cores 1302 of this example may include an example local memory 1320 (e.g.,Level 1 (L1) cache, which may be divided into an L1 data cache and an L1 instruction cache), the microprocessor 1300 also includes an example shared memory 1310 that may be shared by the cores (e.g., Level 2 (L2 cache)) to enable fast access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing to and / or reading from the shared memory 1310. The local memory 1320 of each of the cores 1302 and the shared memory 1310 may be part of a hierarchy of memory devices that may include multiple levels of cache memories and main memory (e.g., main memory 1214, 1216 of FIG. Fig. 12). Typically, higher memory levels in the hierarchy have lower access times and smaller storage capacities than lower memory levels. Changes at the different levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0137] Each core 1302 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuitry. Each core 1302 includes a control unit 1314, an arithmetic and logic (AL) circuit (sometimes referred to as an ALU) 1316, a plurality of registers 1318, the local memory 1320, and a second example bus 1322. Other structures may be present. For example, each core 1302 may include circuitry for a vector unit, a single instruction multiple data (SIMD) unit, a load / store unit (LSU) circuit, a branch / jump unit, a floating point unit (FPU) circuit, etc. The control unit circuitry 1314 includes semiconductor-based circuitry structured to control (e.g., coordinate) data movement within the corresponding core 1302.The AL circuit 1316 includes semiconductor-based circuitry structured to perform one or more mathematical and / or logical operations on the data in the corresponding core 1302. The AL circuit 1316 of some examples performs integer operations. In other examples, the AL circuit 1316 also performs floating-point operations. In other examples, the AL circuit 1316 may include a first AL circuit that performs integer-based operations and a second AL circuit that performs floating-point operations. In some examples, the AL circuit 1316 may be referred to as an arithmetic logic unit (ALU). Registers 1318 are semiconductor-based structures for storing data and / or instructions, such as the results of one or more of the operations performed by the AL circuit 1316 of the corresponding core 1302.Registers 1318 may include, for example, vector registers, SIMD registers, general-purpose registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. Registers 1318 may be arranged in a bank, as shown in FIG. Fig. 13. Alternatively, the registers 1318 may be organized in a different arrangement, format, or structure, including being distributed throughout the core 1302 to reduce access time. The second bus 1322 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0138] Each core 1302 and / or, more generally, the microprocessor 1300 may include additional and / or different structures than those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more converged / common mesh stops (CMSs), one or more shifters (e.g., barrel shifters), and / or other circuitry may be present. The microprocessor 1300 is a semiconductor device fabricated to include many transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuitry may include and / or cooperate with one or more accelerators.In some examples, accelerators are implemented by logic circuitry to perform certain tasks faster and / or more efficiently than is possible with a general-purpose processor. Examples of accelerators include ASICs and FPGAs, as described herein. A GPU or other programmable device can also be an accelerator. Accelerators can be located on the board of the processor circuitry, in the same die package as the processor circuitry, and / or in one or more packages separate from the processor circuitry.

[0139] Fig. 14 is a block diagram of another exemplary implementation of the processor circuit 1212 of Fig. 12. In this example, the processor circuit 1212 is implemented by an FPGA circuit 1400. For example, the FPGA circuit 1400 may be implemented by an FPGA. The FPGA circuit 1400 may, for example, be used to perform operations that would otherwise be performed by the example microprocessor 1300 of Fig. 13, which executes corresponding machine-readable instructions. However, once configured, the FPGA circuit 1400 instantiates the machine-readable instructions in hardware and can therefore often perform the operations faster than would be possible with a general-purpose microprocessor executing the corresponding software.

[0140] More precisely, in contrast to the above-described microprocessor 1300 from Fig. 13 (which is a general-purpose device that can be programmed to carry out any or all of the machine-readable instructions represented by the flowchart in Fig. 8-11, but whose connections and logic circuits are fixed after manufacture), the FPGA circuit 1400 of the example of Fig. 14 Connections and logic circuits that can be configured and / or connected in different ways after manufacture, for example to instantiate some or all of the machine-readable instructions represented by the flowchart in Fig. 8-11. In particular, the FPGA circuit 1400 can be thought of as an arrangement of logic gates, interconnects, and switches. The switches can be programmed to change the way the logic gates are connected to each other by the interconnects, effectively forming one or more dedicated logic circuits (unless the FPGA circuit 1400 is reprogrammed). The configured logic circuits allow the logic gates to work together in different ways to perform various operations on the data received from the input circuits. These operations may correspond to some or all of the software described in the flowchart of Fig. 8-11. The FPGA circuit 1400 may be structured to implement some or all of the machine-readable instructions of the flowchart in Fig. 8-11 as dedicated logic circuits to perform the operations corresponding to these software instructions analogously to an ASIC. Therefore, the FPGA circuit 1400 can perform the operations corresponding to some or all of the machine-readable instructions of Fig. 8-11 faster than the general-purpose microprocessor can.

[0141] In the example of Fig. 14, the FPGA circuit 1400 is structured so that it can be programmed (and / or reprogrammed one or more times) by an end user using a hardware description language (HDL) such as Verilog. The FPGA circuit 1400 of Fig. 14 includes an example input / output circuit 1402 for receiving and / or outputting data from / to an example configuration circuit 1404 and / or external hardware 1406. For example, the configuration circuit 1404 may implement an interface circuit that may receive machine-readable instructions for configuring the FPGA circuit 1400 or portions thereof. In some examples, the configuration circuit 1404 may receive the machine-readable instructions from a user, a machine (e.g., hardware circuits (e.g., programmed or dedicated circuits) that may implement an artificial intelligence / machine learning (AI / ML) model to generate the instructions), etc. In some examples, the external hardware 1406 may be implemented by external hardware circuits. The external hardware 1406 may, for example, be implemented by the microprocessor 1300 of Fig. 13. The FPGA circuit 1400 also includes an array of example logic gate circuits 1408, a plurality of configurable example interconnects 1410, and example memory circuits 1412. The logic gate circuit 1408 and the configurable interconnects 1410 are configurable to instantiate one or more operations corresponding to at least some of the machine-readable instructions of Fig. 8-11 and / or other desired operations. The Fig. The logic gate circuit 1408 shown in Figure 14 is fabricated in groups or blocks. Each block includes semiconductor-based electrical structures that can be configured into logic circuits. In some examples, the electrical structures include logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that represent basic building blocks for logic circuits. Electrically controllable switches (e.g., transistors) are present in each of the logic gate circuits 1408 to enable the configuration of the electrical structures and / or the logic gates to form circuits for performing desired operations. The logic gate circuit 1408 may include other electrical structures such as look-up tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0142] The configurable connections 1410 of the illustrated example are conductive paths, traces, vias, or the like that may include electrically controllable switches (e.g., transistors) whose state may be changed by programming (e.g., using an HDL instruction language) to enable or disable one or more connections between one or more of the logic gate circuits 1408 to program desired logic circuits.

[0143] The memory circuit 1412 of the illustrated example is structured to store the result(s) of one or more of the operations performed by the corresponding logic gates. The memory circuit 1412 can be implemented using registers or the like. In the illustrated example, the memory circuits 1412 are distributed among the logic gate circuits 108 to facilitate access and increase execution speed.

[0144] The exemplary FPGA circuit 1400 from Fig. 14 also includes an example dedicated operating circuit 1414. In this example, the dedicated operating circuit 1414 includes special-purpose circuitry 1416 that can be called to implement frequently used functions so that these functions do not need to be programmed in-place. Examples of such special-purpose circuitry 1416 include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of special-purpose circuitry may be present. In some examples, the FPGA circuitry 1400 may also include a programmable general-purpose circuitry 1418, such as an example CPU 1420 and / or an example DSP 1422. Additionally or alternatively, other programmable circuitry 1418 may be present, such as a GPU, an XPU, etc., that can be programmed to perform other operations.

[0145] Although Fig. 13 and Fig. 14 two exemplary implementations of the processor circuit 1212 of Fig. 12, many other approaches are conceivable. As already mentioned, modern FPGA circuits may, for example, include an integrated CPU, such as one or more of the exemplary CPUs 1420 in Fig. 6. Therefore, the processor circuit 1212 of Fig. 12 additionally by combining the exemplary microprocessor 1300 of Fig. 13 and the exemplary FPGA circuit 1400 of Fig. 14. In some such hybrid examples, a first part of the machine-readable instructions represented by the flowcharts of Fig. 8-11, from one or more of the cores 1302 in Fig. 13, a second part of the machine-readable instructions represented by the flowcharts of Fig. 8-11 can be processed by the FPGA circuit 1400 in Fig. 14, and / or a third part of the machine-readable instructions represented by the flowcharts of Fig. 8-11 can be executed by an ASIC. It is understood that some or all components of the accelerator compiler 200 may be Fig. 2 may therefore be instantiated at the same or different times. For example, some or all of the circuits may be instantiated in one or more threads that execute concurrently and / or in series. Furthermore, in some examples, part or all of the accelerator compiler 200 may be Fig. 2 be implemented in one or more virtual machines and / or containers running on the microprocessor.

[0146] In some examples, the processor circuit 1212 may be Fig. 12 may be housed in one or more housings. For example, the microprocessor 1300 of Fig. 13 and / or the FPGA circuit 1400 of Fig. 14 may be housed in one or more packages. In some examples, an XPU may be implemented by the processor circuitry 1212 of Fig. 12, which can be located in one or more packages. For example, the XPU can contain a CPU in one package, a DSP in another package, a GPU in another package, and an FPGA in another package.

[0147] A block diagram illustrating an example software distribution platform 1505 for distributing software such as the example machine-readable instructions 1232 of Fig. 12 on hardware devices owned and / or operated by third parties is in Fig. 15. The example software distribution platform 1505 may be implemented by any computer server, data facility, cloud service, etc. capable of storing and transmitting software to other computing devices. The third parties may be customers of the company that owns and / or operates the software distribution platform 1505. For example, the company that owns and / or operates the software distribution platform 1505 may be a developer, vendor, and / or licensor of software, such as the machine-readable instructions 1232 of Fig. 12. Third parties may be consumers, users, retailers, OEMs, etc., who acquire and / or license the software for use and / or resale and / or sublicensing. In the illustrated example, the software distribution platform 1505 includes one or more servers and one or more storage devices. The storage devices store the machine-readable instructions 1232, which correspond to the machine-readable instruction examples 800, 900, 1000, and / or 1100 described above. Fig. 8-11. The one or more servers of the example software distribution platform 1505 are in communication with an example network 1510, which may correspond to one or more of the Internet and / or example networks 128, 1226, 1510 described above. In some examples, the one or more servers respond to requests to transfer the software to a requesting party as part of a commercial transaction. Payment for the delivery, sale, and / or licensing of the software may be processed through the one or more servers of the software distribution platform and / or through a third party payment entity. The servers enable purchasers and / or licensors to download the machine-readable instructions 1232 from the software distribution platform 1505. For example, the software corresponding to the example machine-readable instructions 800, 900, 1000, and / or 1100 of Fig. 8-11, may be downloaded to the exemplary processor platform 400 which is to execute the machine-readable instructions 1232 to run the accelerator compiler 200 of Fig. 2. In some examples, one or more servers of the software distribution platform 1505 regularly provide, transmit, and / or enforce updates to the software (e.g., the machine-readable instructions 1232 of Fig. 12) to ensure that improvements, patches, updates, etc. are distributed and applied to the software on end-user devices.

[0148] Certain examples provide an apparatus for performing scale shifting on lower precision values to enable efficient and high-precision performance, including means for identifying a first precision data type and a second precision data type associated with execution of a machine learning model by acceleration hardware, wherein the first precision data type has a first data precision that is greater than a second data precision of the second precision data type. The means for identification may be implemented, for example, by the accelerator compiler 200 of Fig. 2 and / or, more specifically, by the data accuracy selection circuit 220 of Fig. 2. The exemplary apparatus also includes means for determining at least one scaling factor to be applied to first weights of the machine learning model, wherein the first weights are based on the first accuracy data type. The means for determining may be implemented, for example, by the accelerator compiler 200 of Fig. 2 and / or, more specifically, by the scale factor determination circuit 240 of Fig. 2. The exemplary apparatus also includes means for converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, wherein the second weights are based on the second accuracy data type. The means for converting may be implemented, for example, by the accelerator compiler 200 of Fig. 2 and / or, more specifically, by the scale factor determination circuit 240 of Fig. 2. The exemplary apparatus also includes means for generating an output from the execution of the machine learning model based on the second weights. The generating means may be implemented, for example, by the accelerator compiler 200 of Fig. 2 and / or, more specifically, by the model execution circuit 250 and the executable file generation circuit 260 of Fig. 2 should be implemented.

[0149] From the foregoing, it will be appreciated that exemplary systems, methods, devices, and articles of manufacture have been disclosed that perform scale-shifting of lower-fidelity data representation types to enable more accurate computation of machine learning (ML) and / or artificial intelligence (AI) models while increasing computational efficiency and / or reducing resource overhead. The presented systems, methods, devices, and articles of manufacture improve the efficiency of using a computing device by eliminating the problem of loss of fidelity as a barrier to using lower-fidelity data representation formats, such as BF16, thus enabling a massive reduction in computational cost, overhead, and / or resource consumption (e.g., by reducing overall memory requirements).The systems, methods, devices, and articles of manufacture presented accordingly aim at one or more improvements in the operation of a machine, such as a computer or other electronic and / or mechanical device.

[0150] Exemplary methods, devices, systems, and articles of manufacture for improving the accuracy of operations by compensating for lower accuracy with scale shift are disclosed herein. Further examples and combinations thereof include the following:

[0151] Example 1 includes a computer-readable medium containing instructions that, when executed, cause a machine to identify at least a first accuracy data type and a second accuracy data type associated with execution of a machine learning model, the first accuracy data type having a first data accuracy that is greater than a second data accuracy of the second accuracy data type, determine at least one scaling factor to apply to first weights of the machine learning model, the first weights based on the first accuracy data type, convert the first weights to second weights based on a multiplication of the first weights and the at least one scaling factor, the second weights based on the second accuracy data type, and generate an output from execution of the machine learning model based on the second weights.

[0152] Example 2 includes the computer-readable medium of Example 1, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the Bfloat16 floating-point format.

[0153] Example 3 includes the computer-readable medium of Example 1, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the half-precision floating-point format.

[0154] Example 4 includes the computer-readable medium of Example 1, further comprising instructions that, when executed, cause the machine to generate a target weight for a weight of the first weights based on a mask in which one or more least significant bits of the weight are zero, and to determine the at least one scaling factor based on a ratio between the target weight and the weight of the first weights.

[0155] Example 5 includes the computer-readable medium of Example 1, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0156] Example 6 includes the computer-readable medium of Example 1, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first tensor of the machine learning model, the second scaling factor is associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

[0157] Example 7 includes the computer-readable medium of any preceding example, wherein the instructions, when executed, cause the machine to determine the at least one scaling factor during execution of the machine learning model.

[0158] Example 8 includes the computer-readable medium of Example 1, wherein the output generated from executing the machine learning model based on the second weights includes the at least one scaling factor for use in descaling.

[0159] Example 9 includes the computer-readable medium of Example 8, wherein the instructions, when executed, cause the machine to perform descaling by dividing the output from the execution of the machine learning model by the second weights.

[0160] Example 10 includes an apparatus for scaling to lower accuracy values to enable efficient and high-accuracy performance, comprising an interface circuit to obtain a machine learning model, and a processor circuit including one or more of at least one of a central processing unit, a graphics processing unit, or a digital signal processor, wherein the at least one of the central processing unit, the graphics processing unit, or the digital signal processor comprises a control circuit for controlling data movement within the processor circuit, an arithmetic and logic circuit for performing one or more first operations according to instructions, and one or more registers for storing a result of the one or more first operations, the instructions in the apparatus, a field programmable gate array (FPGA), wherein the FPGA comprises a logic gate circuit,a plurality of configurable interconnects and memory circuits, wherein the logic gate circuit and the plurality of configurable interconnects perform one or more second operations, wherein the memory circuit stores a result of the one or more second operations, or application-specific integrated circuits (ASICs) including logic gate circuits to perform one or more third operations, wherein the processor circuit executes at least one of the second operations or the third operations to instantiate a data accuracy selection circuit to identify a first data accuracy type and a second data accuracy type associated with execution of the machine learning model by acceleration hardware, wherein the first data accuracy type has a first data accuracy that is greater than a second data accuracy of the second data accuracy type,a scaling factor determination circuit for determining at least one scaling factor to be applied to first weights of the machine learning model, the first weights being based on the first-precision data type, and converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, the second weights being based on the second-precision data type, and a model execution circuit for generating an output from the execution of the machine learning model based on the second weights.

[0161] Example 11 includes the apparatus of Example 10, wherein the first precision data type is based on the single precision floating point format and the second precision data type is based on the Bfloat16 floating point format.

[0162] Example 12 includes the apparatus of Example 10, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the half-precision floating-point format.

[0163] Example 13 includes the apparatus of Example 10, wherein the scale factor determining circuit further generates a target weight for a weight of the first weights based on masking one or more least significant bits of the weight being zero, and determines the at least one scale factor based on a ratio between the target weight and the weight of the first weights.

[0164] Example 14 includes the apparatus of Example 10, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0165] Example 15 includes the apparatus of any preceding example, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first tensor of the machine learning model, the second scaling factor is associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

[0166] Example 16 includes the apparatus of Example 10, wherein the output generated from the execution of the machine learning model based on the second weights by the model execution circuitry includes the at least one scaling factor for use in descaling.

[0167] Example 17 includes the apparatus of Example 16, wherein the model execution circuitry is to perform descaling by dividing the output from the execution of the machine learning model by the second weights.

[0168] Example 18 includes a method for performing scale shifting on lower precision values to enable efficient and high-precision performance, comprising identifying a first precision data type and a second precision data type associated with execution of a machine learning model by acceleration hardware, wherein the first precision data type has a first data precision greater than a second data precision of the second precision data type; determining, by executing an instruction with at least one processor, at least one scaling factor to be applied to first weights of the machine learning model, the first weights being based on the first precision data type; converting the first weights to second weights based on a multiplication of the first weights and the at least one scaling factor;wherein the second weights are based on the second accuracy data type, and generating an output from the execution of the machine learning model based on the second weights.,

[0169] Example 19 includes the method of Example 18, further comprising generating a target weight for a weight of the first weights based on masking one or more least significant bits of the weight being zero, and determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights.

[0170] Example 20 includes the method of any preceding example, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0171] Example 21 includes the method of Example 18, wherein the output generated from executing the machine learning model based on the second weights includes the at least one scaling factor for use in descaling.

[0172] Example 22 includes the method of Example 21, wherein the scaling is performed by dividing the output from the execution of the machine learning model by the second weights.

[0173] Example 23 includes an apparatus for performing scale shifting on lower-precision values to facilitate efficient and high-accuracy performance, the apparatus comprising means for identifying a first accuracy data type and a second accuracy data type associated with execution of a machine learning model by acceleration hardware, the first accuracy data type having a first data accuracy greater than a second data accuracy of the second accuracy data type, means for determining at least one scaling factor to be applied to first weights of the machine learning model, the first weights based on the first accuracy data type, means for converting the first weights to second weights based on a multiplication of the first weights and the at least one scaling factor, the second weights based on the second accuracy data type,and means for generating an output from the execution of the machine learning model based on the second weights.,

[0174] Example 24 includes the apparatus of Example 23, further comprising means for generating a target weight for a weight of the first weights based on a mask of one or more least significant bits of the weight being zero, and means for determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights.

[0175] Example 25 includes an apparatus having means for performing any method of Examples 18-22.

[0176] The following claims are hereby incorporated by reference into this Detailed Description. Although certain exemplary systems, methods, devices, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not so limited. Rather, this patent covers all systems, methods, devices, and articles of manufacture that fall entirely within the scope of the claims of this patent.

Claims

[1] Computer-readable medium containing instructions which, when executed, cause a machine to do at least the following: Identifying a first accuracy data type and a second accuracy data type associated with the execution of a machine learning model, the first accuracy data type having a first data accuracy that is greater than a second data accuracy of the second accuracy data type; Determining at least one scaling factor to be applied to first weights of the machine learning model, the first weights being based on the first accuracy data type; Converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, wherein the second weights are based on the second accuracy data type; and Generating an output from the execution of the machine learning model based on the second weights. [2] The computer-readable medium of claim 1, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the Bfloat16 floating-point format. [3] The computer-readable medium of claim 1, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the half-precision floating-point format. [4] The computer-readable medium of claim 1, further comprising instructions that, when executed, cause the machine to: Generating a target weight for a weight of the first weights based on a mask in which one or more least significant bits of the weight are zero; and Determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights. [5] The computer-readable medium of claim 1, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor. [6] The computer-readable medium of claim 1, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first tensor of the machine learning model, wherein the second scaling factor is associated with a second tensor of the machine learning model, and wherein the first scaling factor is different from the second scaling factor. [7] The computer-readable medium of any preceding claim, wherein the instructions, when executed, cause the machine to determine the at least one scaling factor during execution of the machine learning model. [8] The computer-readable medium of claim 1, wherein the output generated from executing the machine learning model based on the second weights includes the at least one scaling factor for use in descaling. [9] The computer-readable medium of claim 8, wherein the instructions, when executed, cause the machine to perform descaling by dividing the output from the execution of the machine learning model by the second weights. [10] A device for performing a scale shift on lower accuracy values to enable efficient and high-accuracy performance, comprising: Interface circuits to obtain a machine learning model; and Processor circuits comprising one or more of the following elements: at least one of a central processing unit, a graphics processing unit, or a digital signal processor, wherein the at least one of the central processing unit, the graphics processing unit, or the digital signal processor comprises a control circuit for controlling data movement within the processor circuit, an arithmetic and logic circuit for performing one or more first operations corresponding to instructions, and one or more registers for storing a result of the one or more first operations of the instructions in the device; a field-programmable gate array (FPGA), the FPGA including a logic gate circuit, a plurality of configurable connections, and a memory circuit, the logic gate circuit and the plurality of configurable connections performing one or more second operations, and the memory circuit storing a result of the one or more second operations; or Application-specific integrated circuits (ASICs) with logic gate circuits for performing one or more third operations; the processor circuit to perform at least one of the first operations, the second operations or the third operations for instantiating: Data accuracy selection circuitry to identify a first accuracy data type and a second accuracy data type associated with execution of the machine learning model by the acceleration hardware, the first accuracy data type having a first data accuracy greater than a second data accuracy of the second accuracy data type; Scale factor determination circuit for performing the following: Determining at least one scaling factor to be applied to first weights of the machine learning model, wherein the first weights are based on the first accuracy data type; and Converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, wherein the second weights are based on the second accuracy data type; and Model execution circuitry to generate an output from the execution of the machine learning model based on the second weights. [11] The apparatus of claim 10, wherein the first precision data type is based on the single precision floating point format and the second precision data type is based on the Bfloat16 floating point format. [12] The apparatus of claim 10, wherein the first precision data type is based on the single-precision floating-point format and the second precision data type is based on the half-precision floating-point format. [13] The apparatus of claim 10, wherein the scale factor determining circuit is further configured to: Generating a target weight for a weight of the first weights based on a masking of one or more least significant bits of the weight being zero; and Determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights. [14] The apparatus of claim 10, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor. [15] Apparatus according to any one of the preceding claims, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first tensor of the machine learning model, wherein the second scaling factor is associated with a second tensor of the machine learning model, and wherein the first scaling factor is different from the second scaling factor. [16] The apparatus of claim 10, wherein the output generated from the execution of the machine learning model based on the second weights by the model execution circuit includes the at least one scaling factor for use in descaling. [17] The apparatus of claim 16, wherein the model execution circuit is to perform descaling by dividing the output from the execution of the machine learning model by the second weights. [18] A method for performing a scale shift on lower accuracy values to enable efficient and high accuracy performance, comprising: Identifying a first accuracy data type and a second accuracy data type associated with execution of a machine learning model by acceleration hardware, wherein the first accuracy data type has a first data accuracy that is greater than a second data accuracy of the second accuracy data type; Determining, by executing an instruction with at least one processor, at least one scaling factor to be applied to first weights of the machine learning model, the first weights being based on the first accuracy data type; Converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, wherein the second weights are based on the second accuracy data type; and Generating an output from the execution of the machine learning model based on the second weights. [19] The method of claim 18, further comprising: Generating a target weight for a weight of the first weights based on a masking of one or more least significant bits of the weight being zero; and Determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights. [20] The method of any preceding claim, wherein the at least one scaling factor comprises a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor. [21] The method of claim 18, wherein the output generated from executing the machine learning model based on the second weights includes the at least one scaling factor for use in descaling. [22] The method of claim 21, wherein the scaling is performed by dividing the output from the execution of the machine learning model by the second weights. [23] Apparatus for performing a scale shift on lower accuracy values to enable efficient and high accuracy performance, the apparatus comprising: Means for identifying a first accuracy data type and a second accuracy data type associated with execution of a machine learning model by acceleration hardware, the first accuracy data type having a first data accuracy greater than a second data accuracy of the second accuracy data type; means for determining at least one scaling factor to be applied to first weights of the machine learning model, the first weights being based on the first accuracy data type; means for converting the first weights into second weights based on a multiplication of the first weights and the at least one scaling factor, wherein the second weights are based on the second accuracy data type; and Means for generating an output from the execution of the machine learning model based on the second weights. [24] The apparatus of claim 23, further comprising: means for generating a target weight for a weight of the first weights based on a mask in which one or more least significant bits of the weight are zero; and Means for determining the at least one scaling factor based on a ratio between the target weight and the weight of the first weights. [25] Apparatus comprising means for carrying out one of the methods according to claims 18-22.

Images