Improving accuracy of machine learning operations by compensating for lower precision with scaled transforms

Abstract

Disclosed is a technical solution for improving the accuracy of the operation of machine learning by compensating for lower precision with a scaling transform. An example non-transitory computer readable medium includes instructions that, when executed, cause a machine to identify at least a first precision data type and a second precision data type associated with execution of a machine learning model, the first precision data type has a first data precision greater than a second data precision of the second precision data type, determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type, and converting the first weight to a second weight based on a multiplication of the first weight with the at least one scaling factor, the second weight being based on the second precision data type.

Description

Technical Field

[0001] The present disclosure relates generally to operations in machine learning (ML), such as dot product operations, and more particularly to improving the accuracy of ML operations by compensating for lower precision using scale shifting. Background Art

[0002] Operations that are performed at large scale (e.g., dot product operations) are an element of artificial intelligence (AI), machine learning (ML), and / or general scientific computing. Hardware vendors facilitate the acceleration of these operations by providing new instructions (e.g., Advanced Vector Extensions (AVX), Advanced Matrix Extensions (AMX)) and / or dedicated functional units, while software vendors implement new instructions and / or dedicated units to further make algorithmic changes, data layout changes, fuse operations to improve cache efficiency and other performance related to computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is an illustration of an example electronic system that includes an example accelerator compiler to configure an example acceleration circuit based on an acceleration operation to be performed by the accelerator circuit.

[0004] Figure 2 yes Figure 1 A block diagram of an example implementation of an accelerator compiler.

[0005] Figure 3 yes Figure 1 and / or Figure 2 Illustration of an example conventional convolution operation that may be performed by an example acceleration circuit.

[0006] Figure 4 Illustrated is a bit-by-bit binary representation of an FP32 value.

[0007] Figure 5 Pictured Figure 2 An example enumeration process for selecting a scaling factor performed by an example scaling factor determination circuit.

[0008] Figure 6 Example comparative accuracy graphs are shown for example operations using FP32 values, BF16 values, and scaled BF16 values.

[0009] Figure 7An example accuracy percentage table is illustrated for an example operation using FP32 values, BF16 values, scaled BF16 values ​​of a first technique, and scaled BF16 values ​​of a second technique.

[0010] Figure 8-Figure 11 The flowchart represents an example of a processor circuit that can be executed to implement Figure 1 and / or Figure 2 Example machine readable instructions and / or example operations for an acceleration circuit.

[0011] Fig.12 is a block diagram of an example processing platform including a processor circuit configured to perform Figure 8-Figure 11 Example machine readable instructions and / or example operations to implement Figure 1 and / or Figure 2 The acceleration circuit.

[0012] Fig.13 yes Fig.12 A block diagram of an example implementation of a processor circuit.

[0013] Fig.14 yes Fig.12 A block diagram of another example implementation of a processor circuit.

[0014] Fig.15 is a block diagram of an example software distribution platform (e.g., one or more servers) for distributing software (e.g., with Figure 8-Figure 11 The software corresponding to example machine-readable instructions of the present invention is distributed to client devices associated with end-users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or other end-users such as direct purchasing customers).

[0015] In general, the same reference numerals will be used throughout the various drawings and the accompanying written description to refer to the same or similar components. The drawings are not to scale. Instead, the thickness of a layer or region may be exaggerated in the drawings. Although the drawings show layers and regions with clear lines and boundaries, some or all of these lines and / or boundaries may be idealized. In reality, boundaries and / or lines may be imperceptible, blended, and / or irregular.

[0016] Notwithstanding the above, in the context of semiconductor devices, "above" is not referenced to the Earth, but rather to the bulk region of a base semiconductor substrate (e.g., a semiconductor wafer) on which components of an integrated circuit are formed. Specifically, as used herein, a first component of an integrated circuit is "above" a second component when the first component is farther from the bulk region of the semiconductor substrate than the second component.

[0017] As used herein, unless otherwise indicated, the term "above" describes the relationship of two parts relative to the earth. A first part is above a second part if at least a portion of the second part is between the earth and the first part. Likewise, as used herein, a first part is "below" a second part when the first part is closer to the earth than the second part. As described above, a first part can be above or below a second part in one or more of the following situations: there are other parts in between, there are no other parts in between, the first part and the second part are in contact, or the first part and the second part are not in direct contact with each other.

[0018] Unless otherwise specifically stated, descriptors such as "first," "second," "third," and the like are used herein without importing or otherwise indicating any priority, physical order, arrangement in a list, and / or meaning of being ordered in any manner, but are merely used as labels and / or arbitrary names to distinguish elements so that the disclosed examples are easily understood. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while the same element may be referred to in the claims with a different descriptor, such as "second" or "third." In this case, it should be understood that such descriptors are only used to explicitly identify those elements that, for example, might otherwise share the same name.

[0019] As used herein, "approximately" and "about" modify their subject matter / values ​​to recognize that there may be variations in real-world applications. For example, "approximately" and "approximately" may modify a dimension that may not be exact due to manufacturing tolerances and / or other real-world imperfections that would be understood by one of ordinary skill in the art. For example, "approximately" and "approximately" may indicate that such a dimension may be within a tolerance range of + / - 10%, unless otherwise indicated in the description below. As used herein, "substantially real time" means occurring in a nearly instantaneous manner, recognizing that there may be delays in computing time, transmission, etc. in the real world. Thus, unless otherwise indicated, "substantially real time" means real time + / - 1 second.

[0020] As used herein, "processor circuitry" is defined as including (i) one or more special-purpose electrical circuits that are constructed to perform (one or more) specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electrical circuits that can be programmed with instructions to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuits include programmable microprocessors, field programmable gate arrays (FPGAs) that can instantiate instructions, central processor units (CPUs), graphics processor units (GPUs), digital signal processors (DSPs), XPUs, or microcontrollers and integrated circuits, such as application-specific integrated circuits (ASICs). For example, the XPU may be implemented by a heterogeneous computing system that includes multiple types of processor circuits (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and / or combinations thereof) and (one or more) application programming interfaces (APIs) that may assign (one or more) computing tasks to any one (or more) of the multiple types of processor circuits that is most suitable for performing the (one or more) computing tasks. DETAILED DESCRIPTION

[0021] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables a machine (e.g., a computer, logic circuit, etc.) to process input data using a model to generate outputs based on patterns and / or associations that the model has previously learned through a training process. For example, the model may be trained with data to recognize patterns and / or associations, and follow such patterns and / or associations when processing input data, so that (one or more) other inputs result in (one or more) outputs consistent with the recognized patterns and / or associations.

[0022] There are many different types of machine learning models and / or machine learning architectures. In some examples disclosed herein, a convolutional neural network (CNN) model is used. The use of the CNN model enables weight sharing (e.g., reducing the number of weights that the model must learn), thereby reducing model training time and computational cost. In general, the machine learning model / architecture suitable for use in the example scheme disclosed herein will be a neural network (NN), a deep neural network (DNN) and / or a recurrent neural network (RNN). However, other types of machine learning models, such as support vector machines (SVM), long short-term memory (LSTM), gated recurrent units (GRU), etc., may also be used additionally or alternatively.

[0023] In general, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a model is trained using a training algorithm to operate according to patterns and / or associations based on, for example, training data. In general, a model includes internal parameters that guide how input data is transformed into output data, such as by a series of nodes and connections within the model. In addition, hyperparameters are used as part of the training process to control how learning is performed (e.g., learning rate, the number of layers to be used in a machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0024] Different types of training can be performed based on the type and / or expected output of the ML / AI model. For example, supervised training uses inputs and corresponding expected (e.g., labeled) outputs to select parameters for the ML / AI model (e.g., by iterating over combinations of selected parameters) that reduce model error. As used herein, labeling refers to the expected output of a machine learning model (e.g., a classification, an expected output value, etc.). Alternatively, unsupervised training (e.g., used in deep learning, subsets of machine learning, etc.) involves inferring patterns from inputs to select parameters for an ML / AI model (e.g., without the benefit of expected (e.g., labeled) outputs).

[0025] In the examples disclosed herein, stochastic gradient descent is used to train the ML / AI model. However, any other training algorithm may be used in addition or alternatively. In the examples disclosed herein, training is performed until an acceptable amount of error is reached. In the examples disclosed herein, training may be performed at the electronic system 102 (e.g., on (one or more) ML models 124). Training is performed using hyperparameters that control how learning is performed (e.g., learning rate, the number of layers to be used in the machine learning model, etc.). In the examples disclosed herein, hyperparameters that control the accuracy of the values ​​used as the operating object are used. For example, such hyperparameters are selected manually and / or using statistical (random) sampling. In some examples, retraining may be performed. Such retraining may be performed in response to an accuracy metric not meeting a threshold.

[0026] Training is performed using training data. In the examples disclosed herein, the training data may be derived from a data storage device (e.g., in combination with Figure 2 270 for further explanation). Because supervised training is used, the training data is labeled. The labels are applied to the training data by the accelerator compiler (e.g., in conjunction with Figure 1 In some examples, an interface (e.g., in conjunction with Figure 1 Further described example interface circuit 114) pre-processes the training data. In some examples, Figure 1 The accelerator compiler 104A-C subdivides the training data into: a first portion of data for training (one or more) machine learning models 124, and a second portion of data for validating Figure 1 (One or more) example machine learning (ML) models 124.

[0027] Once trained, the model is deployed to serve as an executable construct that processes inputs and provides outputs based on the network of nodes and connections defined in the model. The model is stored in a data storage device (e.g., Figure 2 The model can then be executed by the example model execution circuit 250 (in conjunction with the example data storage device 270). Figure 2 In some examples, the platform on which the model is executed may have specific constraints on the precision and / or accuracy of the operands.

[0028] Once trained, the deployed model can be operated to process data in the reasoning phase. In the reasoning phase, the data to be analyzed (e.g., live data) is input to the model, and the model executes to create output. This reasoning phase can be thought of as the AI ​​"thinking" to generate output based on what it has learned from training (e.g., by executing the model to apply the learned patterns and / or associations to live data). In some examples, the input data undergoes preprocessing before being used as input to the machine learning model. In addition, in some examples, the output data may undergo post-processing after it is generated by the AI ​​model to transform the output into a useful result (e.g., a display of data, instructions to be executed by the machine, etc.).

[0029] In some examples, the output of the deployed model can be captured and provided as feedback. By analyzing the feedback, the accuracy of the deployed model can be determined. If the feedback indicates that the accuracy of the deployed model is below a threshold or other criteria, the feedback and updated training data sets, hyperparameters, etc. can be used to trigger training of the updated model to generate an updated deployed model.

[0030] There may be many different ways to represent operands and / or values ​​(e.g., integers, scalars, decimals, etc.) on a particular computing device, each representation being characterized by a variable size memory footprint and / or a different quality of precision. For example, the single-precision floating point format (referred to herein as the "FP32" format) is most commonly used in machine learning (ML) and artificial intelligence (AI) applications, particularly deep neural networks (DNNs). The FP32 format is characterized by 1 sign bit, 8 exponent bits, and 23 mantissa bits, which provide high-precision capabilities due to the higher-than-average amount of available bit storage. However, another feature of the FP32 format is a correspondingly larger memory footprint, with each value represented using the format occupying a total of 32 bits. Therefore, in situations where large volumes of values ​​and / or data must be operated on and / or stored in memory on a particular computing device, remote server, etc., a decrease in machine performance may be observed.

[0031] The brain floating point format (referred to herein as the "BF16" format) may also be used in applications involving computations for ML and AI, and is characterized by 1 sign bit, 8 exponent bits, and 7 mantissa bits, which indicates lower precision capabilities due to lower bit storage availability compared to the FP32 format. However, the BF16 format uses 16 fewer mantissa bits for storage in memory, and thus is also characterized by a smaller memory footprint, which makes it advantageous in enabling better performance (e.g., more efficient) of a computing device and / or less resource-intensive computations.

[0032] For many types of computation (e.g., scientific computing), it is perfectly acceptable to work with lower precision values ​​(e.g., BF16 values) to achieve better (e.g., more efficient and / or less resource intensive) computations. For example, an advertising engine may sometimes experience a lower click-through rate (CTR) due to inaccurate classification (e.g., indicating a lower number of users clicking on an ad rather than scrolling past it). In some cases, the inaccurate classification may be related to a reduction in the number of floating point bits (e.g., mantissa bits) used to describe a value, where fewer bits indicate a lower level of precision (e.g., similar to the number of digits listed after the decimal point for a decimal value), and the frequency of classification errors resulting from the reduced level of precision is considered acceptable within the advertising industry.

[0033] However, for many other types of calculations (e.g., cumulative dot product operations), accuracy cannot be sacrificed in exchange for faster and / or better performance. In particular, when artificial intelligence (AI) and / or high performance computing (HPC) techniques are used in fields that require high-accuracy operations, such as finance, robotics, pharmaceuticals, radiology, etc., low-precision operation objects and / or value representation formats (e.g., BF16) are not preferred over high-precision but more memory-intensive value representation formats (e.g., FP32). In these examples, higher-precision operations take precedence over computational efficiency.

[0034] Therefore, increasing the precision provided by a typically lower-precision value representation format (e.g., BF16) while retaining the less resource-intensive nature of such a format is an important goal for improving the performance of electronic systems (e.g., in conjunction with Figure 1 The example electronic system 102 and the example external electronic system 130 may further illustrate a desired approach to improve efficiency of computation and / or operation.

[0035] Current solutions that use lower precision data representation formats focus on frequently retraining machine learning (ML) and / or artificial intelligence (AI) models that utilize these formats, and perform extensive software testing to determine that the accuracy loss caused by the conversion to a lower precision value representation (e.g., BF16) falls within an acceptable range. Additional solutions use higher precision value representations (e.g., FP32) during the training phase of the ML / AI model, and use lower precision value representations (e.g., BF16) during the inference phase of the ML / AI model. These solutions similarly utilize extensive software testing to ensure that the associated accuracy loss generally falls within an acceptable range.

[0036] The extensive software retraining and / or testing adopted and / or required by these schemes often produces unsatisfactory, delayed and / or unclear suggestions (e.g., particularly during the reasoning phase of the ML / AI model). In addition, the frequent repetition of such precision and accuracy testing required to ensure optimal model results is resource-intensive, computationally expensive and / or challenging, especially when the test data set and / or training data set are large and / or frequently evolving. In other words, the software testing required to ensure that the model results fall within an acceptable accuracy range may result in an extended verification cycle. In addition, if new training and / or retraining of the model improves accuracy, but using a lower precision data representation format reduces accuracy, then the decision between using a low precision or high precision data representation format becomes increasingly complex. In addition, due to the foreseeable and / or observed risk of making an incorrect truncation decision, the current scheme described herein may only be applicable to a limited number of situations, especially when the classification (triage) performed by the model using a lower precision data representation format (e.g., BF16) produces an incorrect confidence score. In short, current approaches often overcomplicate the process of deploying ML / AI models by increasing the complexity of the training phase and / or the post-training testing / inference phase.

[0037] An example approach to utilizing lower precision data representation formats while maintaining high accuracy of ML / AI model results focuses on scaling values ​​and / or operations from a high precision data representation format (e.g., FP32) to a lower precision data representation format (e.g., BF16) using a weighting factor, and then inversely scaling its resulting value (e.g., post-operation and / or post-computation) by the same order of magnitude (e.g., the same weighting factor) (e.g., from BF16 to FP32 format). This approach reduces the amount of information stored and / or contained in the low-order bits of the mantissa of the value representation format (e.g., the BF16 representation format is characterized by 16 fewer mantissa bits), thereby effectively reducing the memory footprint and / or computation requirements associated with each value. In operations performed by ML / AI models that perform frequent accumulation of values ​​in large quantities (e.g., dot product operations), the accumulated information stored in the low-order bits of the mantissa of the FP32 representation is effectively captured via scaling and / or weighting factors applied to the values ​​when converted to their BF16 representations, thereby reducing the frequency of truncation errors and / or information / accuracy losses associated with conversions of these value formats. That is, for example, rescaling (e.g., de-scaling) the values ​​represented by the BF16 representation effectively corrects any potential truncation errors observed due to the elimination of the 16 lower-order mantissa bits, which is achieved by uniformly updating those same mantissa bits back to their original FP32 format after completing a given calculation and / or operation (e.g., a dot product operation), thereby effectively eliminating any bias, error, and / or the need for extensive software testing to ensure that the model meets an acceptable range of result accuracy.

[0038] Furthermore, removing the concerns about accuracy loss as a barrier to using lower precision data representation formats (e.g., BF16) enables a massive reduction in computational cost, effort, and / or resource consumption (e.g., by reducing overall memory footprint). In the examples disclosed herein, while FP32 and BF16 data representation formats are used to describe scaling and descaling between higher precision and lower precision data representation formats, any other type of data representation format may be used to perform bit width reduction.

[0039] Figure 1 is an illustration of an example computing environment 100 that includes an example electronic system 102 that includes example accelerator compilers 104A-C for configuring an ML / AI accelerator to perform scaling operations, such as convolution operations, matrix multiplication operations (e.g., MatMul), etc., to achieve improved accelerator efficiency and performance. In some examples, the accelerator compilers 104A-C obtain output from a machine learning framework (e.g., a NN framework) and compile the output based on the scaling operations to be performed by the accelerator and / or otherwise performed for implementation on the accelerator.

[0040] Figure 1 The illustrated example electronic system 102 includes an example central processing unit (CPU) 106, a first example acceleration circuit (ACCELERATION CIRCUITRY A) 108, a second example acceleration circuit (ACCELERATION CIRCUITRY B) 110, an example general processing circuit 112, an example interface circuit 114, an example bus 116, an example power supply 118, and an example data storage device 120. In this example, the data storage device 120 includes example configuration data (CONFIG DATA) 122 and (one or more) example machine learning models (ML MODEL) 124. Figure 1 Further depicted in the illustrated example of FIG. 1 are an example user interface 126 , an example network 128 , and an example external electronic system 130 .

[0041] In some examples, electronic system 102 is a system on a chip (SoC) representing one or more integrated circuits (ICs) (e.g., compact ICs) that incorporate components of a computer or other electronic system in a compact format. For example, electronic system 102 can be implemented with a combination of one or more programmable processors, hardware logic, and / or hardware peripherals and / or interfaces. Additionally or alternatively, Figure 1 An example electronic system 102 may include memory, (one or more) input / output (I / O) ports, and / or secondary storage. For example, the electronic system 102 includes an acceleration compiler 104A-C, a CPU 106, a first acceleration circuit 108, a second acceleration circuit 110, a general processing circuit 112, an interface circuit 114, a bus 116, a power supply 118, a data storage device 120, memory, (one or more) I / O ports, and / or secondary storage, all of which are on the same substrate (e.g., a silicon substrate, a semiconductor-based substrate, etc.). In some examples, the electronic system 102 includes digital, analog, mixed signal, radio frequency (RF), or other signal processing functions.

[0042] exist Figure 1In the illustrated example of , the first acceleration circuit 108 is an artificial intelligence (AI) accelerator. For example, the first acceleration circuit 108 can be implemented by a hardware accelerator configured to accelerate AI tasks or workloads, such as NN (e.g., artificial neural network (ANN)), machine vision, machine learning, and the like. In some examples, the first acceleration circuit 108 can implement an ML / AI accelerator (e.g., a sparse hardware accelerator). In some examples, the first acceleration circuit 108 can implement a vision processing unit (VPU) to implement machine or computer vision computing tasks, train and / or execute physical neural networks, and / or train and / or execute neural networks. In some examples, the first acceleration circuit 108 can train and / or execute convolutional neural networks (CNNs), deep neural networks (DNNs), ANNs, recursive neural networks (RNNs), and the like, and / or combinations of these.

[0043] exist Figure 1 In the illustrated example of FIG. 1 , the second acceleration circuit 110 is a graphics processing unit (GPU). For example, the second acceleration circuit 110 can be a GPU that generates computer graphics, performs general-purpose computations, and the like. In some examples, the second acceleration circuit 110 is another instance of the first acceleration circuit 108. In some such examples, the electronic system 102 can provide (one or more) portions of the AI / ML workload to be executed in parallel by the first acceleration circuit 108 and the second acceleration circuit 110.

[0044] Figure 1 The general processing circuit 112 of the example is a programmable processor, such as a CPU or a GPU. Alternatively, one or more of the first acceleration circuit 108, the second acceleration circuit 110 and / or the general processing circuit 112 may be different types of hardware, such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) and / or a field programmable logic device (FPLD) (e.g., a field-programmable gate array (FPGA)).

[0045] exist Figure 1In the illustrated example of , the interface circuit 114 is hardware that can implement one or more interfaces (e.g., computing interfaces, network interfaces, etc.). For example, the interface circuit 114 can be hardware, software, and / or firmware that implements a communication device (e.g., a network interface card (NIC), a smart NIC, a gateway, a switch, etc.), such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface to facilitate exchanging data with an external machine (e.g., any kind of computing device) via the network 128. In some examples, communication is via connection, Ethernet connection, digital subscriber line (DSL) connection, wireless fidelity (Wi-Fi) connection, telephone line connection, coaxial cable system, satellite system, direct-line wireless system, cellular telephone system, optical connection (e.g., fiber optic connection), etc. For example, the interface circuit 114 can be implemented by any type of interface standard, such as interface, Ethernet interface, Wi-Fi interface, universal serial bus (USB), near field communication (NFC) interface and / or peripheral component interconnect express (PCIe) interface.

[0046] The electronic system 102 includes a power supply 118 to deliver power to the hardware of the electronic system 102. In some examples, the power supply 118 can implement a power delivery network. For example, the power supply 118 can implement an AC to DC (AC / DC) power supply source. In some examples, the power supply 118 can be coupled to a power grid infrastructure, such as an AC main power supply (e.g., a 110 volt (V) AC grid main power supply, a 220V AC grid main power supply, etc.). Additionally, or alternatively, the power supply 118 can be implemented by a battery. For example, the power supply 118 can be a device with limited energy, such as a lithium-ion battery or any other rechargeable battery or power supply. In some such examples, the power supply 118 can be charged using a power adapter or converter (e.g., an AC / DC power converter), a wall socket (e.g., a 110V AC wall socket, a 220V AC wall socket, etc.), a portable energy storage device (e.g., a portable mobile power supply, a portable power battery, etc.), etc.

[0047] Figure 1The electronic system 102 of the illustrated example includes a data storage device 120 to record data (e.g., configuration data 122, (one or more) ML models 124, etc.). The data storage device 120 of this example may be implemented by a volatile memory (e.g., Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM), etc.) and / or a non-volatile memory (e.g., flash memory). The data storage device 120 may additionally or alternatively be implemented by one or more double data rate (DDR) memories, such as DDR, DDR2, DDR3, DDR4, mobile DDR (mDDR), etc. The data storage device 120 may additionally or alternatively be implemented by one or more mass storage devices, such as (one or more) hard disk drives (HDD), (one or more) compact disk (CD) drives, (one or more) digital versatile disk (DVD) drives, (one or more) solid-state disk (SSD) drives, etc. Although in the illustrated example, the data storage device 120 is illustrated as a single data storage device, the data storage device 120 may be implemented by any number and / or (one or more) types of data storage devices. In addition, the data stored in the data storage device 120 may take any data format, such as binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, executable files, etc.

[0048] exist Figure 1 In the illustrated example of , the electronic system 102 communicates with the user interface 126. For example, the user interface 126 may be implemented by a graphical user interface (GUI), an application user interface, and the like that may be presented to a user on a display device connected to and / or otherwise communicating with the electronic system 102. In some such examples, a user (e.g., a developer, an IT administrator, a customer, and the like) controls the electronic system 102, configures, trains, and / or executes (one or more) ML models 124, and the like via the user interface 126. Alternatively, the electronic system 102 may include and / or otherwise implement the user interface 126.

[0049] exist Figure 1 In the illustrated example of , the accelerator compiler 104A-C, the CPU 106, the first acceleration circuit 108, the second acceleration circuit 110, the general processing circuit 112, the interface circuit 114, the power supply 118, and the data storage device 120 communicate with each other via a bus 116. For example, the bus 116 can be implemented by at least one of an Inter-Integrated Circuit (IC) bus, a Serial Peripheral Interface (SPI) bus, a Peripheral Component Interconnect (PCI) bus, or a PCIe bus. Additionally or alternatively, the bus 116 can be implemented by any other type of computing or electrical bus.

[0050] exist Figure 1 In the illustrated example of , the network 128 is the Internet. However, the network 128 of this example can be implemented using any (one or more) suitable wired and / or wireless networks, including, for example, one or more data buses, one or more local area networks (LANs), one or more wireless LANs, one or more cellular networks, one or more private networks, one or more public networks, etc. In some examples, the network 128 enables the electronic system 102 to communicate with one (or more) of the external electronic systems 130.

[0051] exist Figure 1 In the illustrated example of , the external electronic system 130 includes and / or otherwise implements one or more electronic (e.g., computing) devices on which the (one or more) ML models 124 are to be executed. In this example, the external electronic system 130 includes an example desktop computer 132, an example mobile device (e.g., a smart phone, an Internet-enabled smart phone, etc.) 134, an example laptop computer 136, an example tablet device (e.g., a tablet computer, an Internet-enabled tablet computer, etc.) 138, and an example server 140. In some examples, the example server 140 may be used. Figure 1 In addition, or in lieu thereof, the external electronic systems 130 may include, correspond to, and / or otherwise represent any other type and / or number of computing devices.

[0052] In some examples, one or more of the external electronic systems 130 execute one (or more) of the ML models 124 to process a computational workload (e.g., an AI / ML workload). For example, the mobile device 134 may be implemented as a cellular or mobile phone having one or more processors (e.g., CPU, GPU, VPU, AI or NN specific processor, etc.) on a single SoC to utilize one (or more) of the ML models 124 to process the AI / ML workload. In some examples, the desktop computer 132, laptop computer 136, tablet computer, and / or server 140 may be implemented as one (or more) electronic (e.g., computing) devices having one or more processors (e.g., CPU, GPU, VPU, AI / NN specific processor, etc.) on one or more SoCs to utilize one (or more) of the ML models 124 to process the AI / ML workload. In some examples, server 140 may implement one or more servers (e.g., physical servers, virtualized servers, etc., and / or combinations thereof) that may implement data facilities, cloud services (e.g., public or private cloud providers, cloud-based repositories, etc.), etc. to process (one or more) AI / ML workloads using one (or more) of (one or more) ML models 124.

[0053] exist Figure 1 In the illustrated example of , the electronic system 102 includes a first accelerator compiler 104A (e.g., a first instance of the accelerator compiler 104A-C), a second accelerator compiler 104B (e.g., a second instance of the accelerator compiler 104A-C), and a third accelerator compiler 104C (e.g., a third instance of the accelerator compiler 104A-C) (collectively referred to herein as the accelerator compilers 104A-C unless otherwise specified). In this example, the first accelerator compiler 104A is implemented by the CPU 106 (e.g., implemented by hardware, software, and / or firmware of the CPU 106).

[0054] exist Figure 1In the illustrated example of , the second accelerator compiler 104B is implemented by the general processing circuit 112 (e.g., implemented by hardware, software, and / or firmware of the general processing circuit 112). In this example, the third accelerator compiler 104C is located outside the CPU 106. For example, the third accelerator compiler 104C can be implemented by hardware, software, and / or firmware of the electronic system 102. In some such examples, the third accelerator compiler 104C can be implemented by one or more analog or digital circuits, logic circuits, (one or more) programmable processors, (one or more) programmable controllers, (one or more) GPUs, (one or more) DSPs, (one or more) ASICs, (one or more) PLDs, and / or (one or more) FPLDs.

[0055] In some examples, one or more of the first accelerator compiler 104A, the second accelerator compiler 104B, the third accelerator compiler 104C, and / or portions thereof may be virtualized, such as by using one or more containers, one or more virtual resources (e.g., virtualization of computing, memory, networking, storage, etc., physical hardware resources), one or more virtual machines, etc. In some examples, one or more of the first accelerator compiler 104A, the second accelerator compiler 104B, the third accelerator compiler 104C, and / or portions thereof may be implemented by different resources (one or more) of the electronic system 102. Alternatively, the electronic system 102 may not include one or more of the first accelerator compiler 104A, the second accelerator compiler 104B, and / or the third accelerator compiler 104C.

[0056] exist Figure 1 In the illustrated example of , the accelerator compiler 104A-C can compile the AI / ML framework based on the configuration data 122 for implementation on one (or more) of the acceleration circuits 108, 110. In some examples, the configuration data 122 may include AI / ML configuration data (e.g., register configuration, activation data, activation sparse data, weight data, weight sparse data, hyperparameters, etc.), convolution operations to be performed (e.g., 2D convolution, depthwise convolution, grouped convolution, dilated convolution, etc.), non-convolution operations (e.g., element-wise addition operations), etc., and / or combinations thereof. In some examples, the accelerator compiler 104A-C can compile the AI / ML framework to generate an executable construct that can be executed by one (or more) of the acceleration circuits 108, 110.

[0057] exist Figure 1In the illustrated example of , the accelerator compiler 104A-C may instruct, direct, and / or otherwise invoke one (or more) of the acceleration circuits 108, 110 to execute one (or more) of the (one or more) machine learning models 124. For example, the (one or more) ML models 124 may implement an AI / ML model. AI, including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables a machine (e.g., a computer, logic circuit, etc.) to process input data using a model to generate outputs based on patterns and / or associations that the model previously learned via a training process. For example, the (one or more) machine learning models 124 may be trained with data to recognize patterns and / or associations, and follow such patterns and / or associations when processing input data, so that (one or more) other inputs result in (one or more) outputs consistent with the recognized patterns and / or associations.

[0058] There are many different types of machine learning models and / or machine learning architectures. In some examples, the accelerator compiler 104A-C generates (one or more) machine learning models 124 as (one or more) neural network models. The accelerator compiler 104A-C can call the interface circuit 114 to transmit (one or more) machine learning models 124 to one (or more) of the external electronic systems 130. The use of neural network models enables the acceleration circuits 108, 110 to perform AI / ML workloads. In general, machine learning models / architectures suitable for use in the example schemes disclosed herein include recursive neural networks. However, other types of machine learning models, such as supervised learning ANN models, clustering models, classification models, etc., and / or combinations thereof, may be used in addition or in lieu. Example supervised learning ANN models may include two-layer (2-layer) radial basis neural networks (RBN), learning vector quantization (LVQ) classification neural networks, and the like. Example clustering models may include k-means clustering, hierarchical clustering, mean shift clustering, density-based clustering, and the like. Example classification models may include logistic regression, support vector machine (SVM) or network, naive Bayes, etc. In some examples, accelerator compilers 104A-C may compile and / or otherwise generate one (or more) of the machine learning model(s) 124 in the form of a lightweight machine learning model.

[0059] In general, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train (one or more) machine learning models 124 to operate according to patterns and / or associations based on, for example, training data. In general, (one or more) machine learning models 124 include internal parameters (e.g., configuration data 122) that guide how input data is transformed into output data, such as by a series of nodes and connections within (one or more) machine learning models 124. In addition, hyperparameters (e.g., configuration data 122) are used as part of the training process to control how learning is performed (e.g., learning rate, number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters determined before initiating the training process.

[0060] Based on the type and / or expected output of the ML / AI model, different types of training can be performed. For example, the accelerator compiler 104A-C can invoke supervised training to use the input and corresponding expected (e.g., labeled) output to select parameters for (one or more) machine learning models (e.g., by iterating over combinations of selected parameters) that reduce model error. As used herein, "label" refers to the expected output of the machine learning model (e.g., classification, expected output value, etc.). Alternatively, the accelerator compiler 104A-C can invoke unsupervised training (e.g., for use in deep learning, subsets of machine learning, etc.), which involves selecting parameters for (one or more) machine learning models 124 from input reasoning patterns (e.g., without the benefit of expected (e.g., labeled) outputs).

[0061] In some examples, the accelerator compilers 104A-C use unsupervised clustering of operational observables to train the machine learning model(s) 124. However, the accelerator compilers 104A-C may additionally or alternatively use any other training algorithm, such as stochastic gradient descent, simulated annealing, particle swarm optimization, evolutionary algorithms, genetic algorithms, nonlinear conjugate gradient, and the like.

[0062] In some examples, the accelerator compiler 104A-C may train the (one or more) machine learning model 124 until the error level no longer decreases. In some examples, the accelerator compiler 104A-C may train the (one or more) machine learning model 124 locally on the electronic system 102 and / or remotely on an external electronic system (e.g., one (or more) of the external electronic systems 130) that is communicatively coupled to the electronic system 102. In some examples, the accelerator compiler 104A-C trains the (one or more) machine learning model 124 using hyperparameters that control how learning is performed (e.g., learning rate, number of layers to be used in the machine learning model, etc.). In some examples, the accelerator compiler 104A-C may use hyperparameters that control model performance and training speed, such as learning rate and (one or more) regularization parameters. The accelerator compiler 104A-C may select such hyperparameters, for example, by trial and error, to achieve optimal model performance. In some examples, the accelerator compilers 104A-C utilize Bayesian hyperparameter optimization to determine an optimal and / or otherwise improved or more efficient network architecture to avoid model overfitting and improve the overall applicability of the (one or more) machine learning models 124. Alternatively, the accelerator compilers 104A-C may use any other type of optimization. In some examples, the accelerator compilers 104A-C may perform retraining. The accelerator compilers 104A-C may perform such retraining in response to an override (one or more) by a user of the electronic system 102, receiving new training data, and the like.

[0063] In some examples, the accelerator compiler 104A-C facilitates training of the (one or more) machine learning models 124 using training data. In some examples, the accelerator compiler 104A-C utilizes training data derived from locally generated data. In some examples, the accelerator compiler 104A-C utilizes training data derived from externally generated data. In some examples using supervised training, the accelerator compiler 104A-C can label the training data. The label is applied to the training data manually by a user or by an automated data preprocessing system. In some examples, the accelerator compiler 104A-C can preprocess the training data using, for example, an interface (e.g., interface circuit 114). In some examples, the accelerator compiler 104A-C subdivides the training data into: a first portion of data for training (one or more) machine learning models 124, and a second portion of data for validating (one or more) machine learning models 124.

[0064] Once training is complete, the accelerator compiler 104A-C may deploy the machine learning model(s) 124 for use as an executable construct that processes inputs and provides outputs based on the network of nodes and connections defined in the machine learning model(s) 124. The accelerator compiler 104A-C may store the machine learning model(s) 124 in the data storage 120. In some examples, the accelerator compiler 104A-C may call the interface circuit 114 to transmit the machine learning model(s) 124 to one (or more) of the external electronic systems 130. In some such examples, in response to transmitting the machine learning model(s) 124 to one (or more) of the external electronic systems 130, the one (or more) of the external electronic systems 130 may execute the machine learning model(s) 124 to perform the AI / ML workload with at least one of improved efficiency or performance.

[0065] Once trained, one (or more) of the deployed (one or more) machine learning models 124 can be operated to process data in the inference phase. In the inference phase, the data to be analyzed (e.g., live data) is input to the (one or more) machine learning models 124, and the (one or more) machine learning models 124 are executed to create output. This inference phase can be thought of as the AI ​​"thinking" to generate output based on what it learned from training (e.g., applying learned patterns and / or associations to live data by executing the (one or more) machine learning models 124). In some examples, the input data undergoes pre-processing before being used as input to the (one or more) machine learning models 124. In addition, in some examples, the output data may undergo post-processing after it is generated by the (one or more) machine learning models 124 to transform the output into a useful result (e.g., display of data, detection and / or recognition of objects, instructions to be executed by the machine, etc.).

[0066] In some examples, the output of one (or more) of the deployed (one or more) machine learning models 124 can be captured and provided as feedback. By analyzing the feedback, the accuracy of one (or more) of the deployed (one or more) machine learning models 124 can be determined. If the feedback indicates that the accuracy of the deployed model is below a threshold or other criteria, the feedback and updated training data sets, hyperparameters, etc. can be used to trigger the training of the updated model to generate an updated deployed model.

[0067] In some examples, the accelerator compiler 104A-C configures one (or more) of the acceleration circuits 108, 110 to perform a convolution operation, such as a 2D convolution operation. For example, the acceleration circuits 108, 110 may implement a CNN. In some examples, the CNN ingests and / or otherwise processes an image in the form of a tensor, which is a matrix of numbers with additional dimensions. For example, the CNN may obtain an input image represented by a 3D tensor, where the first and second dimensions correspond to the width and height of the matrix, and the third dimension corresponds to the depth of the matrix. For example, the width and height of the matrix may correspond to the width and height of the input image, and the depth of the matrix may correspond to the color depth (e.g., color layers) or color encoding (e.g., red, green, blue (RGB) encoding) of the image.

[0068] A typical CNN may also receive an input and transform the input through a series of hidden layers. For example, a CNN may have multiple convolutional layers, pooling layers, and / or fully connected layers. In some such examples, a CNN may have multiple layer triplets, including convolutional layers, pooling layers, and fully connected layers. In some examples, a CNN may have multiple convolutional layer and pooling layer pairs that output to one or more fully connected layers. In some examples, a CNN may include 20 layers, 30 layers, and so on.

[0069] In some examples, the acceleration circuits 108, 110 can execute a convolution layer to apply a convolution function or operation to map an image of an input (previous) layer to a next layer in the CNN. In some examples, the convolution can be three-dimensional (3D) because each input layer can have multiple input features (e.g., input channels) associated with the input image. The acceleration circuits 108, 110 can execute a convolution layer to perform convolution by forming a regional filter window in each individual input channel and generating output data or activations by calculating the product of (1) filter weights associated with the regional filter window and (2) the input data covered by the regional filter window. For example, the acceleration circuits 108, 110 can determine the output features of the input image by scanning multiple input channels including multiple regional filter windows using a convolution filter.

[0070] In some examples, the acceleration circuits 108, 110 can perform a pooling layer to extract information from a set of activations in each output channel. The pooling layer can perform a maximum pooling operation corresponding to a maximum pooling layer, or an average pooling operation corresponding to an average pooling layer. In some examples, the maximum pooling operation may include selecting the maximum value of the activations within the pooling window. In some examples, the average pooling operation may include calculating the average value of the activations within the pooling window.

[0071] In some examples, the acceleration circuits 108, 110 may execute a fully connected layer to obtain data computed by the convolutional layer(s) and / or the pooling layer(s), and / or classify the data into one or more categories. In some examples, the fully connected layer may determine whether the classified data corresponds to a particular image feature of the input image. For example, the acceleration circuits 108, 110 may execute a fully connected layer to determine whether the classified data corresponds to a simple image feature (e.g., a horizontal line) or a more complex image feature, such as an animal (e.g., a cat).

[0072] In some examples, the accelerator compiler 104A-C may configure one (or more) of the acceleration circuits 108, 110 to perform a non-2D convolution operation as a 2D convolution operation. For example, the accelerator compiler 104A-C may configure one (or more) of the acceleration circuits 108, 110 to implement a depthwise convolution operation, an elementwise addition operation, a grouped convolution operation, a dilated convolution operation, a custom operation (e.g., a custom convolution, a custom acceleration operation, etc.), etc., as a 2D convolution operation. In some such examples, the accelerator compiler 104A-C may instruct one (or more) of the acceleration circuits 108, 110 to generate data internally, rather than receiving data from the accelerator compiler 104A-C, the configuration data 122, etc. For example, the accelerator compiler 104A-C may instruct the first acceleration resource to generate at least one of activation sparse data, weight sparse data, or weight data based on the acceleration operation to be performed by the first acceleration circuit 108. In some such examples, the accelerator compilers 104A-C may instruct one (or more) of the acceleration circuits 108, 110 to execute one (or more) of the ML model(s) 124 based on data generated by the one (or more) of the acceleration circuits 108, 110, which may be based on a convolution operation to be performed by the one (or more) of the acceleration circuits 108, 110.

[0073] Figure 2 is a block diagram of an example accelerator compiler 200. In some examples, Figure 2 The accelerator compiler 200 can implement Figure 1 One or more of the accelerator compilers 104A-C may be configured to perform scaling transformations of operands to improve computational and / or resource efficiency and operation accuracy while using lower precision operand values. Figure 2 The accelerator compilers 104A-104C may be instantiated (e.g., an instance of it is created, it exists for any length of time, is materialized, implemented, etc.) by a processor circuit such as a central processing unit that executes instructions. Additionally, or alternatively, Figure 1The accelerator compilers 104A-104C may be instantiated (e.g., an instance of it is created, it exists for any length of time, is materialized, implemented, etc.) by an ASIC or FPGA configured to perform operations corresponding to the instructions. It should be understood that Figure 2 Some or all of the circuits may thus be instantiated at the same or different times. Some or all of the circuits may be instantiated, for example, in one or more threads that execute simultaneously on hardware and / or serially on hardware. Furthermore, in some examples, Figure 2 Some or all of the circuitry may be implemented by microprocessor circuitry executing instructions to implement one or more virtual machines and / or containers.

[0074] exist Figure 2 In the illustrated example, the accelerator compiler 200 may configure a hardware accelerator, such as an example interface circuit 210, an example data precision selection circuit 220, an example scaling factor technology selection circuit 230, an example scaling factor determination circuit 240, an example model execution circuit 250, an example executable file generation circuit 260, an example bus 280, and an example data storage device 270, and also includes an example (one or more) machine learning (ML) model 272, an example operator value 274, an example scaling factor 276, and (one or more) example executable files 278.

[0075] In operation, the example interface circuit 210 obtains any number of operators (e.g., operator values ​​274) to perform a machine learning (ML) operation (e.g., a dot product operation). In the examples disclosed herein, the interface circuit 210 may obtain the operator values ​​274 from the example data storage device 270. In the examples disclosed herein, this source may be any type of database, Internet source, etc. In addition, in the examples disclosed herein, the interface circuit 210 may receive and / or send data (e.g., operator values ​​274) to a network or Figure 1 Other parts of the electronic system 102, such as Figure 1 In some examples, the interface circuit 210 may also be an interface and / or cable from a laptop to a GPU and / or FPGA for configuring operations, such as loading an image. In some examples, the example interface circuit 210 is executed by the interface circuit 210 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0076] In some examples, interface circuit 210 includes means for obtaining an arbitrary number of operators (e.g., operator values ​​274) to perform a machine learning (ML) operation (e.g., a dot product operation). For example, means for obtaining an arbitrary number of operators (e.g., operator values ​​274) to perform a ML operation (e.g., a dot product operation) may be implemented by interface circuit 210. In some examples, interface circuit 210 may be instantiated by a processor circuit, such as Fig.12 For example, the interface circuit 210 may be Fig.13 The example microprocessor 1300 performs at least Fig.11 In some examples, the interface circuit 210 may be instantiated by a hardware logic circuit that may be configured to perform operations corresponding to the machine-readable instructions. Fig.14 The interface circuit 210 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally or alternatively, the interface circuit 210 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the interface circuit 210 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0077] The example data precision selection circuit 220 determines both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed. For example, the data precision selection circuit 220 may determine an initial format of FP32 for each operand and / or value, and a scaled format of BF16 to improve computational efficiency. In some examples, the example data precision selection circuit 220 is executed by the data precision selection circuit 220 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0078] In some examples, the data precision selection circuit 220 includes means for determining both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed. For example, the means for determining both a low-precision data representation type and a high-precision data representation type associated with the execution of the ML model for which scaling is to be performed can be implemented by the data precision selection circuit 220. In some examples, the data precision selection circuit 220 can be instantiated by a processor circuit, such as Fig.12 For example, the data precision selection circuit 220 may be Fig.13 The example microprocessor 1300 performs at least Figure 8 and / or Fig.11 In some examples, the data precision selection circuit 220 may be instantiated by a hardware logic circuit that may be configured to perform operations corresponding to the machine-readable instructions. Fig.14 The data precision selection circuit 220 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally, or alternatively, the data precision selection circuit 220 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the data precision selection circuit 220 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0079] The example scaling factor technique selection circuit 230 performs pre-processing of the data in order to select a scaling technique to be employed by the scaling factor determination circuit 240. That is, for example, the scaling factor technique selection circuit 230 may consider accuracy acceptability constraints, the amount of data, the specific operation to be performed, etc. Equation 1 shown below shows that two vectors x i and w i Example dot product of (where all values ​​are represented in FP32 format). Formula 2 similarly shows the BF16 representation of the two vectors in Formula 1 for the same dot product operation x' i 、w' i .

[0080]

[0081] The resulting difference in value between each dot product operation may accordingly be expressed as shown below in Equation 3. The example scaling factor technique selection circuit 230, in conjunction with the example scaling factor determination circuit 240, selects a scaling factor intended to reduce the value of the resulting difference (Δ)), which represents any potential loss of accuracy in the conversion of values ​​between high-precision and low-precision data representation formats.

[0082]

[0083] In order to reduce the value of the overall dot product value difference (Δ), the scaling factor technique selection circuit 230 determines x′ by using weighting and / or scaling factors to account for any rounding and / or truncation errors. i Δ wi +w′ i Δ xi The minimum value of the sum. In the first term x′ i Δ wi In the example, the input x′ of the ML / AI model is i It may be dynamic during inference, and its associated weight value w′ i is a static value. Therefore, due to the single scaling factor Δ wi is applied to all weights, so the effect of this factor in the first term is constant, whereas in the product x′ i Δ wi is constant in , so it can be imagined that if the weights are rescaled accordingly after the dot product operation, then Δ wi The value of can be greatly reduced, thereby reducing its contribution to the overall loss (e.g., difference) value. The second term w′ i Δ xi It is difficult to reduce as effectively as the first term because the scaling factor technique selection circuit 230 cannot accurately predict every unknown input of the ML / AI model before the inference stage. However, by training the model with weights and / or scaling factors by the scaling factor technique selection circuit 230 and / or the example scaling factor determination circuit 240, the weights can be normalized to smaller values ​​(e.g., by penalizing larger w i value), thereby effectively reducing the overall difference value (e.g., indicating a reduced loss of accuracy). In some examples, the example scaling factor technique selection circuit 230 is executed by the scaling factor technique selection circuit 230 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0084] In some examples, the scaling factor technique selection circuit 230 includes means for selecting a scaling technique for use by the scaling factor determination circuit 240 to perform a machine learning (ML) operation (e.g., a dot product operation) while maintaining high accuracy. For example, the means for selecting a scaling technique for use by the scaling factor determination circuit 240 to perform a machine learning (ML) operation (e.g., a dot product operation) while maintaining high accuracy can be implemented by the scaling factor technique selection circuit 230. In some examples, the scaling factor technique selection circuit 230 can be instantiated by a processor circuit, such as Fig.12 For example, the scaling factor technique selection circuit 230 may be configured to select the scaling factor technique of the example processor circuit 1212. Fig.13 The example microprocessor 1300 executes machine executable instructions to instantiate, for example, at least Figure 8 , Fig. 9 and / or Fig.10 In some examples, the scaling factor technology selection circuit 230 may be instantiated by hardware logic circuitry that is constructed to perform operations corresponding to the machine-readable instructions. Fig.14 The scaling factor technology selection circuit 230 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally, or alternatively, the scaling factor technology selection circuit 230 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the scaling factor technology selection circuit 230 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or perform some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0085] The example scaling factor determination circuit 240 determines the scaling factor to be used in the conversion and / or de-conversion of a high-precision data representation format (e.g., FP32) to a low-precision data representation format (e.g., BF16) based on the preprocessing performed by the scaling factor technique selection circuit 230. In the examples disclosed herein, the scaling and / or de-scaling is performed by the scaling factor determination circuit 240 in the following manner: first, a target value associated with each FP32-represented operand is calculated, and the target value is obtained by masking a number of lower-order bits to reduce the associated memory footprint. In the examples disclosed herein, the lower 16-bit mantissa bits (e.g., the least significant bits) of each FP32 representation value will be masked (e.g., set to zero) in order to determine each target value, as shown in the example formula 4 below (where w FP32 represents the original FP32 representation value, and t FP32 represents its corresponding target value, where the lower 16 mantissa bits are masked).

[0086] w FP32 =0x3f937fff

[0087] t FP32 =0x3f9e0000

[0088] Formula 4

[0089] Once the scale factor technique selection circuit 230 determines each corresponding target value, the scale factor (s) is determined by Equation 5, as shown below, at which minimal or no loss of precision associated with the target BF16 representation value is observed.

[0090]

[0091] In the examples disclosed herein, when the scaling factor determination circuit 240 selects the scaling factor, the exponent and / or sign bits associated with each of the BF16 and / or FP32 are not important because each associated target value t FP32 With its corresponding weight value w FP32 Same sign and / or exponent. Therefore, only information loss reduction, bias reduction and / or truncation error reduction related to the lower order mantissa bits is performed by the scaling factor determination circuit 240.

[0092] Furthermore, in the examples disclosed herein, when the value w of a single element, operand, and / or value is known in advance by the scaling factor determination circuit 240 and remains constant, its optimal scaling factor can be found by enumerating all FP32 values ​​between 1.0 and 2.0 (e.g., values ​​that can be discretely represented on a particular computer device). By enumerating such values, the scaling factor determination circuit 240 can select the scaling factor that has the smallest precision loss associated with all weight values. That is, in these examples, when abs((w*s) FP32 -(w*s) BF16 ) is reduced, the scaling factor s may be selected by the scaling factor determination circuit 240. In these examples, when the value of w is not known in advance by the scaling factor determination circuit 240 and / or is dynamic, the mantissa bits of the result value (e.g., after the operation) are considered by the scaling factor determination circuit 240. For example, the following formula 6 shows two example FP32 values, which are represented below in their bit-by-bit format.

[0093]

[0094] When w and s in Formula 6 are multiplied together, taking into account the mantissa bits (i.e. and ) values, which will produce a 47 (when the result is less than 2) or 48 (when the result is greater than or equal to 2) bit result value. In order to return to the original FP32 value representation format, the lowest-order 23 bits or 24 bits in the mantissa must be discarded. In addition, when the FP32 result is projected into the BF16 value representation format, the lowest-order 16 mantissa bits will need to be further truncated (e.g., by the scaling factor determination circuit 240) in order to meet the standards of the particular value representation format. That is, the value of the discarded mantissa bits (e.g., in integer form) is proportional to the loss of precision experienced when the scaled FP32 value is projected into the BF16 format (e.g., by the scaling factor determination circuit 240), as shown in Formula 7 below. Therefore, when (one or more) input values ​​w are unknown, when enumerating all FP32 values ​​between 1.0 and 2.0 (e.g., values ​​that can be discretely represented on a particular computer device), the scaling factor determination circuit 240 selects the scaling factor with the smallest associated discarded value and therefore the smallest associated loss of precision.

[0095]

[0096] In some examples, the example scaling factor determination circuit 240 is executed by the scaling factor determination circuit 240 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0097] In some examples, the scaling factor determination circuit 240 includes means for determining a scaling factor to be used in the conversion and / or deconversion of a high precision data representation format (e.g., FP32) to a low precision data representation format (e.g., BF16). For example, the means for determining a scaling factor to be used in the conversion and / or deconversion of a high precision data representation format (e.g., FP32) to a low precision data representation format (e.g., BF16) may be implemented by the scaling factor determination circuit 240. In some examples, the scaling factor determination circuit 240 may be instantiated by a processor circuit, such as Fig.12 For example, the scaling factor determination circuit 240 may be configured to determine the scaling factor of the circuit 1212. Fig.13 The example microprocessor 1300 executes machine executable instructions to instantiate, for example, at least Figure 8 and / or Fig. 9 In some examples, the scaling factor determination circuit 240 may be instantiated by hardware logic circuitry that is configured to perform operations corresponding to the machine-readable instructions. Fig.14 The scaling factor determination circuit 240 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally, or alternatively, the scaling factor determination circuit 240 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the scaling factor determination circuit 240 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0098] The example model execution circuitry 250 applies weights and / or scaling factors determined by the scaling factor determination circuitry 250 to all values ​​utilized by the ML / AI model. In some examples, the example model execution circuitry 250 is executed by the model execution circuitry 250 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0099] In some examples, model execution circuitry 250 includes means for applying weights and / or scaling factors determined by scaling factor determination circuitry 250 on all values ​​utilized by the ML / AI model. For example, means for applying weights and / or scaling factors determined by scaling factor determination circuitry 250 on all values ​​utilized by the ML / AI model may be implemented by model execution circuitry 250. In some examples, model execution circuitry 250 may be instantiated by a processor circuit, such as Fig.12 For example, the model execution circuit 250 may be Fig.13 The example microprocessor 1300 executes machine executable instructions to instantiate, for example, at least Figure 8 and / or Fig.11 In some examples, the model execution circuit 250 may be instantiated by a hardware logic circuit that is constructed to perform operations corresponding to the machine-readable instructions. Fig.14 The model execution circuit 250 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally or alternatively, the model execution circuit 250 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the model execution circuit 250 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0100] The example executable file generation circuit 260 outputs an executable file that can be executed and / or instantiated by an accelerator (e.g., a GPU, an FPGA, etc.) to perform an ML / AI workload. In some examples, the example executable file generation circuit 260 is executed by the executable file generation circuit 260 instructions and / or is configured to perform, for example Figure 8 , Fig. 9 , Fig.10 and / or Fig.11 The operations represented by the flowchart are instantiated by a processor circuit.

[0101] In some examples, the executable file generation circuit 260 includes a device for outputting an executable file that an accelerator (e.g., GPU, FPGA, etc.) can execute and / or instantiate to execute the ML / AI workload. For example, the device for outputting an executable file that an accelerator (e.g., GPU, FPGA, etc.) can execute and / or instantiate to execute the ML / AI workload can be implemented by the model execution circuit 260. In some examples, the model execution circuit 260 can be instantiated by a processor circuit, such as Fig.12 For example, the model execution circuit 260 can be Fig.13 The example microprocessor 1300 executes machine executable instructions to instantiate, for example, at least Figure 8 and / or Fig. 9 In some examples, the model execution circuit 260 may be instantiated by a hardware logic circuit that is configured to perform operations corresponding to the machine-readable instructions. Fig.14 The model execution circuit 260 may be implemented by an ASIC, XPU, or FPGA circuit 1400. Additionally or alternatively, the model execution circuit 260 may be instantiated by any other combination of hardware, software, and / or firmware. For example, the model execution circuit 260 may be implemented by at least one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, XPUs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to execute some or all machine-readable instructions and / or to execute some or all operations corresponding to machine-readable instructions without executing software or firmware, but other structures are equally appropriate.

[0102] exist Figure 2 In the illustrated example of , the interface circuit 210, the data precision selection circuit 220, the scaling factor technique selection circuit 230, the scaling factor determination circuit 240, the model execution circuit 250, the executable file generation circuit 260 and the data storage device 270 (including (one or more) machine learning (ML) models 272, operation operator values ​​274, scaling factors 276 and (one or more) executable files 278) communicate with one (or more) of each other via a bus 280. For example, the bus 280 can be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a peripheral component interconnect (PCI) bus, or a peripheral component interconnect express (PCIe or PCIE) bus. Additionally, or alternatively, the bus 280 can be implemented by any other type of computing or electrical bus.

[0103] In the examples disclosed herein, the example data storage device 270 can be any type of data storage device, database, etc., containing (one or more) example machine learning (ML) models 272 and example operation operator values ​​274 obtained by the interface circuit 210, the example scaling factor 276 determined by the scaling factor technology selection circuit 230 and the scaling factor determination circuit 240, and (one or more) example executable files 278 used and / or generated by the model execution circuit 250 and the executable file generation circuit 260.

[0104] Although in Figure 2 The implementation is shown in the figure Figure 1 104A-C, but Figure 2 One or more of the elements, processes and / or devices shown in the example may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, the example interface circuit 210, the example data precision selection circuit 220, the example scaling factor technology selection circuit 230, the example scaling factor determination circuit 240, the example model execution circuit 250, the example executable file generation circuit 260, and / or more generally, Figure 1 The example accelerator compiler 104A-C may be implemented in hardware alone or in combination with software and / or firmware. Thus, for example, the example interface circuit 210, the example data precision selection circuit 220, the example scaling factor technique selection circuit 230, the example scaling factor determination circuit 240, the example model execution circuit 250, the example executable file generation circuit 260, and / or more generally, Figure 1 Any of the example accelerator compilers 104A-C may be implemented by a processor circuit, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable processors, (one or more) programmable microcontrollers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs), and / or (one or more) field programmable logic devices (FPLDs) (e.g., field programmable gate arrays (FPGAs)). In addition, Figure 1 Example accelerator compilers 104A-C may include in addition to Figure 2 In addition to or in place of those shown in Figure 2 One or more of the elements, processes and / or devices shown in the drawings, and / or may include any or all of more than one of the illustrated elements, processes and devices.

[0105] Figure 3 is a diagram of an example conventional convolution operation 300, which may be performed by Figure 1 The first acceleration circuit 108, Figure 1 The second acceleration circuit 110 and / or Figure 2 The accelerator compiler 200 of the embodiment of the present invention is executed. In some examples, the traditional convolution operation 300 can implement spatial convolution on one or more images (e.g., pictures, still frames of a video, etc.) and / or operations (e.g., dot product operations). In some examples, the accelerator compiler 300 can be configured to operate in a traditional convolution mode, a 2D convolution mode, a three-dimensional (3D) convolution mode, etc. based on the traditional convolution operation to be performed by the accelerator compiler 200.

[0106] The conventional convolution operation 300 includes applying an example filter 302 to an example input tensor 304 to generate an example output tensor 306. In this example, the input tensor 304 is a 3D object with a size of x i *y i *z i In this example, there are K filters 302, each filter 402 has a size of f*f*z k Alternatively, any other size may be used to implement one (or more) of filters 302. For example, one or more of filters 402 may have f x *f y *z k The size of x and y can be different, so that f x and f y In this example, filter 302 is a square filter, so that f x Equal to f y , but the examples described in this article are not limited to this. In this example, the size of the output tensor 306 is x o *y o *z o In this example, z k =z i And z o = K. In this example, filter 402 is applied to input tensor 404 together with a nonlinear activation function to produce output tensor 306. For example, Figure 2 The accelerator compiler 200 may obtain the input tensor 304 as activation data, obtain one of the filters 302 as weight data, and output an output tensor 306. In some such examples, the accelerator compiler 200 may implement the traditional convolution operation 300 in a "dense" manner, while in other examples, the accelerator compiler 200 may implement the traditional convolution operation 300 with sparsity.

[0107] Advantageously, the accelerator compiler 300 can perform the convolution operation 300 based on sparse data to reduce the amount of computation. For example, the accelerator compiler 200 can obtain and / or generate activation sparse data and / or weight sparse data to output the output tensor 306 by calling a sparse technique.

[0108] Figure 4 A bit-by-bit binary representation 400 of an example FP32 value is illustrated. In the illustrated bit-by-bit binary representation 400, 1 sign bit 402, 8 exponent bits 404, and 23 fraction (mantissa) bits 406 are shown, along with their corresponding bit indices 408. In this illustrated example, the leftmost bits (e.g., sign bit 402, exponent bit 404) that are enumerated as higher values ​​according to bit index 408 are not considered by the scale factor technique selection circuit 230 and / or the scale factor determination circuit 240 when scaling, which is described in more detail herein. When performing scaling between data representation formats, the lowest order bits (e.g., mantissa bits 406) represented by the lower values ​​enumerated by bit index 408 are masked and / or truncated by the scale factor technique selection circuit 230 and / or the scale factor determination circuit 240. Figure 5 Pictured Figure 2 The example enumeration process for selecting a scaling factor is performed by the example scaling factor determination circuit 240. Figure 2 As described, the input values ​​of individual elements, operands, and / or values ​​are Figure 2 The example scaling factor determination circuit 240 of is known and always unchanged, so the optimal scaling factor for the value(s) can be found by enumerating all FP32 values ​​between 1.0 and 2.0 (e.g., values ​​that can be discretely represented on a particular computing device). By enumerating such values, Figure 2 The scaling factor determination circuit 240 may select the scaling factor associated with the smallest loss of precision. Figure 5 The illustrated example of depicts a scaling factor enumeration graph 500 in which discarded integer values ​​502 are plotted against enumerated scaling factors 504. In the examples disclosed herein, the discarded integer values ​​502 represent values ​​discarded from a value in a conversion to a lower precision data representation format (e.g., integer form) and are proportional to the loss of precision when converting between data representation formats. The scaling factor enumeration graph 500 is a periodic variation graph of an arbitrary input value (e.g., 1.66666667) showing a trend of discarded integer values ​​502 as the enumerated scaling factor 504 varies. As shown in FIG. Figure 5 The minimum drop change point 506 shown in the illustrated example of represents the point at which the minimum drop integer value 502 loss is observed, and thus, accordingly, the minimum drop change point 506 further represents the point at which the loss of precision is minimal. Thus, the enumerated scaling factor 504 of the x-axis at this coordinate will be Figure 2 The scaling factor determination circuit 240 selects as the scaling factor to be applied to the ML / AI model value.

[0109] Figure 6 An example comparative accuracy graph 600 is shown for an example dot product operation using a baseline FP32 value 602, a converted BF16 value 604, and a scaled BF16 value 606. As depicted in the comparative accuracy graph 600, the dot product operation performed using the scaled BF16 value 606 exhibits the greatest change in accuracy compared to what the baseline FP32 value 602 can provide. The comparative accuracy graph 600 further shows that the accuracy results associated with the scaled BF16 value 606 more closely follow the baseline FP32 value, unlike the converted BF16 value 604, indicating that a higher level of accuracy is achieved by using the scaling techniques described herein when using a lower precision data representation format to improve efficiency of computation and / or reduce resource consumption.

[0110] Figure 7 For using the baseline FP32 value 602 ( Figure 6 ), the converted BF16 value 604 ( Figure 6 ), a first technique's scaled BF16 value 704, and a second technique's scaled BF16 value 706 illustrate an example accuracy percentage table 700 on a set of tested machine learning (ML) and / or artificial intelligence (AI) models 702. Figure 2 In more detail, the scaled BF16 value 704 of the first technique corresponds to the value obtained by Figure 2 Similarly, the scaled BF16 value 706 of the second technique corresponds to the BF16 value 706 obtained by the scaling factor selection circuit 230 and / or the scaling factor determination circuit 240 when the input value (w) is known. Figure 2 The rightmost column of the accuracy percentage table 700 shows the average accuracy 708 provided on all tested ML / AI models 702. The average accuracy 708 associated with the baseline FP32 value 602 represents the ideal accuracy value on the tested ML / AI model 702. The associated average accuracy 708 shown by the scaled BF16 value 704 of the first technique and the scaled BF16 value 706 of the second technique are both greater than the associated average accuracy of the converted BF16 value 604 (e.g., the average accuracy 708 closest to the baseline FP32 value 602), further indicating that high operating accuracy is achieved through the scaling process (e.g., both the first scaling technique and / or the second scaling technique) even when using a lower precision data representation format (e.g., BF16).

[0111] exist Figure 8-Figure 11 A flowchart representing example machine-readable instructions is shown in Figure 2 The machine-readable instructions may be one or more executable programs or (one or more) parts of executable programs for execution by a processor circuit, such as the one described below. Fig.12 The processor circuit 1212 shown in the example processor platform 1200 discussed below and / or in connection with Fig.13 and / or Fig.14 Example processor circuits discussed. The program may be embodied in software stored on one or more non-transitory computer-readable storage media, such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disc, a volatile memory (e.g., any type of random access memory (RAM), etc.) or a non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), flash memory, HDD, SSD, etc.), associated with a processor circuit located in one or more hardware devices, but the entire program and / or a portion thereof may also be executed by one or more hardware devices other than the processor circuit and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that facilitates communication between a server and an endpoint client hardware device). Similarly, a non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, although reference is made to Figure 8-Figure 11The example program is described by the flowchart shown in , but many other methods of implementing the example accelerator compiler 200 may be used alternatively. For example, the execution order of the blocks may be changed, and / or some of the described blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are constructed to perform the corresponding operations without executing software or firmware. The processor circuit may be distributed in different network locations and / or in one or more hardware devices in a single machine (e.g., a single-core processor (e.g., a single-core central processing unit (CPU)), a multi-core processor (e.g., a multi-core CPU, an XPU, etc.)), multiple processors distributed on multiple servers in a server rack, multiple processors distributed on one or more server racks, and local to a CPU and / or FPGA in the same package (e.g., the same integrated circuit (IC) package or in two or more separate housings, etc.).

[0112] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, and the like. The machine-readable instructions described herein may be stored as data or data structures (e.g., stored as portions of instructions, code, representations of code, and the like) that may be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in the cloud, in an edge device, and the like). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, and the like, so that they may be directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, machine-readable instructions may be stored as multiple portions that are individually compressed, encrypted, and / or stored on separate computing devices, where the portions, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that may together form a program such as that described herein.

[0113] In another example, machine-readable instructions may be stored in a state in which they can be read by processor circuitry but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., to store settings, input data, record network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Thus, a machine-readable medium as used herein may include machine-readable instructions and / or program(s), regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in transit.

[0114] The machine-readable instructions described herein may be represented by any past, current, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented by any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0115] As described above, the present invention may be implemented using executable instructions (eg, computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media. Figure 8-Figure 11, the medium is, for example, an optical storage device, a magnetic storage device, an HDD, a flash memory, a read-only memory (ROM), a CD, a DVD, a cache, any type of RAM, a register, and / or any other storage device or storage disk in which information can be stored for any duration (e.g., storage for a longer period of time, permanent storage, short-term storage, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer-readable medium, non-transitory computer-readable storage medium, non-transitory machine-readable medium, and non-transitory machine-readable storage medium are expressly defined to include any type of computer-readable storage device and / or storage disk, and exclude propagation signals and exclude transmission media. As used herein, the terms "computer-readable storage device" and "machine-readable storage device" are defined to include any physical (mechanical and / or electrical) structure that stores information, but exclude propagation signals and exclude transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant arrays of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as mechanical and / or electrical equipment, hardware, and / or circuitry, which may or may not be configured with computer-readable instructions, machine-readable instructions, etc., and / or manufactured to perform computer-readable instructions, machine-readable instructions, etc.

[0116] "Include" and "comprising" (and all forms and tenses thereof) are used as opening terms herein. Thus, whenever a claim employs any form of "include" or "comprising" (e.g., includes, comprises, has, etc.) as a preamble or used in any type of claim recitation, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is opening in the same manner as the terms "include" and "comprising" are opening. The term "and / or" when used, for example, in a form such as A, B, and / or C, refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, C, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or operation of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or operation of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to refer to an implementation that includes any of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0117] As used herein, singular references (e.g., "a," "first," "second," etc.) do not exclude a plurality. As used herein, the term "an" object refers to one or more of that object. The terms "a," "one or more," and "at least one" are used interchangeably herein. Furthermore, although listed separately, multiple means, elements, or method actions may be implemented, for example, by the same entity or object. Furthermore, although individual features may be included in different examples or claims, they may be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or not advantageous.

[0118] Figure 8 The flowchart of 800 represents example machine readable instructions and / or example operations that may be executed and / or instantiated by a processor circuit to perform a scaling transformation of lower precision data representation format values ​​to achieve greater computational accuracy and / or efficiency. Figure 8 The machine-readable instructions and / or operations 800 begin at block 802, where the data precision selection circuit 220 identifies a first precision data type and a second precision data type associated with the acceleration hardware (e.g., the accelerator compiler 200) executing a machine learning (ML) model. As described herein, the type of data representation format adopted by the accelerator compiler 200 may vary based on hardware and / or execution constraints. In the examples disclosed herein, FP32 and BF16 data representation formats are described, the former being a high-precision data representation format and the latter being a low-precision data representation format. However, in other examples, any other type of data type at high and low precision levels may be identified by the scaling factor determination circuit 240 at block 802.

[0119] At block 804, the model execution circuit 250 determines a scaling factor of a first weight having a first precision data type to be applied to a machine learning (ML) model. In an embodiment of the present disclosure, the first precision data type may be a high precision data type or a low precision data type, depending on the application. The first weight determined by the model execution circuit 250 is a target value, such as a combination of Figure 2 In the examples disclosed herein, when the input value (w) is known before executing the ML model, the target value is given by Figure 2 The model execution circuit 250 calculates and / or determines.

[0120] At block 806, the model execution circuit 250 converts the first weight into a second weight having a second precision data type based on the multiplication of the first weight and the scaling factor(s). Figure 2 ) As described in more detail, the second weight may be calculated (e.g., by scaling factor determination circuitry 240 and / or model execution circuitry 250) by enumerating all possible scaling factors and determining a scaling factor that produces the lowest loss of precision during multiplication of values.

[0121] At block 808, the model execution circuit 250 executes a machine learning (ML) model based on the second weight(s). In examples disclosed herein, the same weights may be calculated and / or applied across all values ​​utilized by the ML model, however, in other examples, multiple weights may be calculated and / or applied based on a particular loss of accuracy that each value may provide.

[0122] At block 810, the scaling factor technique selection circuit 230 determines whether the accuracy of the machine learning (ML) model has improved by more than a threshold. If the scaling factor technique selection circuit 230 determines that the accuracy of the ML model has improved by more than a threshold, the process proceeds to block 812. However, if the scaling factor technique selection circuit 230 determines that the accuracy of the ML model has not improved by more than a threshold, the process proceeds to block 814.

[0123] At block 812, the scaling factor determination circuit 240 adjusts the scaling factor(s) to achieve a higher level of accuracy in response to the scaling factor technique selection circuit 230 determining at block 810 that the accuracy of the ML model has improved beyond a threshold. Figure 2 As described, the scaling factor determination circuit 240 adjusts the scaling factor(s) by enumerating all scaling values ​​to determine the next best value that gives the lowest loss of precision.

[0124] At block 814, the scaling factor technique selection circuit 230 selects another technique to determine the scaling factor(s) in response to the scaling factor technique selection circuit 230 determining at block 810 that the accuracy of the ML model has not improved beyond the threshold. For example, an example first technique may involve enumerating all possible scaling factors, and then selecting, by the scaling factor technique selection circuit 230, the factor that best reduces the loss of accuracy. If the scaling factor technique selection circuit 230 determines that another technique is to be selected to determine the scaling factor(s), the process returns to block 804. However, if the scaling factor technique selection circuit 230 determines not to select another technique, the process proceeds to block 816.

[0125] At block 816, the executable file generation circuit 260 outputs an executable file based on the machine learning (ML) model, the scaling factor(s), and the second weights. In the examples disclosed herein, this executable file may represent a set of optimal weights and / or scaling factors(s) for achieving improved computing performance while reducing resource consumption and having higher operating accuracy.

[0126] Fig. 9 The flowchart of represents example machine-readable instructions and / or example operations 900 that may be executed and / or instantiated by a processor circuit to perform a per-tensor scaling transform for all values ​​involved in the operation of an ML / AI model. Fig. 9 The machine readable instructions and / or operations 900 of the present invention begin at block 902 where the scaling factor technique selection circuit 230 selects a scaling factor selection technique. The process of the scaling factor technique selection circuit 230 selecting the scaling factor selection technique is combined with Fig.10 To further describe.

[0127] At block 904, the scaling factor technique selection circuit 230 determines whether the scaling factor selection technique is based on a scaling factor for each tensor, which in the examples disclosed herein may be a data store associated with the scaling factor(s) or the like (e.g., Figure 2 If the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is based on the scaling factor of each tensor, the process goes to block 906. However, if the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is not based on the scaling factor of each tensor, the process proceeds to block 908.

[0128] At block 906, the scaling factor determination circuitry 240 determines the scaling factor to use for each tensor. In examples disclosed herein, the scaling factor determination circuitry 240 may make this determination based on hardware constraints for the particular tensor, an associated available data representation format, and so forth.

[0129] At block 908, the scaling factor technique selection circuit 230 determines whether the scaling factor selection technique is based on a scaling factor per channel after determining at block 904 that the scaling factor selection technique is not based on a scaling factor per tensor. If the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is based on a scaling factor per channel, the process proceeds to block 910. However, if the scaling factor technique selection circuit 230 determines that the scaling factor selection technique is not based on a scaling factor per channel, the process proceeds to block 912.

[0130] At block 910, the scaling factor determination circuit 240 determines the scaling factor to be used for each channel. Similar to the determination made by the scaling factor determination circuit 240 at block 906, the scaling factor(s) of all tensors within a particular channel may be aggregated, averaged, etc. to determine a single scaling factor to be used for each channel.

[0131] At block 912, the scaling factor determination circuit 240 outputs the scaling factor(s) to be used for executing the machine learning (ML) model. In the examples disclosed herein, the scaling factor(s) may be stored in an example database (e.g., Figure 2 in the data storage device 270).

[0132] At block 914, the executable file generation circuit 260 outputs an executable file based on the machine learning (ML) model and the scaling factor(s). In the examples disclosed herein, both the ML model and the scaling factor(s) may be stored in Figure 2In the example data storage device 270 of . In the examples disclosed herein, the executable file may be specific to the hardware constraints specified with a particular ML model, but in some examples, it may be general to various types of ML models.

[0133] Fig.10 The flowchart of represents example machine-readable instructions and / or example operations 1000 that may be executed and / or instantiated by a processor circuit to select a scaling factor selection technique to be employed in order to determine scaling factors for all values ​​used for operations of an ML / AI model. Fig.10 The machine readable instructions and / or operations 1000 begin at block 1002, where the scaling factor technique selection circuitry 230 determines whether a scaling factor based on reducing the total absolute difference value between the scaled FP32 value and the converted lower precision data type using the scaling factor is the selected scaling factor technique selection. In the examples disclosed herein, the scaling factor technique selection circuitry 230 can determine any particular scaling factor technique selection selected based on pre-specified hardware constraints provided for a particular ML model, a particular workload domain, a data representation format type provided by a particular ML model and / or set of operands, and / or the like.

[0134] At block 1004, the scaling factor technique selection circuit 230 determines whether to select a scaling factor based on reducing the total absolute difference value between the scaled FP32 values ​​of the first n values ​​and the converted lower precision data type value using the scaling factor. In examples disclosed herein, the first n values ​​may represent the square root of the total number of dimensions of a set of values, indicating a sampling algorithm for the n values ​​to which the scaling factor is applied.

[0135] At block 1006, the scaling factor technique selection circuit 230 determines whether the scaling factor to be selected is based on a weighted total absolute difference value set between the scaled FP32 values ​​of the first n values ​​and the converted lower precision data type using the scaling factor. Similar to the determination made by the scaling factor technique selection circuit 230 at block 1004, the scaling factor technique selection circuit 230 determines at block 1006 whether the weighted total absolute difference value has decreased. In the example disclosed herein, the weighted total absolute difference value represents a higher level of accuracy that can be achieved by assigning a smaller weight to the logarithm in descending order. For the lower half of the values ​​(between the first n / 2 and the first n), the weight is 1, for the half of the values ​​above it (between the first n / 4 and the first n / 2), the weight is 2, and so on and so forth. Thus, the weight of the first 1 value is (1+log2n), so it is a technique that biases accuracy more highly without overfitting scaling.

[0136] At block 1008, the scaling factor technique selection circuit 230 determines whether to select a scaling factor based on minimizing the total absolute difference value between the scaled FP32 value and the converted lower precision data type value using the scaling factor to reduce the variation in the difference value, which represents another variation that may provide a higher level of accuracy.

[0137] At block 1010, the scale factor technique selection circuit 230 selects a scale factor based on another technique different from the techniques of blocks 1002-1008.

[0138] At block 1012 , the scale factor technique selection circuit 230 outputs the selected scale factor selection technique.

[0139] Fig.11 The flowchart of represents example machine readable instructions and / or example operations 1100 that may be executed and / or instantiated by a processor circuit to perform descaling of lower precision values ​​once operations performed by an ML / AI model are completed. Fig.11 The machine-readable instructions and / or operations 1100 of the embodiment of the present invention begin at block 1102, where the interface circuit 210 obtains one (a set of) operation operators to perform machine learning operations. In the examples disclosed herein, the operation operators may be stored in a database (e.g., Figure 2 The data storage device 270 may be provided in conjunction with a particular ML model and / or selected operations.

[0140] At block 1104, the data precision selection circuit 220 determines whether the operation operator is based on a lower precision data type. In the examples disclosed herein, via a database (e.g., Figure 2 Some operators provided by the model execution circuit 250 (e.g., the data storage device 270 of the embodiment of the present invention) may not need to be further scaled to a lower precision data type if they are already provided as a low precision data type and / or cannot be further reduced. Therefore, if the model execution circuit 250 determines that the operator is based on a lower precision data type, the process goes to block 1106. However, if the model execution circuit 250 determines that the operator is not based on a lower precision data type, the process proceeds to block 1108.

[0141] At block 1106, the model execution circuit 250 scales the operator using the scaling factor(s). In the examples disclosed herein, the scaling factor(s) may be provided in the form of weights to be input to the ML model and / or multiplied with each operator value.

[0142] At block 1108, the model execution circuit 250 determines output values ​​of the machine learning (ML) operation based on the scaled operation operator. In the examples disclosed herein, these output values ​​can be the results of the operation performed on the scaled operation operator.

[0143] At block 1110, the model execution circuitry 250 descales the output value determined by the model execution circuitry 250 at block 1108. In examples disclosed herein, descaling is performed, for example, by dividing (e.g., multiplying by the inverse of) the output value by the same scaling factor that was multiplied thereto during scaling, thereby effectively reversing the scaling process.

[0144] At block 1112, the model execution circuit 250 outputs the descaled output value to another logical entity. In the examples disclosed herein, the (one or more) logical entities may be, for example, Figure 1 An external computing device 130, or any other type of computing device and / or logical entity.

[0145] At block 1114, the model execution circuit 250 determines whether to perform another machine learning (ML) operation (e.g., using the same and / or different operators). If the model execution circuit 250 determines to perform another ML operation, the process returns to block 1102. However, if the model execution circuit 250 determines not to perform another ML operation, the process ends.

[0146] Fig.12 is a block diagram of an example processor platform 1200 that is configured to execute and / or instantiate Figure 8-Figure 11 Machine readable instructions and / or operations to implement Figure 2 The processor platform 1200 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.). TM , a tablet device such as a iPod, a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headphones (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

[0147] The processor platform 1200 of the illustrated example includes a processor circuit 1212. The processor circuit 1212 of the illustrated example is hardware. For example, the processor circuit 1212 can be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired family or manufacturer. The processor circuit 1212 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the processor circuit 1212 implements the example interface circuit 210, the example data precision selection circuit 220, the example scaling factor technology selection circuit 230, the example scaling factor determination circuit 240, the example model execution circuit 250, and / or the example executable file generation circuit 260.

[0148] The processor circuit 1212 of the illustrated example includes a local memory 1213 (e.g., cache, registers, etc.). The processor circuit 1212 of the illustrated example communicates with a main memory including a volatile memory 1214 and a non-volatile memory 1216 via a bus 1218. The volatile memory 1214 may be a synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), Dynamic Random Access Memory (DRAM) Dynamic Random AccessMemory, ) and / or any other type of RAM device. The non-volatile memory 1216 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1214, 1216 of the illustrated example is controlled by a memory controller 1217.

[0149] The processor platform 1200 of the illustrated example further includes an interface circuit 1220. The interface circuit 1220 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, a near field communication (NFC) interface, a peripheral component interconnect (PCI) interface, and / or a peripheral component interconnect Express (PCIe) interface.

[0150] In the illustrated example, one or more input devices 422 are connected to the interface circuit 1220. The input device(s) 1222 allow a user to enter data and / or commands into the processor circuit 1212. The input device(s) 1222 may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, an isopoint device, and / or a voice recognition system.

[0151] One or more output devices 1224 are also connected to the interface circuit 1220 of the illustrated example. The output device(s) 1224 may be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1220 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit, such as a GPU.

[0152] The interface circuitry 1220 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any kind of computing device) via the network 1226. Communications may be made through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, an optical connection, and the like.

[0153] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 to store software and / or data. Examples of such mass storage devices 1228 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant arrays of independent disks (RAID) systems, solid-state storage devices (such as flash memory devices and / or SSDs), and DVD drives.

[0154] can be Figure 8-Figure 11The machine-readable instructions 1232 implemented by the machine-readable instructions may be stored in the mass storage device 1228, in the volatile memory 1214, in the non-volatile memory 1216, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0155] Fig.13 yes Fig.12 1212 of the processor circuit 1212. In this example, Fig.12 The processor circuit 1212 is implemented by the microprocessor 1300. For example, the microprocessor 1300 may be a general-purpose microprocessor (eg, a general-purpose microprocessor circuit). The microprocessor 1300 executes Figure 8-Figure 11 A portion or all of a flowchart of machine-readable instructions to effectively Figure 2 The circuits are instantiated as logic circuits to perform operations corresponding to these machine-readable instructions. In some such examples, Figure 2 The accelerator compiler 200 is instantiated by hardware circuits of the microprocessor 1300 in combination with instructions. For example, the microprocessor 1300 may be implemented by multi-core hardware circuits, such as a CPU, DSP, GPU, XPU, and the like. Although it may include any number of example cores 1302 (e.g., 1 core), the microprocessor 1300 of this example is a multi-core semiconductor device including N cores. The cores 1302 of the microprocessor 1300 may operate independently, or may cooperate to execute machine-readable instructions. For example, machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one of the cores 1302, or may be executed by multiple cores in the cores 1302 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is divided into threads and executed in parallel by two or more cores in the cores 1302. The software program may correspond to Figure 8-Figure 11 The flowchart may represent part or all of the machine-readable instructions and / or operations.

[0156] The core 1302 can communicate via a first example bus 1304. In some examples, the first bus 1304 can be implemented by a communication bus to implement communications associated with one (or more) of the cores 1302. For example, the first bus 1304 can be implemented by at least one of an Inter-Integrated Circuit (IC) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the first bus 1304 can be implemented by any other type of computing or electrical bus. The core 1302 can obtain data, instructions, and / or signals from one or more external devices via an example interface circuit 1306. The core 1302 can output data, instructions, and / or signals to one or more external devices via the interface circuit 1306. While the core 1302 of this example includes an example local memory 1320 (e.g., a level 1 (L1) cache that can be partitioned into an L1 data cache and an L1 instruction cache), the microprocessor 1300 also includes an example shared memory 1310 (e.g., a level 2 (L2 cache)) that can be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by writing to and / or reading from the shared memory 1310. The local memory 1320 and shared memory 1310 of each core 1302 can be a plurality of levels of cache memory and main memory (e.g., Fig.12 Cache memory is part of a hierarchy of storage devices (e.g., main memory 1214, 1216). Typically, higher level memories in the hierarchy exhibit lower access times and have smaller storage capacities than lower level memories. Changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by cache coherence policies.

[0157] Each core 1302 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit. Each core 1302 includes a control unit circuit 1314, an arithmetic and logic (AL) circuit (sometimes referred to as an ALU) 1316, a plurality of registers 1318, a local memory 1320, and a second example bus 1322. Other structures may also exist. For example, each core 1302 may include a vector unit circuit, a single instruction multiple data (SIMD) unit circuit, a load / store unit (LSU) circuit, a branch / jump unit circuit, a floating-point unit (FPU) circuit, and the like. The control unit circuit 1314 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 1302. The AL circuit 1316 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on data within the corresponding core 1302. Some examples of the AL circuit 1316 perform integer-based operations. In other examples, the AL circuit 1316 also performs floating point operations. In other examples, the AL circuit 1316 may include a first AL circuit that performs integer-based operations and a second AL circuit that performs floating point operations. In some examples, the AL circuit 1316 may be referred to as an Arithmetic Logic Unit (ALU). The register 1318 is a semiconductor-based structure used to store data and / or instructions, such as the results of one or more operations performed by the AL circuit 1316 of the corresponding core 1302. For example, the register 1318 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general registers, (one or more) flag registers, (one or more) fragment registers, (one or more) machine-specific registers, (one or more) instruction pointer registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, and the like. The register 1318 may be as follows Fig.13 The registers 1318 are shown arranged as a bank group. Alternatively, the registers 1318 may be organized in any other arrangement, format, or structure, including being distributed throughout the core 1302 to reduce access time. The second bus 1322 may be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0158] Each core 1302 and / or more generally the microprocessor 1300 may include additional and / or alternative structures to those shown and described above. For example, there may be one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHA), one or more converged / common mesh stops (CMS), one or more shifters (e.g., (one or more) barrel shifters) and / or other circuits. The microprocessor 1300 is a semiconductor device that is manufactured to include many interconnected transistors so as to implement the above structure in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit may include one or more accelerators and / or cooperate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit to perform certain tasks faster and / or more efficiently than a general-purpose processor. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. A GPU or other programmable device may also be an accelerator. The accelerator may be on the board of the processor circuit, in the same chip package as the processor circuit, and / or in one or more packages separate from the processor circuit.

[0159] Fig.14 yes Fig.12 1400. In this example, the processor circuit 1212 is implemented by an FPGA circuit 1400. For example, the FPGA circuit 1400 can be implemented by an FPGA. For example, the FPGA circuit 1400 can be used to perform, for example, Fig.13 The example microprocessor 1300 of the embodiment of the present invention executes the operations performed by the corresponding machine-readable instructions. However, once configured, the FPGA circuit 1400 instantiates the machine-readable instructions with hardware, thereby often performing the operations faster than the execution speed of a general-purpose microprocessor executing the corresponding software.

[0160] More specifically, Fig.13 The microprocessor 1300 (which is a general-purpose device that can be programmed to perform Figure 8-Figure 11 Compared to a flowchart that represents part or all of the machine-readable instructions, but whose interconnections and logic circuits are fixed once they are manufactured, Fig.14 The example FPGA circuit 1400 includes interconnects and logic circuits that can be configured and / or interconnected in different ways after manufacturing to instantiate, for example, Figure 8-Figure 111400. Specifically, FPGA circuit 1400 can be thought of as an array of logic gates, interconnects, and switches. The switches can be programmed to change the way the logic gates are interconnected by the interconnects, effectively forming one or more dedicated logic circuits (unless and until FPGA circuit 1400 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuits. These operations can correspond to Figure 8-Figure 11 Thus, FPGA circuit 1400 can be constructed to effectively Figure 8-Figure 11 A portion or all of the machine-readable instructions of the flowchart of FIG. 1400 are instantiated as a dedicated logic circuit to perform operations corresponding to these software instructions in a dedicated manner similar to an ASIC. Figure 8-Figure 11 The speed at which some or all of the machine-readable instructions correspond to operations may be faster than the speed at which a general-purpose microprocessor can execute those instructions.

[0161] exist Fig.14 In the example of , FPGA circuit 1400 is constructed to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) (eg, Verilog). Fig.14 FPGA circuit 1400 includes example input / output (I / O) circuit 1402 to obtain data from and / or output data to example configuration circuit 1404 and / or external hardware 1406. For example, configuration circuit 1404 can be implemented by an interface circuit that can obtain machine-readable instructions to configure FPGA circuit 1400, or (one or more) portions thereof. In some such examples, configuration circuit 1404 can obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit (e.g., a programmed or dedicated circuit) that can implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, external hardware 1406 can be implemented by external hardware circuits. For example, external hardware 1406 can be implemented by Fig.13 FPGA circuit 1400 also includes an array of example logic gate circuits 1408, a plurality of example configurable interconnects 1410, and example storage circuits 1412. Logic gate circuits 1408 and configurable interconnects 1410 may be configured to instantiate and Figure 8-Figure 11 At least some of the machine-readable instructions correspond to one or more operations, and / or other desired operations. Fig.14The logic gate circuits 1408 shown are manufactured in groups or blocks. Each block includes semiconductor-based electrical structures that can be configured into logic circuits. In some examples, the electrical structures include logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that provide basic building blocks for logic circuits. There are electrically controllable switches (e.g., transistors) in each logic gate circuit 1408 so that the electrical structures and / or logic gates can be configured to form circuits to perform desired operations. The logic gate circuits 1408 may include other electrical structures, such as look-up tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, and the like.

[0162] The configurable interconnect 1410 of the illustrated example is a conductive path, trace, via, or the like, which may include an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more logic gate circuits 1408 to program the desired logic circuit.

[0163] The storage circuit 1412 of the illustrated example is configured to store (one or more) results of one or more operations performed by the corresponding logic gates. The storage circuit 1412 can be implemented by registers or the like. In the illustrated example, the storage circuit 1412 is distributed between the logic gate circuits 108 to facilitate access and improve execution speed.

[0164] Fig.14 The example FPGA circuit 1400 also includes an example dedicated operation circuit 1414. In this example, the dedicated operation circuit 1414 includes a dedicated circuit 1416, which can be called to implement common functions to avoid the need to program these functions on site. Examples of such dedicated circuits 1416 include memory (e.g., DRAM) controller circuits, PCIe controller circuits, clock circuits, transceiver circuits, memories, and multiplier-accumulator circuits. Other types of dedicated circuits may also exist. In some examples, the FPGA circuit 1400 may also include an example general-purpose programmable circuit 1418, such as an example CPU 1420 and / or an example DSP 1422. Other general-purpose programmable circuits 1418 may exist in addition or alternatively, such as GPUs, XPUs, etc., which can be programmed to perform other operations.

[0165] Although Fig.13 and Fig.14 Pictured Fig.12 These are two example implementations of the processor circuit 1212, but many other approaches are also contemplated. For example, as described above, modern FPGA circuits may include an onboard CPU, such as Figure 6 One or more example CPUs 1420. Thus, Fig.12 The processor circuit 1212 may additionally be combined with Fig.13 An example microprocessor 1300 and Fig.14 In some such hybrid examples, the Figure 8-Figure 11 The first portion of the machine-readable instructions represented by the flowchart may be represented by Fig.13 One or more cores 1302 execute, Figure 8-Figure 11 The second portion of the machine-readable instructions represented by the flowchart may be represented by Fig.14 FPGA circuit 1400 is executed, and / or by Figure 8-Figure 11 The third portion of the machine-readable instructions represented by the flowchart may be executed by the ASIC. It should be understood that Figure 2 Part or all of the accelerator compiler 200 can thus be instantiated at the same or different times. Some or all of the circuits can be instantiated, for example, in one or more threads that execute simultaneously and / or serially. In addition, in some examples, Figure 2 Part or all of the accelerator compiler 200 may be implemented within one or more virtual machines and / or containers executing on a microprocessor.

[0166] In some examples, Fig.12 The processor circuit 1212 may be in one or more packages. For example, Fig.13 The microprocessor 1300 and / or Fig.14 The FPGA circuit 1400 can be in one or more packages. In some examples, the XPU can be composed of Fig.12 The XPU may be implemented in a processor circuit 1212 that may be in one or more packages. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in another package, and an FPGA in another package.

[0167] Fig.15 A block diagram is shown in FIG. 1, which illustrates an example software distribution platform 1505 for distributing software such as Fig.12 The example software distribution platform 1505 may be implemented by any computer server, data facility, cloud service, etc. that is capable of storing software and transmitting it to other computing devices. The third party may be a customer of the entity that owns and / or operates the software distribution platform 1505. For example, the entity that owns and / or operates the software distribution platform 1505 may be a software (e.g., Fig.12The third party may be a consumer, user, retailer, OEM, etc. who purchases and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 1505 includes one or more servers and one or more storage devices. The storage device stores the machine-readable instructions 1232, which may correspond to the instructions described above. Figure 8-Figure 11 Example machine-readable instructions 800, 900, 1000, and / or 1100 of the example software distribution platform 1505. One or more servers of the example software distribution platform 1505 communicate with an example network 1510, which may correspond to any one or more of the Internet and / or any of the example networks 128, 1226, 1510 described above. In some examples, as part of a commercial transaction, one or more servers respond to a request to transfer software to a requesting party. Payment for the delivery, sale, and / or license of the software may be processed by one or more servers of the software distribution platform and / or by a third-party payment entity. These servers enable purchasers and / or licensors to download machine-readable instructions 1232 from the software distribution platform 1505. For example, Figure 8-Figure 11 Software corresponding to the example machine readable instructions 800, 900, 1000 and / or 1100 may be downloaded to the example processor platform 400, which is to execute the machine readable instructions 1232 to implement Figure 2 In some examples, one or more servers of the software distribution platform 1505 periodically provide, transmit and / or force software updates (e.g., Fig.12 Example machine readable instructions 1232) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user device.

[0168] Certain examples provide an apparatus for performing a scaling transformation on a lower precision value to facilitate efficient and high accuracy performance, including means for identifying a first precision data type and a second precision data type associated with accelerated hardware execution of a machine learning model, the first precision data type having a first data precision greater than a second data precision of the second precision data type. The means for identifying may be, for example, Figure 2 The accelerator compiler 200 and / or more specifically the Figure 2 The example apparatus further includes means for determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type. The means for determining may be, for example, Figure 2 The accelerator compiler 200 and / or more specifically the Figure 2The example apparatus further includes means for converting the first weight into a second weight based on multiplying the first weight by at least one scaling factor, the second weight being based on the second precision data type. The means for converting may be, for example, Figure 2 The accelerator compiler 200 and / or more specifically the Figure 2 The example apparatus further includes means for generating an output by executing a machine learning model based on the second weight. The means for generating may be, for example, Figure 2 The accelerator compiler 200 and / or more specifically the Figure 2 The model execution circuit 250 and the executable file generation circuit 260 are implemented.

[0169] As will be appreciated from the foregoing, example systems, methods, apparatus, and articles of manufacture have been disclosed that perform scaling transformations on data representation types in lower precision formats in order to provide higher accuracy computations for machine learning (ML) and / or artificial intelligence (AI) models while improving computational efficiency and / or reducing resource consumption. The disclosed systems, methods, apparatus, and articles of manufacture improve the efficiency of using computing devices by removing concerns about accuracy loss that are an obstacle to using lower precision data representation formats (e.g., BF16), enabling large-scale reductions in computational costs, workload, and / or resource consumption (e.g., by reducing overall memory usage). The disclosed systems, methods, apparatus, and articles of manufacture are therefore directed to one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices.

[0170] Disclosed herein are example methods, apparatus, systems, and articles of manufacture for improving operational accuracy by compensating for lower precision with scaling transforms. Further examples and combinations thereof include the following:

[0171] Example 1 includes a computer-readable medium comprising instructions that, when executed, cause a machine to at least: identify a first precision data type and a second precision data type associated with execution of a machine learning model, the first precision data type having a first data precision greater than a second data precision of the second precision data type, determine at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type, convert the first weight to a second weight based on multiplying the first weight by the at least one scaling factor, the second weight being based on the second precision data type, and generate an output by executing the machine learning model based on the second weight.

[0172] Example 2 includes the computer-readable medium of Example 1, wherein the first precision data type is based on a single precision floating point format and the second precision data type is based on a single precision floating point format.

[0173] Example 3 includes the computer-readable medium of Example 1, wherein the first precision data type is based on a single-precision floating point format and the second precision data type is based on a half-precision floating point format.

[0174] Example 4 includes a computer-readable medium as described in Example 1, further including instructions that, when executed, further cause the machine to: generate a target weight for a weight in the first weights based on masking one or more least significant bits of the weight to zero, and determine the at least one scaling factor based on a ratio of the target weight to the weight in the first weights.

[0175] Example 5 includes a computer-readable medium as described in Example 1, wherein the at least one scaling factor includes a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0176] Example 6 includes a computer-readable medium as described in Example 1, wherein the at least one scaling factor includes a first scaling factor and a second scaling factor, the first scaling factor is associated with a first tensor of the machine learning model, the second scaling factor is associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

[0177] Example 7 includes a computer-readable medium as described in any of the preceding examples, wherein the instructions, when executed, cause the machine to determine the at least one scaling factor during execution of the machine learning model.

[0178] Example 8 includes the computer-readable medium of Example 1, wherein the output generated by executing the machine learning model based on the second weights includes the at least one scaling factor for descaling.

[0179] Example 9 includes the computer-readable medium of Example 8, wherein the instructions, when executed, cause the machine to perform descaling by dividing an output from execution of the machine learning model by the second weight.

[0180] Example 10 includes an apparatus for performing a scaling transformation on lower precision values ​​to facilitate efficient and high accuracy performance, the apparatus including an interface circuit for obtaining a machine learning model, and a processor circuit including one or more of the following: at least one of a central processing unit, a graphics processing unit, or a digital signal processor, the central processing unit, the graphics processing unit, or the digital signal processor having a control circuit for controlling the movement of data within the processor circuit, arithmetic and logic circuits for performing one or more first operations corresponding to instructions, and one or more registers for storing results of the one or more first operations, the instructions being in the apparatus, a field programmable gate array (FPGA), the FPGA including logic gate circuits, a plurality of configurable interconnects, and a storage circuit, the logic gate circuits and the plurality of configurable interconnects being used to perform one or more second operations, the storage circuit being used to store the one or more first operations. The results of multiple second operations, or an application-specific integrated circuit (ASIC), which includes a logic gate circuit to perform one or more third operations, the processor circuit is used to perform at least one of the first operation, the second operation or the third operation to instantiate a data precision selection circuit, which is used to identify a first precision data type and a second precision data type associated with the acceleration hardware to execute the machine learning model, the first precision data type having a first data precision greater than the second data precision of the second precision data type, a scaling factor determination circuit, which is used to determine at least one scaling factor to be applied to a first weight of the machine learning model, the first weight is based on the first precision data type, and the first weight is converted to a second weight based on the multiplication of the first weight and the at least one scaling factor, the second weight is based on the second precision data type, and a model execution circuit, which is used to generate an output by executing the machine learning model based on the second weight.

[0181] Example 11 includes the apparatus of Example 10, wherein the first precision data type is based on a single precision floating point format and the second precision data type is based on a single precision floating point format.

[0182] Example 12 includes the apparatus of Example 10, wherein the first precision data type is based on a single precision floating point format and the second precision data type is based on a half precision floating point format.

[0183] Example 13 includes an apparatus as described in Example 10, wherein the scaling factor determination circuit also generates a target weight for a weight in the first weights based on masking one or more least significant bits of the weight to zero, and determines the at least one scaling factor based on a ratio of the target weight to the weight in the first weights.

[0184] Example 14 includes an apparatus as described in Example 10, wherein the at least one scaling factor includes a first scaling factor and a second scaling factor, the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0185] Example 15 includes an apparatus as described in any of the preceding examples, wherein the at least one scaling factor includes a first scaling factor and a second scaling factor, the first scaling factor being associated with a first tensor of the machine learning model, the second scaling factor being associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

[0186] Example 16 includes an apparatus as described in Example 10, wherein the output generated by the model execution circuit by executing the machine learning model based on the second weight includes the at least one scaling factor for descaling.

[0187] Example 17 includes an apparatus as described in Example 16, wherein the model execution circuit is used to: perform descaling by dividing the output from the execution of the machine learning model by the second weight.

[0188] Example 18 includes a method for performing a scaling transformation on a lower precision value to facilitate efficient and high accuracy performance, the method comprising identifying a first precision data type and a second precision data type associated with execution of a machine learning model by accelerated hardware, the first precision data type having a first data precision greater than a second data precision of the second precision data type, determining by executing instructions using at least one processor at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type, converting the first weight to a second weight based on multiplying the first weight by the at least one scaling factor, the second weight being based on the second precision data type, and generating an output by executing the machine learning model based on the second weight.

[0189] Example 19 includes the method as described in Example 18, further comprising: generating a target weight for a weight in the first weights based on masking one or more least significant bits of the weight to zero, and determining the at least one scaling factor based on a ratio of the target weight to the weight in the first weights.

[0190] Example 20 includes a method as described in any of the preceding examples, wherein the at least one scaling factor includes a first scaling factor and a second scaling factor, the first scaling factor being associated with a first channel of a tensor of the machine learning model, the second scaling factor being associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

[0191] Example 21 includes a method as described in Example 18, wherein the output generated by executing the machine learning model based on the second weight includes the at least one scaling factor for descaling.

[0192] Example 22 includes a method as described in Example 21, wherein the descaling is performed by dividing an output from an execution of the machine learning model by the second weight.

[0193] Example 23 includes an apparatus for performing scaling transformations on lower precision values ​​to facilitate efficient and high accuracy performance, the apparatus comprising an apparatus for identifying a first precision data type and a second precision data type associated with execution of a machine learning model by accelerated hardware, the first precision data type having a first data precision greater than a second data precision of the second precision data type, an apparatus for determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type, an apparatus for converting the first weight to a second weight based on multiplying the first weight by the at least one scaling factor, the second weight being based on the second precision data type, and an apparatus for generating an output by executing the machine learning model based on the second weight.

[0194] Example 24 includes an apparatus as described in Example 23, further comprising an apparatus for generating a target weight for a weight in the first weights based on masking one or more least significant bits of the weight to zero, and an apparatus for determining the at least one scaling factor based on a ratio of the target weight to the weight in the first weights.

[0195] Example 25 includes an apparatus comprising components for performing any of the methods described in Examples 18-22.

[0196] The attached claims are hereby incorporated into this Detailed Description section by this reference. Although certain example systems, methods, devices, and articles are disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, devices, and articles that fairly fall within the scope of the claims of this patent.

Claims

1. A computer readable medium comprising instructions which, when executed, cause a machine to at least: identifying a first precision data type and a second precision data type associated with execution of a machine learning model, the first precision data type having a first data precision greater than a second data precision of the second precision data type; determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type; converting the first weight to a second weight based on a multiplication of the first weight and the at least one scaling factor, the second weight being based on the second precision data type; and An output is generated by executing the machine learning model based on the second weights.

2. The computer-readable medium of claim 1, wherein: The first precision data type is based on a single precision floating point format, and the second precision data type is based on a single precision floating point format.

3. The computer-readable medium of claim 1, wherein: The first precision data type is based on a single precision floating point format, and the second precision data type is based on a half precision floating point format.

4. The computer readable medium of claim 1, further comprising instructions that, when executed, further cause the machine to: generating a target weight for a weight among the first weights based on masking one or more least significant bits of the weight to zero; and The at least one scaling factor is determined based on a ratio of the target weight to the weight in the first weights.

5. The computer-readable medium of claim 1, wherein: The at least one scaling factor includes a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

6. The computer-readable medium of claim 1, wherein: The at least one scaling factor includes a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first tensor of the machine learning model, the second scaling factor is associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

7. A computer readable medium as claimed in any preceding claim, wherein: The instructions, when executed, cause the machine to determine the at least one scaling factor during execution of the machine learning model.

8. The computer-readable medium of claim 1, wherein: The output generated by executing the machine learning model based on the second weights includes the at least one scaling factor for descaling.

9. The computer-readable medium of claim 8, wherein: The instructions, when executed, cause the machine to perform descaling by dividing the output from execution of the machine learning model by the second weight.

10. An apparatus for performing a scaling transformation on a lower precision value to facilitate efficient and high accuracy performance, comprising: An interface circuit for obtaining a machine learning model; as well as A processor circuit comprising one or more of the following: at least one of a central processor unit, a graphics processor unit, or a digital signal processor, the at least one of the central processor unit, the graphics processor unit, or the digital signal processor having a control circuit, an arithmetic and logic circuit, and one or more registers, the control circuit being used to control the movement of data within the processor circuit, the arithmetic and logic circuit being used to perform one or more first operations corresponding to an instruction, the one or more registers being used to store results of the one or more first operations, the instruction being in the device; a field programmable gate array (FPGA), the FPGA comprising a logic gate circuit, a plurality of configurable interconnects, and a storage circuit, the logic gate circuit and the plurality of configurable interconnects being used to perform one or more second operations, the storage circuit being used to store results of the one or more second operations; or an application specific integrated circuit (ASIC) including logic gate circuits to perform one or more third operations; The processor circuit is configured to perform at least one of the first operation, the second operation, or the third operation to instantiate: a data precision selection circuit to identify a first precision data type and a second precision data type associated with execution of the machine learning model by acceleration hardware, the first precision data type having a first data precision greater than a second data precision of the second precision data type; Scaling factor determination circuitry to: determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type; and converting the first weight to a second weight based on a multiplication of the first weight and the at least one scaling factor, the second weight being based on the second precision data type; as well as Model execution circuitry for generating an output by executing the machine learning model based on the second weights.

11. The device of claim 10, wherein: The first precision data type is based on a single precision floating point format, and the second precision data type is based on a single precision floating point format.

12. The device of claim 10, wherein: The first precision data type is based on a single precision floating point format, and the second precision data type is based on a half precision floating point format.

13. The device of claim 10, wherein: The scaling factor determination circuit is further configured to: generating a target weight for a weight among the first weights based on masking one or more least significant bits of the weight to zero; and The at least one scaling factor is determined based on a ratio of the target weight to the weight in the first weights.

14. The device of claim 10, wherein: The at least one scaling factor includes a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

15. A device as claimed in any preceding claim, wherein The at least one scaling factor includes a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first tensor of the machine learning model, the second scaling factor is associated with a second tensor of the machine learning model, and the first scaling factor is different from the second scaling factor.

16. The device of claim 10, wherein: The output generated by the model execution circuit by executing the machine learning model based on the second weights includes the at least one scaling factor for descaling.

17. The device of claim 16, wherein: The model execution circuit is used to perform descaling by dividing the output from the execution of the machine learning model by the second weight.

18. A method of performing a scaling transformation on a lower precision value to facilitate efficient and high accuracy performance, comprising: Identifying a first precision data type and a second precision data type associated with execution of a machine learning model by acceleration hardware, the first precision data type having a first data precision greater than a second data precision of the second precision data type; determining, by executing instructions with at least one processor, at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type; converting the first weight to a second weight based on a multiplication of the first weight and the at least one scaling factor, the second weight being based on the second precision data type; and An output is generated by executing the machine learning model based on the second weights.

19. The method of claim 18, further comprising: generating a target weight for a weight among the first weights based on masking one or more least significant bits of the weight to zero; and The at least one scaling factor is determined based on a ratio of the target weight to the weight in the first weights.

20. A method as claimed in any preceding claim, wherein: The at least one scaling factor includes a first scaling factor and a second scaling factor, wherein the first scaling factor is associated with a first channel of a tensor of the machine learning model, the second scaling factor is associated with a second channel of the tensor, and the first scaling factor is different from the second scaling factor.

21. The method of claim 18, wherein: An output generated by executing the machine learning model based on the second weights includes the at least one scaling factor for descaling.

22. The method of claim 21, wherein: Descaling is performed by dividing the output from execution of the machine learning model by the second weight.

23. An apparatus for performing a scaling transform on a lower precision value to facilitate efficient and high accuracy performance, the apparatus comprising: means for identifying a first precision data type and a second precision data type associated with execution of a machine learning model by acceleration hardware, the first precision data type having a first data precision greater than a second data precision of the second precision data type; means for determining at least one scaling factor to be applied to a first weight of the machine learning model, the first weight being based on the first precision data type; means for converting the first weight into a second weight based on a multiplication of the first weight and the at least one scaling factor, the second weight being based on the second precision data type; as well as Means for generating an output by executing the machine learning model based on the second weights.

24. The apparatus of claim 23, further comprising: means for generating a target weight for a weight among the first weights based on masking one or more least significant bits of the weight to zero, and Means for determining the at least one scaling factor based on a ratio of the target weight to the weight in the first weights.

25. An apparatus comprising means for performing any of the methods of claims 18-22.

Images