Artificial neural network training using flexible floating point tensors

Abstract

The invention discloses artificial neural network training using flexible floating point tensors. Thus, the present disclosure relates to systems and methods for training a neural network using a tensor that includes a plurality of FP16 values and a plurality of bits that define an index shared by some or all of the FP16 values included in the tensor. The FP16 value may include a 16-bit floating point value in the IEEE 754 format, and the tensor may include a plurality of bits defining the shared index. The tensor may include a shared index and an FP16 value that includes a variable bit length mantissa and a variable bit length index that may be dynamically set by processor circuitry. The tensor may include a shared index and an FP16 value including a variable bit length mantissa that may be dynamically set by processor circuitry, a variable bit length index, and a shared index switch that is set by the processor circuitry to selectively combine an index of the FP16 value with the shared index.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to training of neural networks. BACKGROUND

[0002] In its simplest form, a neural network includes an input layer having a plurality of inputs, an output layer containing one or more outputs, and at least one hidden layer containing a plurality of neurons coupled to the input and output layers. Each of the neurons included in the hidden layer can have any number of inputs and any number of outputs. Each of the neuron inputs and / or neuron outputs is weighted to affect the decision-making ability of the neural network. These weights are typically initially assigned and then updated based on information contained in a training data set provided during a training period for the neural network.

[0003] In supervised training, both input data (i.e., data at the input layer) and output data (i.e., data at the output layer) are provided to the neural network. The network processes the input data and compares the result to the output data. Errors can be backpropagated through the network to adjust the weights that control the network, thereby improving network accuracy and / or speed. The training process occurs over a number of epochs. Each epoch includes one forward pass of data through the neural network and one backward pass of data through the neural network. Training can require multiple epochs to generate reliable and predictable outputs within an acceptable threshold.

[0004] Deep learning (also known as deep structured learning or hierarchical learning) is part of a broader family of machine learning methods based on learning a data representation, in contrast to task-specific algorithms. Deep learning models loosely relate to the patterns of information processing and communication in biological nervous systems such as neural coding, which seeks to define the relationship between various stimuli and associated neural responses in the brain. Deep learning is computationally intensive, thus requiring significant processing power and resources.

[0005] IEEE numeric formats include 16-bit floating point [FP16] and 32-bit floating point [FP32]. The IEEE 754 standard for FP16 includes a single sign bit, 10 bits of fraction, and 5 bits of exponent. The IEEE 754 standard for FP32 includes a single sign bit, 23 bits of fraction, and 8 bits of exponent. The use of FP32 in neural network training significantly increases the computational load of the system, especially when compared to FP16. However, in FP16, the gradients in deep neural network training can become too shallow (i.e., drop to 2 -24 below) and become "0". Thus, despite the high computational overhead and time cost, much deep learning is done in FP32. BRIEF DESCRIPTION OF DRAWINGS

[0006] The features and advantages of the claimed subject matter will become more apparent from the detailed description in the following drawings in which like numerals refer to like parts throughout the several figures which schematically depict:

[0007] Figure 1 A high-level block diagram of an illustrative system including a neural network, processor circuitry to provide a training tensor to the neural network, and a storage device to store instructions executed by the processor circuitry is provided in accordance with at least one embodiment described herein;

[0008] Figure 2 is an illustrative tensor including a plurality of 16-bit floating-point numbers sharing a common 6-bit exponent (5:0), and wherein each of the 16-bit floating-point numbers in the tensor includes a single sign bit (15:15), an 8-bit exponent (14:7), and a 7-bit mantissa (6:0) in accordance with at least one embodiment described herein;

[0009] Figure 3 is another illustrative tensor including a plurality of 16-bit floating-point numbers sharing a common 6-bit exponent (5:0), and wherein each of the 16-bit floating-point numbers in the tensor includes a single sign bit (15:15), a single exponent on / off switch bit (14:14), an 8-bit exponent (13:6), and a 6-bit mantissa (5:0) in accordance with at least one embodiment described herein;

[0010] Figure 4 is a schematic diagram of an illustrative electronic processor-based device including processor circuitry to train a neural network coupled to the processor-based device using a training tensor including a plurality of 16-bit floating-point values as described in Figure 1 、 Figure 2 and Figure 3 in accordance with at least one embodiment described herein;

[0011] Figure 5 is a high-level flow diagram of an illustrative method to train a neural network using a training tensor including a plurality of 16-bit floating-point values having a common shared exponent in accordance with at least one embodiment described herein;

[0012] Figure 6 is a high-level flow diagram of an illustrative method to generate 16-bit floating-point values having variable bit-length mantissas, variable bit-length exponents, and a shared exponent in accordance with at least one embodiment described herein;

[0013] Figure 7is a high level flowchart of an illustrative method of generating a 16-bit floating point value having a variable bit length mantissa, a variable bit length exponent, and a shared exponent in response to detecting data indicative of a potential overflow or underflow condition in accordance with at least one embodiment described herein; and

[0014] Figure 8 is a high level flowchart of an illustrative method of selectively combining an exponent portion of a variable bit length exponent included in a 16-bit floating point value with a shared exponent value based on a logical state of a shared exponent switch bit included in the 16-bit floating point value in accordance with at least one embodiment described herein.

[0015] While the following detailed description will reference illustrative embodiments, numerous alternatives, modifications, and variations thereof will be apparent to those skilled in the art in view of the foregoing detailed description. DETAILED DESCRIPTION

[0016] The systems and methods described herein provide systems and methods for enhancing the speed and accuracy of deep learning using neural networks trained on tensors comprising 16-bit floating point (FP16) numbers that share a common exponent. One potential variable includes neural network training tensors in which each floating point number is represented in a non-IEEE standard format that includes a variable length fraction portion and a variable length exponent portion. Another potential variable includes neural network training tensors in which each floating point number includes a single on (ON) / off (OFF) bit that is used to: (a) combine the exponent of the corresponding floating point number with a shared exponent; or (b) not combine the corresponding floating point exponent with a shared exponent (i.e., use only the floating point exponent). Thus, the systems and methods described herein provide the advantages of FP16 computation while beneficially increasing the range of possible test values by using a five bit shared exponent.

[0017] Some systems and methods described herein generate a neural network training tensor that includes a 6-bit shared exponent that is shared by each of a plurality of numbers that conform to the IEEE 754 FP16 format, each of the plurality of numbers including a single sign bit, a five-bit exponent, and a 10-bit mantissa. Some systems and methods described herein generate a neural network training tensor that includes an exponent that is shared by each of a plurality of numbers that do not conform to the IEEE 754 FP16 format, each of the plurality of numbers including a single sign bit, an 8-bit exponent, and a 7-bit mantissa. Some systems and methods described herein generate a neural network training tensor that includes a plurality of FP16 numbers and a 6-bit shared exponent that is shared by all of the FP16 numbers included in the tensor. Each of the FP16 numbers can have a single sign bit, a single shared exponent on / off switch, an 8-bit exponent, and a 6-bit mantissa. Some systems and methods described herein allow processor circuitry to adjust the value of the shared exponent to avoid potential overflow or underflow conditions. Some systems and methods described herein allow processor circuitry to additionally or alternatively selectively adjust the value of the exponent and the mantissa of some or all of the FP16 included in the tensor to avoid potential overflow or underflow conditions. Processor circuitry can therefore beneficially and advantageously adjust some or all of the following while training a neural network to avoid potential overflow or underflow conditions: the value of the shared exponent, the value of the exponent in some or all of the FP16 numbers in the tensor, and / or the value of the mantissa in some or all of the FP16 numbers in the tensor.

[0018] A system for training a neural network is provided. The system can include processor circuitry; and a storage device coupled to the processor circuitry, the storage device including machine-readable instructions. The machine-readable instructions cause the processor circuitry to generate a neural network training tensor including: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a 6-bit shared exponent that is common to and shared by each of the 16-bit floating point values included in the training tensor.

[0019] A neural network training method is provided. The neural network training method can include generating, by processor circuitry, a neural network training tensor including: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a 6-bit shared exponent that is common to each of the 16-bit floating point values included in the training tensor; and providing at least one of the plurality of 16-bit floating point values included in the training tensor as an input to the neural network.

[0020] A neural network training system is provided. The training system can include means for generating, by processor circuitry, a neural network training tensor comprising: a plurality of 16-bit floating point values, each of the 16-bit floating point values comprising: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and means for providing at least one of the plurality of 16-bit floating point values included in the training tensor as an input to the neural network.

[0021] A non-transitory machine-readable storage medium is provided. The non-transitory machine-readable storage medium comprises machine-readable instructions that, when executed by processor circuitry, cause the processor circuitry to: generate a neural network training tensor comprising: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values comprising: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

[0022] A processor-based device for training a neural network is provided. The processor-based device can include: a printed circuit board; processor circuitry coupled to the printed circuit board; an input / output interface couplable to a neural network; and a storage device coupled to the processor circuitry, the storage device comprising machine-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to: generate a neural network training tensor comprising: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values comprising: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a 6-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

[0023] Figure 1 A high-level block diagram of an illustrative system 100 according to at least one embodiment described herein is provided, including a neural network 110, processor circuitry 130 for providing a training tensor 150 to the neural network 110, and a storage device 170 for storing instructions executed by the processor circuitry 130. The training tensor 150 includes: a plurality of 16-bit floating point values 1521to 152 n(collectively, "16-bit floating-point values 152"), and an exponent 154 that is common to and shared by each of at least some of the 16-bit floating-point values 152. In embodiments, the processor circuitry 130 generates or causes generation of a training tensor 150 for training (e.g., setting weights and / or biases of) the neural network 110. In embodiments, execution of instructions stored in, on, or about the storage device 170 can cause the processor circuitry 130 to forward propagate and / or backward propagate the training tensor 150 through the neural network 110.

[0024] The neural network 110 includes a plurality of inputs 1121to 112 n (collectively, "neurons 116") coupled by a hidden layer 116 that includes a plurality of neurons 1161to 116 n (collectively, "inputs 112") and a plurality of outputs 1141to 114 n (collectively, "outputs 114"). While the hidden layer 116 includes only a single layer of neurons, one of ordinary skill in the related art will readily understand that the hidden layer 116 can include any number of layers. In embodiments, the neural network 110 can include a neural network capable of deep learning - a deep neural network.

[0025] The processor circuitry 130 can include any number of currently available and / or future developed electrical components, semiconductor devices, and / or logic elements, and / or combinations of these, capable of providing the 16-bit floating-point values 152 and the 6-bit shared exponent 154 included in the training tensor 150 to the neural network 110. In some implementations, the processor circuitry 130 can dynamically adjust the exponent 154 based on one or more parameters associated with the neural network 110 and / or the training tensor 150. For example, the processor circuitry 130 can execute instructions that cause the shared exponent 154 to increase in response to detecting potential overflow of one or more of the 16-bit floating-point values 152 included in the training tensor 150. In another example, the processor circuitry 130 can execute instructions that cause the shared exponent 154 to decrease in response to detecting potential underflow of one or more of the 16-bit floating-point values 152 included in the training tensor 150.

[0026] In embodiments, the processor circuitry 130 can include circuitry to compute or otherwise determine a rate of increase or decrease of one or more 16-bit floating point values 152 included in the training tensor 150 to identify those 16-bit floating point values 152 that exhibit behavior indicative of a potential underflow or overflow condition. In other embodiments, the processor circuitry 130 can include circuitry to compute or otherwise determine a delta increase or decrease of one or more 16-bit floating point values 152 included in the training tensor 150 to identify those 16-bit floating point values 152 that exhibit behavior indicative of a potential underflow or overflow condition.

[0027] In embodiments, the processor circuitry 130 can cause some or all of the 16-bit floating point values 152 included in the training tensor 150 to be propagated forward through the neural network 110. In embodiments, the processor circuitry 130 can cause some or all of the 16-bit floating point values 152 included in the training tensor 150 to be propagated backward or in reverse through the neural network 110. In embodiments, the processor circuitry 130 can include one or more general purpose systems and / or devices configured to generate the 16-bit floating point values 152 included in the training tensor 150 and / or provide the 16-bit floating point values 152 included in the training tensor 150 to the neural network 110. In embodiments, the processor circuitry 130 can receive at least a portion of the 16-bit floating point values 152 included in the training tensor 150 from one or more external sources, such as a networked source.

[0028] The training tensor 150 includes a plurality of 16-bit floating point values 152. The training tensor 150 also includes a shared exponent 154 that can be shared by each of at least a portion of the 16-bit floating point values 152. The shared exponent 154 can include a sign bit to provide a signed shared exponent 154. The shared exponent 154 can include a 3-bit value; a 4-bit value; a 5-bit value; a 6-bit value; a 7-bit value; or an 8-bit value. For example, using the IEEE 754 standard 16-bit floating point format, each 16-bit floating point value 152 included in the training tensor 150 will include: a single sign bit; a 10-bit mantissa; and a mantissa that includes the sum of a 5-bit exponent included in the 16-bit floating point value 152 plus the 5-bit shared exponent 154. In embodiments, the training tensor 150 includes a plurality of IEEE 754 compliant half-precision (16-bit floating point) values 152. In embodiments, the training tensor 150 includes a plurality of IEEE 754 non-compliant half-precision (16-bit floating point) values 152.

[0029] The storage device 170 can include any number of currently available and / or future developed data storage systems and / or devices, and / or combinations of the same. In embodiments, the storage device 170 can store a set of machine-readable instructions executable by the processor circuitry 130. In embodiments, the storage device 170 can maintain, store, or otherwise retain at least some of the 16-bit floating point values 152 forming the training tensor 150 in one or more data storage devices, data structures, and / or databases.

[0030] Figure 2 is an illustrative tensor 200 including a plurality of 16-bit floating point (FP16) numbers 1521to 152 n according to at least one embodiment described herein, where each floating point number shares a common 6-bit exponent (5:0) 154, and where each of the FP16 numbers includes: a corresponding single sign bit (15:15) 2041to 204 n ; a corresponding 8-bit exponent (14:7) 2061to 206 n ; and a corresponding 7-bit mantissa (6:0) 2081to 208 n The ability to dynamically alter at least one of the value of the shared exponent 154 included in the training tensor 200, the mantissa value of one or more FP16 numbers, and the exponent value of one or more FP16 numbers beneficially allows the processor circuitry 130 to proactively adjust one or more FP16 values in the tensor 200 to avoid potential overflow or underflow conditions during the neural network training process. Using the format depicted, each 16-bit floating point value 152 can be represented as: Figure 2

[0031]

[0032] where the exponent bias is given by:

[0033] (2) (E-M) -1(2)

[0034] Figure 3 is an illustrative tensor 300 including a plurality of 16-bit floating point (FP16) numbers 1521to 152 n according to at least one embodiment described herein, where each floating point number shares a common 6-bit exponent (5:0) 154, and where each of the FP16 numbers includes: a corresponding single sign bit (15:15) 2041to 204 n ; a corresponding shared exponent on / off switch bit (14:14) 3021to 302 n ; a corresponding 8-bit exponent (14:7) 2061to 206 n ; and a corresponding 6-bit mantissa (5:0) 2081to 208​n The ability to dynamically alter at least one of the following: the value of the shared exponent 154 included in the training tensor 200, the mantissa value of the one or more FP16 numbers, and the exponent value of the one or more FP16 numbers, beneficially allows the processor circuitry 130 to proactively adjust one or more FP16 values in the tensor 300 to avoid potential overflow or underflow conditions during the neural network training process. Additionally, the processor circuitry 130 can selectively designate whether the exponent 206 of each of the FP16 numbers 152 included in the training tensor 300 should be combined with the shared exponent combination - further increasing the flexibility of the training tensor 300. Using the format depicted in Figure 3 With the format depicted in

[0035]

[0036] where the exponent bias is given by

[0037] (2) (E-M) -1 (4)

[0038] Using the format depicted in Figure 3 With the format depicted in

[0039]

[0040] where the exponent bias is given by

[0041] (2) (E-M) -1 (6)

[0042] Figure 4 is a schematic diagram of an illustrative electronic, processor-based device 400 described herein that includes processor circuitry 130 for using, for example, the format depicted in Figure 1 , Figure 2 and Figure 3The training tensors 150 described in the middle that include the plurality of 16-bit floating-point values 152 are used to train the neural network 110 coupled to the processor-based device 400. The processor-based device 400 can additionally include one or more of the following: a graphics processing unit 412, a wireless input / output (I / O) interface 420, a wired I / O interface 430, a memory circuitry 440, a power management circuitry 450, a non-transitory storage device 170, and a network interface 470. The following discussion provides a brief, general description of the components of the illustrative processor-based device 400. An example non-limiting processor-based device 400 can include, but is not limited to, a smartphone, a wearable computer, a portable computing device, a handheld computing device, a desktop computing device, a blade server device, a workstation, etc.

[0043] The processor-based device 400 includes the processor circuitry 130 capable of creating or otherwise generating the 16-bit floating-point values 152 included in the tensors 150 used to train the neural network 110. In an embodiment, the processor-based device 400 can additionally include the graphics processor circuitry 412. In an embodiment, the processor-based device 400 includes the processor circuitry 130 capable of executing one or more machine-readable instruction sets 414, reading data and / or instruction sets 414 from one or more storage devices 170, and writing data to the one or more storage devices 170. The processor circuitry 130 can forward propagate some or all of the 16-bit floating-point values 152 included in the training tensors 150 through the neural network 110. The processor circuitry 130 can backward propagate some or all of the 16-bit floating-point values 152 included in the training tensors 150 through the neural network 110. In an embodiment, the processor circuitry 130 can include prediction and / or detection circuitry for predicting and / or detecting information and / or data indicative of potential overflow or underflow conditions in one or more of the floating-point values included in the 16-bit floating-point values 152 included in the training tensors 150.

[0044] In some embodiments, the processor-based device 400 includes a graphics processor circuitry 412 capable of executing a machine-readable instruction set 414 and generating output signals capable of providing display output to a system user. Those skilled in the relevant art will appreciate that the illustrated embodiments, as well as other embodiments, can be practiced using other processor-based device configurations, including portable electronic or hand-held electronic devices, such as smartphones, portable computers, wearable computers, consumer electronic devices, personal computers (“PCs”), network PCs, minicomputers, mainframe computers, etc. The processor circuitry 130 can include any number of hardwired or configurable circuits, some or all of which can include programmable and / or configurable combinations of electronic components, semiconductor devices, and / or logic elements, arranged in part or in whole within a PC, server, or other computing system capable of executing processor-readable instructions.

[0045] The processor-based device 400 includes a bus or similar communication link 416 communicably coupling and facilitating the exchange of information and / or data between the various system components, including the processor circuitry 130, the graphics processor circuitry 412, one or more wireless I / O interfaces 420, one or more wired I / O interfaces 430, one or more storage devices 170, and / or one or more network interfaces 470. The processor-based device 400 can be referred to herein in the singular, but this is not intended to limit the embodiments to a single processor-based device 400, as in certain embodiments there can be more than one processor-based device 400 incorporated, included, or comprised of any number of communicably coupled, collocated, or remotely networked circuits or devices.

[0046] The processor circuitry 130 can include any number, type, or combination of currently available or future developed devices capable of executing machine-readable instruction sets. The processor circuitry 130 can include, but is not limited to, any currently or future developed single-core or multi-core processor or microprocessor, such as one or more system-on-a-chip (SOC); central processing units (CPUs); digital signal processors (DSPs); graphics processing units (GPUs); application-specific integrated circuits (ASICs), programmable logic units, field-programmable gate arrays (FPGAs), etc. Unless otherwise specified, Figure 4 The construction and the operation of the various blocks shown in the FIGURE 1 are of conventional design. Accordingly, further details of these blocks are not further detailed herein, as they will be understood by those skilled in the relevant art. The bus 416, which interconnects at least some of the components of the processor-based device 400, can employ any currently available or future developed serial or parallel bus structure or architecture, in series or in parallel.

[0047] System memory 440 can include read-only memory ("ROM") 442 and random access memory ("RAM") 446. A portion of ROM 442 can be used to store or otherwise retain a basic input / output system ("BIOS") 444. BIOS 444 provides the basic functionality to processor-based device 400, for example, by causing processor circuitry 130 to load and / or execute one or more machine-readable instruction sets 414. In embodiments, at least some of the one or more machine-readable instruction sets 414 cause at least a portion of processor circuitry 130 to provide, create, generate, transform, and / or act as a special, specific, and particular machine, such as a word processing machine, a digital image acquisition machine, a media playback machine, a gaming system, a communication device, a smartphone, etc.

[0048] Processor-based device 400 can include at least one wireless input / output (I / O) interface 420. The at least one wireless I / O interface 420 can be communicably coupled to one or more physical output devices 422 (haptic devices, video displays, audio output devices, hard copy output devices, etc.). The at least one wireless I / O interface 420 can be communicably coupled to one or more physical input devices 424 (pointing devices, touch screens, keyboards, haptic devices, etc.). The at least one wireless I / O interface 420 can include any currently available or future developed wireless I / O interface. Example wireless I / O interfaces include, but are not limited to: Near Field Communication (NFC), etc.

[0049] Processor-based device 400 can include one or more wired input / output (I / O) interfaces 430. The at least one wired I / O interface 430 can be communicably coupled to one or more physical output devices 422 (haptic devices, video displays, audio output devices, hard copy output devices, etc.). The at least one wired I / O interface 430 can be communicably coupled to one or more physical input devices 424 (pointing devices, touch screens, keyboards, haptic devices, etc.). Wired I / O interface 430 can include any currently available or future developed I / O interface. Example wired I / O interfaces include, but are not limited to: Universal Serial Bus (USB), IEEE 1394 ("Firewire"), etc.

[0050] The processor-based device 400 can include one or more communicably coupled non-transitory data storage devices 170. The data storage devices 170 can include one or more hard disk drives (HDDs) and / or one or more solid state storage devices (SSDs). The one or more data storage devices 170 can include any currently or future developed storage, network storage device, and / or system. Non-limiting examples of such data storage devices 170 can include, but are not limited to, any currently or future developed non-transitory storage or device, such as one or more magnetic storage devices, one or more optical storage devices, one or more resistive storage devices, one or more molecular storage devices, one or more quantum storage devices, or various combinations thereof. In some implementations, the one or more data storage devices 170 can include one or more removable storage devices, such as one or more flash drives, flash memory, flash memory units, or similar devices or equipment capable of being communicably coupled to and decoupled from the processor-based device 400.

[0051] The one or more data storage devices 170 can include an interface or controller (not shown) communicably coupling the corresponding storage device or system to the bus 416. The one or more data storage devices 170 can store, retain, or otherwise contain machine-readable instruction sets, data structures, program modules, data stores, databases, logical structures, and / or other data beneficial to the processor circuitry 130 and / or graphics processor circuitry 412, and / or one or more applications executing on or by the processor circuitry 130 and / or graphics processor circuitry 412. In some instances, the one or more data storage devices 170 can be communicably coupled to the processor circuitry 130, for example, via the bus 416 or one or more wired communication interfaces 430 (e.g., Universal Serial Bus or USB); one or more wireless communication interfaces 420 (e.g., Bluetooth®, Bluetooth Low Energy, ZigBee®, Wi-Fi®, near field communication or NFC); and / or one or more network interfaces 470 (IEEE 802.3 or Ethernet, IEEE 802.11, or Wi-Fi®).

[0052] ​​The processor-readable instruction set 414 and other programs, applications, logic, and / or modules may be stored in whole or in part in the system memory 440. Such instruction set 414 may be transferred in whole or in part from one or more data storage devices 170. The instruction set 414 may be loaded, stored, or otherwise retained in whole or in part in the system memory 440 during execution by the processor circuitry 130 and / or graphics processor circuitry 412. The processor-readable instruction set 414 may include machine-readable and / or processor-readable code, instructions, or similar logic that enables the processor circuitry 130 to dynamically adjust one or more of the following: the value of the shared exponent 154 included in the training tensor 150, the mantissa value of one or more FP16 numbers 208, and the exponent value of one or more FP16 numbers 206. The processor-readable instruction set 414 may include code, instructions, or similar logic that enables the processor circuitry 130 to selectively and independently adjust each FP16 number 1521 to 152 n Shared index on / off switch 3021 to 302 n Machine-readable and / or processor-readable code, instructions, or similar logic configured to be set to a first logic state ("on") in which the shared index 154 is selectively associated with the corresponding FP16 numbers 1521 to 1522. n Index 2061 to 206 n The processor readable instruction set 414 may include instructions that enable the processor circuitry 130 to selectively and independently convert each FP16 number 1521 to 152 n Shared index on / off switch 3021 to 302 n Machine-readable and / or processor-readable code, instructions, or similar logic configured to set the shared index 154 to a second logic state ("off") in which the shared index 154 is not associated with the corresponding FP16 numbers 1521 to 1522. n Index 2061 to 206 n Combined.

[0053] The processor-based device 400 can include power management circuitry 450 that controls one or more operational aspects of the energy storage device 452. In embodiments, the energy storage device 452 can include one or more primary (i.e., non-rechargeable) batteries or secondary (i.e., rechargeable) batteries or similar energy storage devices. In embodiments, the energy storage device 452 can include one or more supercapacitors or ultracapacitors. In embodiments, the power management circuitry 450 can alter, adjust, or control the flow of energy from an external power source 454 to the energy storage device 452 and / or to the processor-based device 400. The power source 454 can include, but is not limited to, a solar power system, a commercial power grid, a portable generator, an external energy storage device, or any combination thereof.

[0054] For convenience, the processor circuitry 130, the graphics processor circuitry 412, the wireless I / O interface 420, the wired I / O interface 430, the power management circuitry 450, the storage device 460, and the network interface 470 are shown communicatively coupled to one another via a bus 416, providing connectivity between the aforementioned components. In alternative embodiments, the aforementioned components can be communicatively coupled in a different manner than that shown in FIG. 4. For example, one or more of the aforementioned components can be directly coupled to the other components, or can be coupled to one another via one or more intervening components (not shown). In another example, one or more of the aforementioned components can be integrated into the processor circuitry 130 and / or the graphics processor circuitry 412. In some embodiments, all or a portion of the bus 416 can be omitted, and the components can be directly coupled to one another using appropriate connections. Figure 4

[0055] Figure 5 is a high-level flowchart of an illustrative method 500 of training a neural network 110 using a training tensor 150 that includes a plurality of 16-bit floating-point values 152 having a shared exponent 154 in accordance with at least one embodiment described herein. The shared exponent 154 beneficially increases the range of possible values in the training tensor 150, thereby reducing the likelihood of overflow or underflow conditions occurring in the training tensor 150. The method 500 begins at 502.

[0056] At 504, the processor circuitry 130 generates a training tensor 150 that includes a plurality of 16-bit floating-point values 152. The exponent of each of at least a portion of the 16-bit floating-point values 152 can be combined with a shared exponent 154 associated with the training tensor 150 to provide the 16-bit floating-point values 152 with an extended range that exceeds that of IEEE 754-compliant 16-bit floating-point values.

[0057] ​At 506, the processor circuitry 130 provides at least a portion of the training tensor 150 to the neural network 110. In embodiments, the processor circuitry 130 causes the 16-bit floating point values 152 to be forward propagated to the neural network 110. In embodiments, the neural network receives an output from the neural network 110 and causes at least a portion of the received 16-bit floating point values 152 to be backward propagated through the neural network. The method 500 ends at 508.

[0058] Figure 6 is a high-level flowchart of an illustrative method 600 of selectively adjusting values of one or more of the shared exponent 154, one or more FP16 mantissa values 2081to 208 n , and one or more FP16 exponent values 2061to 206 n in response to detecting data indicative of a potential overflow or underflow condition in accordance with at least one embodiment described herein. The method 600 begins at 602.

[0059] At 604, the processor circuitry 130 detects data indicative of a potential overflow or underflow condition in one or more of the 16-bit floating point values 152 included in the training tensor 150. In embodiments, the processor circuitry 130 can include prediction circuitry and / or comparator circuitry capable of predicting the 16-bit floating point values 152 during future training epochs.

[0060] At 606, in response to determining that one or more of the 16-bit floating point values 152 will overflow or underflow during a future training epoch, the processor circuitry 130 dynamically alters values of one or more of the shared exponent 154, one or more FP16 mantissa values 2081to 208 n , and one or more FP16 exponent values 2061to 206 n to minimize or eliminate the likelihood of the overflow or underflow condition occurring. The method 600 ends at 608.

[0061] Figure 7 is a high-level flowchart of an illustrative method 700 of selectively combining an exponent portion of a variable-bit length exponent included in the 16-bit floating point values 152 with a shared exponent value based on a logical state of a shared exponent on / off switch bit 3021to 302 n included in each of the FP16 numbers 1521to 152 n in the training tensor 300 in accordance with at least one embodiment described herein. In embodiments, each of the 16-bit floating point values 1521to 152 n may include a corresponding sign bit 2041to 204 n , a corresponding shared exponent on / off switch bit 3021to 302 n, corresponding to 6-bit mantissas 2081 through 208 n , and corresponding to 8-bit exponents 2081 through 208 n . The method 700 advantageously allows each of the FP16 numbers 1521 through 152 n included in the training tensor 150 to use a plurality of different exponent values. The method 700 begins at 702.

[0062] At 704, the processor circuitry 130 selectively and independently sets a shared exponent switch bit 3021 through 302 n for each of the FP16 numbers 1521 through 152 n included in the training tensor 150. In embodiments, the processor circuitry 130 can selectively set the shared exponent switch bit 304 based on determining that the exponent values 2061 through 206 n of the FP16 numbers 1521 through 152 n included in the training tensor 150 are insufficient to prevent future overflow or underflow of the corresponding FP16 numbers 1521 through 152 n . By combining the exponent bits included in the 16-bit floating point values 152 with the shared exponent associated with the training tensor 150, the processor circuitry 130 minimizes or eliminates the likelihood of an overflow or underflow condition occurring. The method 700 ends at 706.

[0063] Figure 8 is a high-level flowchart of an illustrative method 800 of selectively adjusting the value of one or more of the shared exponent 154, one or more FP16 mantissa values 2081 through 208 n , and one or more FP16 exponent values 2061 through 206 n and / or selectively controlling whether to combine each exponent value 2061 through 206 n of each FP16 number 1521 through 152 n with the shared exponent 154 in response to detecting data indicative of a potential overflow or underflow condition in accordance with at least one embodiment described herein. The method 800 begins at 802.

[0064] At 804, the processor circuitry 130 detects data indicative of a potential overflow or underflow condition in one or more of the 16-bit floating point values 1521 through 152 n included in the training tensor 150. In embodiments, the processor circuitry 130 can include prediction circuitry and / or comparator circuitry capable of predicting the 16-bit floating point values 152 during future training epochs.

[0065] At 806, in response to determining that one or more of the 16-bit floating-point values 152 will overflow or underflow during a future training epoch, the processor circuitry 130 dynamically alters the value of one or more of the shared exponent 154, the one or more FP16 mantissa values 2081-208 n , the one or more FP16 exponent values 2061-206 n , and / or the shared exponent on / off switches 3021-302 n to a desired logical state to minimize or eliminate the likelihood of the overflow or underflow condition occurring. The method 800 ends at 808.

[0066] While Figures 5 to 8 various operations according to one or more embodiments are illustrated, it will be understood that, for other embodiments, not all of the operations depicted in the Figures 5 to 8 are necessarily required. In fact, one having ordinary skill in the art will recognize that, in other embodiments, the operations depicted in the Figures 5 to 8 and / or other operations described herein can be combined in ways not specifically shown in the drawings, but still fall within the scope of the present disclosure. Thus, claims that are directed to features and / or operations that are not exactly shown in one drawing are deemed to be within the scope and content of the present disclosure.

[0067] As used in this application and in the claims, the term “and / or” means any one or any combination of the listed terms. For example, the phrase “A and / or B” means A; B; or A and B. As used in this application and in the claims, the term “at least one of’ a list of items means any one of the items in the list. For example, the phrase “at least one of A, B, and C” means A; B; C; A and B; A and C; B and C; or A, B, and C.

[0068] As used in any embodiment herein, the terms “system” or “module” can refer to, for example, software, firmware, and / or circuitry configured to perform any of the aforementioned operations. Software can be embodied as a software package, code, instructions, instruction sets, and / or data recorded on non-transitory computer readable storage medium. Firmware can be embodied as code, instructions or instruction sets hard-coded (e.g., nonvolatile) in memory devices. As used in any embodiment herein, “circuitry” can comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry (e.g., including one or more independent processors), state machine circuitry, and / or firmware that stores instructions executed by programmable circuitry or future computing paradigm (including hardware embodiments of accelerators such as neural net processors, and non-silicon implementations of the above). The circuitry may

[0069] Any of the operations described herein can be implemented in a system that includes one or more media (such as non-transitory storage media) having stored thereon instructions that, when executed by one or more processors, perform the methods. Herein, the term “processor” is meant generally to include all processors of a system, including, for example, server CPUs, mobile device CPUs, and / or other programmable circuitry. Also, it is intended that operations described herein can be distributed across several physical devices, such as processing structures at more than one different physical location. The storage media can include any type of tangible medium, for example, any type of disk including hard disks, floppy disks, optical disks, compact disk-read only memories (CD-ROMs), compact disk-read / write (CD-R / W) and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic and static RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, solid-state drives (SSDs), embedded multimedia controllers (eMMCs), secure digital input / output (SDIO) cards, magnetic or optical cards, or any type of media suitable for storing electronic instructions. Other embodiments can be implemented as software executed by programmable circuitry.

[0070] Accordingly, the present disclosure is directed to systems and methods for training a neural network using tensors that include a plurality of 16-bit floating point values and a plurality of bits that define an exponent shared by some or all of the 16-bit floating point values included in the tensor. The 16-bit floating point values can include IEEE 754 format 16-bit floating point values, and the tensor can include a plurality of bits that define a shared exponent. The tensor can include a shared exponent and 16-bit floating point values that include a variable bit length mantissa and a variable bit length exponent that can be dynamically set by the processor circuitry. The tensor can include a shared exponent and 16-bit floating point values that include: a variable bit length mantissa that can be dynamically set by the processor circuitry, a variable bit length exponent, and a shared exponent switch set by the processor circuitry to selectively combine the exponent of the 16-bit floating point value with the shared exponent.

[0071] The following examples pertain to further embodiments. As provided below, the following examples of the present disclosure can include subject matter such as at least one device, a method, at least one machine-readable medium for storing instructions that when executed by a machine cause the machine to perform actions based on the method, apparatus for performing actions based on the method, and / or a system for forming a magnetically aligned via in a semiconductor package substrate.

[0072] According to example 1, a system for training a neural network is provided. The system can include: processor circuitry; and a storage device coupled to the processor circuitry, the storage device including machine-readable instructions. The machine-readable instructions cause the processor circuitry to: generate a neural network training tensor, the neural network training tensor including: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values including: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

[0073] Example 2 can include elements as in example 1 wherein each of the plurality of 16-bit floating point values included in the neural network training tensor includes: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0074] Example 3 can include the elements of any of examples 1 or 2, wherein each of the plurality of 16-bit floating point values included in the neural network training tensor can include: a variable bit length mantissa provided by the first plurality of bits; a variable bit length exponent provided by the second plurality of bits; and a one-bit sign; wherein the instructions further cause the processor circuitry to select, based on the one or more neural network parameters: a number of bits to use to represent the variable bit length mantissa; and a number of bits to use to represent the variable bit length exponent.

[0075] Example 4 can include the elements of any of examples 1 to 3, wherein the one or more neural network parameters can include data indicative of a trend in at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

[0076] Example 5 can include the elements of any of examples 1 to 4, wherein each of the plurality of 16-bit floating point values included in the neural network training tensor can include: a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch to selectively combine the exponent portion of the corresponding 16-bit floating point number with the shared index.

[0077] According to example 6, a neural network training method is provided. The neural network training method can include: generating, by processor circuitry, a neural network training tensor including: a plurality of 16-bit floating point values, each of the 16-bit floating point values including: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and providing at least one of the plurality of 16-bit floating point values included in the training tensor as an input to the neural network.

[0078] Example 7 can include the elements of example 6, wherein generating the neural network training tensor can include: generating the neural network training tensor including the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0079] Example 8 can include elements of either of examples 6 or 7 wherein generating the neural network training tensor can include generating the neural network training tensor including the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a variable bit- length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign.

[0080] Example 9 can include elements of any of examples 6 through 8 and the method can additionally include selecting, by the processor circuitry, based on one or more neural network parameters: a first number of bits for representing the variable bit-length mantissa; and a second number of bits for representing the variable bit-length exponent.

[0081] Example 10 can include elements of any of examples 6 through 9 and the method can additionally include detecting, by the processor circuitry, a trend in one or more floating point values of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicating at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

[0082] Example 11 can include elements of any of examples 6 through 10 wherein selecting the first number of bits for representing the variable bit-length mantissa and the second number of bits for representing the variable bit-length exponent can include selecting, by the processor circuitry, the first number of bits included in the first plurality of bits and the second number of bits included in the second plurality of bits in response to detecting a trend indicating at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

[0083] Example 12 can include elements of any of examples 6 through 11 wherein generating the neural network training tensor can include generating the neural network training tensor including the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes: a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch for selectively combining the exponent portion of the corresponding 16-bit floating point number with the shared exponent.

[0084] According to example 13, a neural network training system is provided. The training system can include: means for generating, by processor circuitry, a neural network training tensor comprising: a plurality of 16-bit floating point values, each of the 16-bit floating point values comprising: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor; and means for providing at least one of the plurality of 16-bit floating point values included in the training tensor as an input to the neural network.

[0085] Example 14 can include the elements of example 13, wherein the means for generating the neural network training tensor can include: means for generating the neural network training tensor comprising the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values comprises: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0086] Example 15 can include the elements of any of examples 13 or 14, wherein the means for generating the neural network training tensor can further include: means for generating the neural network training tensor comprising the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values comprises: a variable-bit length mantissa provided by the first plurality of bits; a variable-bit length exponent provided by the second plurality of bits; and a one-bit sign.

[0087] Example 16 can include the elements of any of examples 13 to 15, and the system can further include: means for selecting: a first number of bits for representing the variable-bit length mantissa; and a second number of bits for representing the variable-bit length exponent.

[0088] Example 17 can include the elements of any of examples 13 to 16, and the system can further include: means for detecting a trend in one or more of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicating at least one of: a potential underflow condition or a potential overflow condition for one or more of the 16-bit floating point values included in the neural network training tensor.

[0089] Example 18 can include the elements of any of examples 13 to 17, wherein the means for selecting a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits to represent the variable bit- length exponents can further include means for selecting the first number of bits included in the first plurality of bits and the second number of bits included in the second plurality of bits in response to detecting a trend indicating at least one of an underflow condition or an overflow condition in one or more of the 16-bit floating-point values included in the network training tensor.

[0090] Example 19 can include the elements of any of examples 13 to 18, wherein the means for generating the neural network training tensor can further include means for generating the neural network training tensor including the plurality of 16-bit floating-point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating-point values includes: a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch for selectively combining the exponent portion of the corresponding 16-bit floating-point number with the shared index.

[0091] According to example 20, there is provided a non-transitory computer- readable storage medium comprising machine-readable instructions that, when executed by processor circuitry, cause the processor circuitry to: generate a neural network training tensor, the neural network training tensor including: a plurality of 16-bit floating-point values, each of the plurality of 16-bit floating-point values including: a first plurality of bits forming a mantissa of a corresponding floating-point number; and a second plurality of bits forming an exponent of the corresponding floating-point number; and a five-bit shared exponent common to each of the 16-bit floating-point values included in the training tensor.

[0092] Example 21 can include the elements of example 20, wherein the machine- readable instructions that cause the processor circuitry to generate a neural network training tensor including a plurality of 16-bit floating-point values further cause the processor circuitry to:

[0093] generate a neural network training tensor including a plurality of 16-bit floating-point values, each of the plurality of 16-bit floating-point values including: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0094] Example 22 can include the elements of any of examples 20 or 21, wherein the machine-readable instructions causing the processor circuitry to generate the neural network training tensor comprising a plurality of 16-bit floating-point values further cause the processor circuitry to: generate the neural network training tensor comprising a plurality of 16-bit floating-point values, each of the plurality of 16-bit floating-point values comprising: a variable-bit- length mantissa provided by the first plurality of bits; a variable-bit-length exponent provided by the second plurality of bits; and a one-bit sign; and select based on one or more neural network parameters:

[0095] a number of bits for representing the variable-bit-length mantissa; and a number of bits for representing the variable-bit-length exponent.

[0096] Example 23 can include the elements of any of examples 20 or 22, wherein the machine-readable instructions causing the processor circuitry to select, based on one or more neural network parameters, a number of bits for representing the variable-bit-length mantissa and the variable-bit-length exponent further cause the processor circuitry to: select the number of bits for representing the variable-bit-length mantissa and the variable-bit-length exponent based on one or more of: an underflow condition or an overflow condition in one or more of the 16-bit floating-point values included in the network training tensor.

[0097] Example 24 can include the elements of any of examples 20 or 23, wherein the machine-readable instructions causing the processor circuitry to generate the neural network training tensor comprising a plurality of 16-bit floating-point values further cause the processor circuitry to: generate the neural network training tensor comprising a plurality of 16-bit floating-point values, each of the plurality of 16-bit floating-point values comprising: a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch for selectively combining the exponent portion of the corresponding 16-bit floating-point number with the shared pointer.

[0098] According to example 25, a processor-based device is provided. The processor-based device can include a printed circuit board; a processor circuitry coupled to the printed circuit board; an input / output interface couplable to a neural network; and a storage device coupled to the processor circuitry, the storage device comprising machine-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to: generate a neural network training tensor, the neural network training tensor comprising: a plurality of 16-bit floating point values, each of the plurality of 16-bit floating point values comprising: a first plurality of bits forming a mantissa of a corresponding floating point number; and a second plurality of bits forming an exponent of the corresponding floating point number; and a five-bit shared exponent common to each of the 16-bit floating point values included in the training tensor.

[0099] Example 26 can include elements of example 25, wherein each of the plurality of 16-bit floating point values included in the neural network training tensor comprises: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0100] Example 27 can include elements of any of examples 25 or 26, wherein each of the plurality of 16-bit floating point values included in the neural network training tensor can comprise: a variable bit-length mantissa provided by the first plurality of bits; a variable bit-length exponent provided by the second plurality of bits; and a one-bit sign; wherein the instructions further cause the processor circuitry to select, based on one or more neural network parameters: a number of bits used to represent the variable bit-length mantissa; and a number of bits used to represent the variable bit-length exponent.

[0101] Example 28 can include elements of any of examples 25 to 27, wherein the one or more neural network parameters comprise a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the network training tensor.

[0102] Example 29 can include elements of any of examples 25 to 28, wherein each of the plurality of 16-bit floating point values included in the neural network training tensor comprises:

[0103] a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch for selectively combining the exponent portion of the corresponding 16-bit floating point number with the shared index.

[0104] According to Example 30, there is provided a tensor data structure. The tensor data structure may include: a plurality of 16-bit floating-point registers, each of the plurality of 16-bit floating-point registers including: a first plurality of bits corresponding to a mantissa value; a second plurality of bits corresponding to an exponent value; and a sign bit; and a shared exponent register associated with each corresponding floating-point register in the plurality of 16-bit floating-point registers.

[0105] Example 31 may include the elements of Example 30, wherein each of the plurality of 16-bit floating point registers comprises: a ten-bit mantissa provided by the first plurality of bits; a five-bit exponent provided by the second plurality of bits; and a one-bit sign.

[0106] Example 32 may include elements of any of Examples 30 or 31, wherein each of the plurality of 16-bit floating point registers includes: a variable bit length mantissa provided by the first plurality of bits; a variable bit length exponent provided by the second plurality of bits; and

[0107] A single symbol.

[0108] Example 33 may include elements as described in any of Examples 30 to 32, wherein each of the plurality of 16-bit floating-point registers includes: a mantissa provided by the first plurality of bits; an exponent provided by the second plurality of bits; a one-bit sign; and a one-bit switch for selectively combining the exponent portion of the corresponding 16-bit floating-point number with the shared exponent.

[0109] The terms and expressions used herein are used to describe rather than limit the terms, and when such terms and expressions are used, it is not intended to exclude any equivalents of the features (or parts thereof) shown and described, and it is recognized that various modifications within the scope of the claims are possible. Thus, the claims are intended to cover all such equivalents. Various features, aspects and embodiments have been described herein. As will be understood by those skilled in the art, features, aspects and embodiments are susceptible to combination with each other and to variations and modifications. Therefore, the present disclosure should be considered to include such combinations, variations and modifications.

[0110] As described herein, various embodiments may be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements may include: processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc.

[0111] Throughout this specification, references have been made to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, throughout this specification, the phrase "in one embodiment" or "in an embodiment" has been dropped in various places for clarity. Moreover, the inclusion of a particular feature, structure, or characteristic in one or more embodiments should not necessarily be construed as being dependent on the presence of the same feature, structure, or characteristic in other embodiments.

Claims

1. An apparatus for training a machine learning model, the apparatus comprising: programmable circuitry; and a storage device, the storage device being accessible by the programmable circuitry, the storage device comprising machine-readable instructions to cause the programmable circuitry to: train the machine learning model using a tensor having a sequence of 16-bit floating point values, a portion of the sequence of 16-bit floating point values comprising respective mantissa values, at least two of the 16-bit floating point values being related to a shared exponent, and wherein a first of the 16-bit floating point values comprises a bit representing whether the shared exponent is to be used; predict an occurrence of an overflow condition or an underflow condition affecting the tensor using the shared exponent of the machine learning model; and based on the prediction of the overflow condition or the underflow condition, adjust a value of the shared exponent of the machine learning model.

2. The apparatus of claim 1, wherein, at least one of the 16-bit floating point values is not related to the shared exponent.

3. The apparatus of claim 1, wherein, the programmable circuitry is to adjust the value of the shared exponent during the training of the machine learning model.

4. The apparatus of claim 1, wherein, the prediction of the overflow condition or the underflow condition represents a prediction of at least one of a future overflow or a future underflow.

5. The apparatus of claim 1, wherein, the respective mantissa values are represented by six bits.

6. The apparatus of claim 1, wherein, the portion of the sequence of 16-bit floating point values comprising the respective mantissa values is a first portion, and a second portion of the sequence of 16-bit floating point values comprises an exponent, and wherein the programmable circuitry is to selectively combine the shared exponent and the exponent in response to the adjustment of the value of the shared exponent.

7. The apparatus of claim 1, wherein, the programmable circuitry is to store the machine learning model in the storage device.

8. At least one non-transitory computer-readable storage medium comprising instructions that, when executed, cause at least one processor to at least: training a machine learning model using a tensor of sequences of 16-bit floating point values, a portion of the sequences of 16-bit floating point values including respective mantissa values, at least two of the 16-bit floating point values being related to sharing an exponent, and wherein, a first of the 16-bit floating point values comprises a bit representing whether the shared exponent is to be used; predict an occurrence of an overflow condition or an underflow condition affecting the tensor using the shared exponent of the machine learning model; based on the prediction of the overflow condition or the underflow condition, adjust a value of the shared exponent of the machine learning model; and store the trained machine learning model.

9. The at least one non-transitory computer-readable storage medium of claim 8, wherein, at least one of the 16-bit floating point values is not related to the shared exponent.

10. The at least one non-transitory computer-readable storage medium of claim 8, wherein, the instructions, when executed, cause the at least one processor to adjust the value of the shared exponent during the training of the machine learning model.

11. The at least one non-transitory computer-readable storage medium of claim 8, wherein, the respective mantissa values are represented by six bits.

12. The at least one non-transitory computer-readable storage medium of claim 8, wherein, the portion of the sequence of 16-bit floating point values comprising the respective mantissa values is a first portion, and a second portion of the sequence of 16-bit floating point values comprises an exponent, and wherein the instructions, when executed, cause the at least one processor to selectively combine the shared exponent and the exponent in response to the adjustment of the value of the shared exponent.

13. A method comprising: training a machine learning model using a tensor of a sequence of 16-bit floating point values, a portion of the sequence of 16-bit floating point values including respective mantissa values, at least two of the 16-bit floating point values being related to a shared exponent, and wherein a first of the 16-bit floating point values includes a bit representing whether the shared exponent is to be used; predicting an occurrence of an overflow condition or an underflow condition affecting the tensor using the shared exponent of the machine learning model; adjusting a value of the shared exponent of the machine learning model based on the prediction of the overflow condition or the underflow condition; and storing the trained machine learning model.

14. The method of claim 13, wherein, at least one of the 16-bit floating point values is not related to the shared exponent.

15. The method of claim 13, further comprising: adjusting the value of the shared exponent during the training of the machine learning model.

16. The method of claim 13, wherein, the respective mantissa values are represented by six bits.

17. The method of claim 13, wherein, the portion of the sequence of 16-bit floating point values including the respective mantissa values is a first portion, and a second portion of the sequence of 16-bit floating point values includes an exponent, and further includes selectively combining the shared exponent and the exponent in response to the adjustment of the value of the shared exponent.

18. A computing apparatus comprising means for performing the method of any one of claims 13 to 17.

19. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 13 to 17.

Images