Machine code instructions

Abstract

1. A processing device comprising: a plurality of operand registers, a first subset of the operand registers configured to store state information for a plurality of bins including a bin count and a range of values ​​associated with each bin, and a second subset of the operand registers configured to store a vector of floating-point values; and an execution unit configured to execute a first instruction taking as operands the state information for the plurality of bins and the vector of floating-point values, and to perform, in response to execution of the first instruction, the following: for each floating-point value, identify a bin among the plurality of bins within which the floating-point value falls in an associated value range based on an exponent of the floating-point value; and increment a bin count associated with the identified bin.

Description

Technical Field

[0001] The present disclosure relates to machine code instructions for processing vectors.

Background Art

[0002] There is an increasing interest in the development of processors designed for specific applications such as graphics processing units (GPUs) and digital signal processors (DSPs). Another type of specialized processor that has recently attracted attention is what the applicant refers to as an "IPU" (intelligent processing unit), which is specialized for machine intelligence applications. These can be employed, for example, as accelerator processors arranged to perform tasks assigned by a host, such as training or assisting in training a knowledge model such as a neural network, or making predictions or inferences based on such a model or assisting in its implementation.

[0003] Processors designed for machine intelligence applications may include in their instruction set dedicated instructions for performing arithmetic operations commonly employed in machine intelligence applications (the instruction set is a basic set of machine code instruction types configured to be recognized by the execution units of the processor, and each type is defined by its respective opcode and zero or more operands).

[0004] Machine intelligence models are often trained on large amounts of data, which can be expensive in terms of computational resources. To reduce the memory usage required to process large amounts of data, recent methods using low-precision representations of data and / or model parameters have been developed to improve the efficiency of such training. Recent training methodologies for neural networks use mixed-precision training, where some values ​​of the network have high-precision representations such as 32-bit floating-point, while other values ​​are represented in a low-precision format such as 16-bit floating-point. Low-precision formats have a narrower range of representations and are therefore more vulnerable to numerical underflow and overflow. One method used to overcome this challenge is known as "loss scaling," where the loss function used when training the neural network model is scaled to prevent the gradient of the loss function from falling below a threshold of representable values ​​in the chosen low-precision format. [Overview of the project] [Problems that the invention aims to solve]

[0005] Histograms are used to approximate the distribution of various types of values. Computer-processed floating-point numbers may be added to a histogram to generate approximate statistics of a set of values ​​processed within a computer program. As will be described in more detail herein, this can be useful for collecting data relating to the distribution of values ​​in a given floating-point representation, for example, to identify the degree of underflow and / or overflow that occurs when processing a given dataset, which can be reduced by adjusting the representation of the values ​​(e.g., adjusting for bias). [Means for solving the problem]

[0006] Described herein are machine language instructions defined within the processor's instruction set architecture, configured to assign the floating-point values ​​of a vector to histogram bins, and defined for multiple numeric representations, including 8-bit, 16-bit, and 32-bit floating-point formats. This allows for the collection of histograms of sets of values ​​with floating-point representations, which can be used to gain insights into the distribution of data and to enable the reduction of underflow and overflow. A particular advantage of the instructions described herein is that they enable the collection of such statistics with low overhead and high code density, as they allow for the generation and updating of histograms with fewer instructions than would be possible with standard arithmetic instructions.

[0007] A first aspect disclosed herein provides a processing device, the processing device comprising: a plurality of operand registers, a first subset of the operand registers configured to store state information for a plurality of bins, where for each of the plurality of bins the state information includes a range of values ​​associated with the bin and a bin count, and a second subset of the operand registers configured to store a vector of floating-point values; and an execution unit configured to execute a first instruction taking the plurality of bins state information and a vector of floating-point values ​​as operands, and in response to the execution of the first instruction, to identify, for each floating-point value, a bin among the plurality of bins in which the floating-point value falls within a range of associated values, based on the exponent of the floating-point value, and to increment the bin count associated with the bin identified for the floating-point value.

[0008] Identifying the bins within the range of values ​​associated with each floating-point value may involve selecting each bin from a set of bins and, for each bin, using a comparator circuit, comparing the exponent of each floating-point value with the conditions that define the range of values ​​associated with each bin.

[0009] The execution unit may be configured to perform in parallel, in response to the execution of a first instruction, whether each of several floating-point values ​​falls within the range of values ​​associated with each of several bins.

[0010] For each bin, the state information may include a threshold exponent field. The execution unit is configured to identify the bin for each floating-point value, depending on the value of the bin's threshold exponent field.

[0011] A threshold bin count saturation value may be defined. The execution unit is configured, in response to the execution of a first instruction, to compare the bin count for each bin with the threshold bin count saturation value, and to identify the bins from the unsaturated subset of bins whose floating-point values ​​are within the range of their associated values. The unsaturated subset of bins includes bins whose bin counts are less than the threshold bin count saturation value.

[0012] The state information for each bin may include the bin's sign indicator. The execution unit further includes a sign-checking circuit for comparing the sign of each floating-point value with the sign indicator of each bin. The range of values ​​associated with each bin includes only values ​​that match the sign of that bin.

[0013] The state information for each bin may include at least one mode indicator. The execution unit includes a bin confirmation circuit configured to determine a range of values ​​associated with each bin based on the value of the mode indicator for that bin.

[0014] In response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit may be further configured to identify a bin among several bins that falls within the range of values ​​associated with 0, and to increment the bin count associated with the identified bin. Alternatively, in response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit may be further configured to identify a bin among several bins that falls within the range of values ​​associated with the denormalized value, and to increment the bin count associated with the identified bin.

[0015] At least one mode indicator may include at least one of a threshold exponent field and a threshold range field.

[0016] If at least one mode indicator indicates the default mode, the lower limit of the value range may be the value in the threshold exponent field. The upper limit of the value range may be the sum of the value in the threshold exponent field and the value in the threshold range field.

[0017] When the mode indicator indicates a first special mode, the range of values ​​associated with the bin may include (i) 0 and (ii) one of the ranges of denormalized values.

[0018] The mode indicator may include a threshold index field. The first special mode is indicated by a predefined special value of the threshold index field.

[0019] When the mode indicator indicates a second special mode, the range of values ​​associated with the bin may include all values ​​below the threshold of that bin; when the mode indicator indicates a third special mode, the range of values ​​associated with that bin may include all values ​​above the threshold of that bin.

[0020] The mode indicator may include a threshold range field. The second and third special modes are indicated by special values ​​in the threshold range field, respectively.

[0021] The mode indicator may further include a threshold range field. If the threshold range value takes a special value of 0, the range of values associated with the bin includes only 0. If the threshold range field is not 0, the range of values associated with the bin includes the range of non-normalized values.

[0022] The execution unit may be configured to process the gradients of the machine learning application scaled by a scaling factor. The first instruction is executed by the execution unit on a plurality of vectors of gradients to generate a histogram including a plurality of bins, and the loss scaling factor is adjusted with respect to the total count of all bins based on the relative counts of a predetermined set of the plurality of bins.

[0023] The first subset and the second subset of operand registers may be registers of an arithmetic register file.

[0024] The floating-point value may be provided as (i) a 32-bit representation, (ii) a 16-bit representation, or (iii) an 8-bit representation.

[0025] The first subset and the second subset of registers may each include four 32-bit registers.

[0026] The second aspect disclosed herein provides a computer program including code configured to be executed on a processing device of any preceding claim. The code includes one or more instances of an instruction that takes, as operands, state information including the bin count of each of at least a plurality of bins and a vector of floating-point values. When executed, the code causes the processor, for each of the floating-point values, based on the exponent of the floating-point value, to identify, among the plurality of bins, the bin in which the floating-point value is within the associated range of values; and to increment the bin count associated with the bin identified for the floating-point value among the plurality of bins.

[0027] Another aspect disclosed herein provides a method of operating any of the processing devices disclosed herein. The method includes executing a first instruction that takes as operands state information including the bin count of each of a plurality of bins and a vector of floating point values, and, in response to execution of the first instruction, for each of the floating point values, identifying, based on the exponent of the floating point value, a bin among the plurality of bins in which the floating point value is within an associated range of values; incrementing the bin count associated with the bin identified for the floating point value among the plurality of bins.

[0028] The step of identifying the bin for each floating point value includes selecting each bin of the plurality of bins, selecting each of the floating point values of the vector for each bin, and comparing the exponent of each floating point value with a condition defining a range of values associated with the bin.

[0029] The method may include processing a plurality of combinations of bins and floating point values in parallel.

[0030] The state information may further include a threshold exponent field. The method may further include, for each of the floating point values, identifying one of the plurality of bins depending on the value of the threshold exponent field of each of the plurality of bins.

[0031] A threshold bin count saturation value may be defined. The method further includes, for each bin, comparing the bin count with the threshold bin count saturation value and identifying, among the non-saturated subset of bins, the bins in which each of the floating point values is within an associated range of values, the non-saturated subset of bins including bins whose bin count is less than the threshold bin count saturation value.

[0032] The state information for each bin may include a bin sign indicator. The method further includes comparing the sign of each floating point value with the bin sign indicator of each bin. The range of values associated with each bin includes only values that match the sign of the bin.

[0033] The state information for each bin may include at least one mode indicator. The method further includes performing bin confirmation to determine the range of values ​​associated with each bin based on the value of the bin's mode indicator. The mode indicator may include one or both of a threshold exponent field and a threshold range field.

[0034] This method may include, if each floating-point value is a denormalized floating-point value, identifying a bin among several bins that falls within the range of values ​​associated with 0, and incrementing the bin count associated with the identified bin among several bins. Alternatively, if each floating-point value is a denormalized floating-point value, this method may include, if several bins that fall within the range of values ​​associated with the denormalized value, and incrementing the bin count associated with the identified bin among several bins.

[0035] If at least one mode indicator indicates the default mode, the lower limit of the value range may be the value in the threshold exponent field. The upper limit of the value range is the sum of the value in the threshold exponent field and the value in the threshold range field. If at least one mode indicator indicates a first special mode, the value range associated with the bin may include 0 and one of the denormalized value ranges. The first special mode may be indicated by a predefined special value in the threshold exponent field. In the first special mode, if the threshold range value takes the special value of 0, the value range associated with the bin may include only 0. If the threshold range field is not 0, the value range associated with the bin may include the denormalized value range.

[0036] When the mode indicator indicates a second special mode, the range of values ​​associated with the bin may include all values ​​below the bin's threshold. When the mode indicator indicates a third special mode, the range of values ​​associated with the bin may include all values ​​above the bin's threshold.

[0037] The vector of values ​​may include the gradient of the machine intelligence algorithm. This method may further include scaling the gradient of the machine intelligence algorithm by a scaling factor. The scaling factor is adjusted based on the relative counts of a given subset of bins to the total count of all bins.

[0038] Another aspect disclosed herein provides a non-temporary computer-readable storage medium comprising code configured to run on a processing device disclosed herein, wherein the code comprises one or more instances of an instruction that takes state information, each containing a bin count for at least a plurality of bins, and a vector of floating-point values ​​as operands, and when executed, causes the processor to, for each bin of floating-point values, Based on the exponent of the floating-point value, identify the bin among several bins in which the floating-point value falls within the range of the associated value. Increment the bin count associated with the bin identified for the floating-point value among multiple bins, Execute this. [Brief explanation of the drawing]

[0039] [Figure 1] This is a schematic block diagram of an exemplary multithreaded processor. [Figure 2] A schematic diagram of the logical block structure of an exemplary processor is shown. [Figure 3] This is a schematic block diagram of a processor, including the array of constituent processors. [Figure 4] This is a schematic diagram of a graph used in machine intelligence algorithms. [Figure 5] This is a schematic block diagram of the logic circuit for executing the hist instruction. [Figure 6] This is a flowchart illustrating the logic of the hist command. [Figure 7] This is a flowchart illustrating the logic of the second hist instruction. [Figure 8]This diagram schematically illustrates how the histogram generated by the hist instruction is used to adjust the scaling factor for training a machine intelligence model. [Modes for carrying out the invention]

[0040] Figure 1 shows an example of a processor 4 according to an embodiment of the present disclosure. The architectures of Figures 1 and 2 are exemplary architectures in which the present invention may be implemented. As will be apparent to those skilled in the art of computer architecture, the present invention may be implemented in various processors having various architectures. This architecture is described in more detail in U.S. Patent Application Publication No. 16 / 276834, which is incorporated herein by reference in its entirety.

[0041] The processor 4 includes a multithreaded processing unit 10 in the form of a barrel-threaded processing unit and local memory 11 (i.e., on the same tile in the case of a multi-tile array or on the same chip in the case of a single processor chip). A barrel-threaded processing unit is a type of multithreaded processing unit in which the execution time of a pipeline is divided into an iterative sequence of interleaved time slots (each of which may be occupied by a given thread). This may also be called parallel execution. The memory 11 includes instruction memory 12 and data memory 22 (which may be implemented in various addressable memory units or in various regions of the same addressable memory unit). The instruction memory 12 stores machine code executed by the processing unit, while the data memory 22 stores both data manipulated by the executable code and data output by the executable code (e.g., as a result of such operations).

[0042] Memory 12 stores multiple different threads of the program. Each thread contains a sequence of instructions for performing one or more specific tasks. Note that the instructions referred to herein mean machine code instructions consisting of a single opcode and zero or more operands, i.e., one instance of a basic instruction in the processor's instruction set. In embodiments, the program includes multiple worker threads and a supervisor subprogram which may be structured as one or more supervisor threads.

[0043] The execution pipeline 13 includes a fetch stage 14, a decryption stage 16, and an execution stage 18 which includes an execution unit capable of performing arithmetic and logical operations, address calculations, load and store operations, and other operations as defined by the instruction set architecture.

[0044] The dedicated hardware includes a set of context registers 26, one per slot in a cycle, for at least each separate set of threads that may run concurrently. When discussing multithreaded processors, "context" refers to the program state of each thread running in parallel with each other (e.g., program counter value, state, and current operand value). A context register refers to the respective register for representing this program state of each thread. Each set of context registers 26 includes each of the one or more control registers, which also include at least the program counter (PC) for each thread (to maintain tracking of the instruction address the thread is currently executing) and, in some embodiments, one or more control state registers (CSRs) that record the current state of each thread (e.g., whether it is currently running or suspended). Each set of context registers 26 also includes the operand registers for each set to temporarily hold the operands of the instructions executed by the thread, i.e., values ​​that, when executed, have been operated on or result from the operations defined by the opcode of the thread's instruction. Each set of registers 26 may be implemented in one or more register files. Note that while "operand" strictly refers to the portion of an instruction that specifies the data to be manipulated, it is used more generally in this specification to refer to both the register index specified within the instruction and the data held within said register.

[0045] The fetch stage 14 can access each program counter (PC) of the context. For each thread, the fetch stage 14 fetches the next instruction for that thread from the next address in program memory 12 indicated by the program counter. The program counter automatically increments with each execution cycle unless branched by a branch instruction. The fetch stage 14 then passes the fetched instruction to the decryption stage 16, which then passes the decryption instruction specifier, along with the decrypted addresses of any operand registers specified in the instruction, to the execution unit 18 so that the instruction may be executed. In the case of arithmetic instructions, the execution unit 18 can access operand registers and control state registers that can be used when executing the instruction based on the decrypted register addresses (for example, by adding, multiplying, subtracting, or dividing values ​​in two operand registers and outputting the results to other operand registers of each thread). Alternatively, if the instruction defines a memory access (load or store), the load / store logic of the execution unit 18 loads the value from data memory into the operand register of each thread, or stores the value from the operand register of each thread into data memory 22, in accordance with the instruction.

[0046] The fetch stage 14 is connected to fetch instructions to be executed from the instruction memory 12 under the management of the scheduler 24. The scheduler 24 is configured to control the fetch stage 14 to sequentially fetch instructions from each of the sets of concurrently executing threads within the time slots of the iterative sequence, and thus divide the resources of the pipeline 13 into multiple time interleaved time slots.

[0047] Figure 2 shows details of the multithreaded processor 4, including details of the execution unit 18 and context register 26.

[0048] The processor includes each instruction buffer 53 for each of the M threads that may run concurrently. The context register 26 includes each main register file (MRF) 26M for each of the worker M context and the supervisor context. The context register further includes at least each auxiliary register file (ARF) 26A for the worker context. The context register 26 further includes a common weight register file (WRF) 26W that all currently running worker threads can access to read. The WRF may be related to the supervisor context in that the supervisor thread is the only thread that can write to the WRF. The context register 26 may also include each group of control state registers 26CSR for each of the supervisor context and the worker context. The execution unit 18 includes a main execution unit 18M and an auxiliary execution unit 18A. The main execution unit 18M includes a load / storage unit (LSU) 55 and an integer arithmetic logic unit (IALU) 56. The auxiliary execution unit 18A includes at least a floating-point unit (FPU).

[0049] In each of the J interleaved time slots S0...SJ-1, the scheduler 24 controls the fetch stage 14 to fetch at least one instruction for each thread from the instruction memory 11 into each of the J instruction buffers 53 corresponding to the current time slot. In this embodiment, each time slot is one execution cycle of the processor, but other methods (e.g., weighted round robin) are not excluded. In each execution cycle of the processor 4 (i.e., each cycle of the processor clock that times the program counter), the fetch stage 14 fetches either a single instruction or a small "instruction bundle" (e.g., a 2-instruction bundle or a 4-instruction bundle), depending on the implementation. Each instruction is then issued via the decoding stage 16 to one of the LSU 55 or IALU 56 of the main execution unit 18M or the FPU of the auxiliary execution unit 18A, depending on whether the instruction is a memory access instruction, an integer arithmetic instruction or a floating-point arithmetic instruction (according to its opcode). The LSU55 and IALU56 of the main execution unit 18M execute those instructions by using registers from the MRF26M, with specific registers in the MRF26M defined by the operands of the instruction. The FPU of the auxiliary execution unit 18A performs arithmetic by using registers in the ARF26A and WRF26W, with specific registers in the ARF defined by the operands of the instruction. In embodiments, the registers in the WRF may be implicit in terms of instruction type (i.e., predetermined with respect to that instruction type). The auxiliary execution unit 18A may also include circuitry in the form of internal logical latches for holding some internal states 57 for use when performing one or more operations of several types of floating-point arithmetic instructions.

[0050] In embodiments where instructions are fetched and executed in bundles, individual instructions within a given instruction bundle are executed simultaneously in parallel under independent pipelines 18M and 18A (as shown in Figure 3). In embodiments where two bundles of instructions are executed, the two instructions may be executed simultaneously across their respective auxiliary and main pipelines. In this case, the main pipeline is configured to execute several types of instructions using MRF, and the auxiliary pipeline is used to execute several types of instructions using ARF. Pairing instructions into suitable complementary bundles can be handled by the compiler.

[0051] Each worker thread context has instances of the main register file (MRF) 26M and auxiliary register file (ARF) 26A themselves (i.e., one MRF and one ARF for each barrel thread slot). The functionality described herein with respect to the MRF or ARF is understood to operate on a context-by-context basis. However, there is a single shared weight register file (WRF) shared among threads. Each thread can access only its own context's MRF and ARF. However, all currently running worker threads can access the common WRF. Thus, the WRF provides a common set of weights for use by all worker threads. In embodiments, only the supervisor can write to the WRF, and workers can only read from the WRF.

[0052] The instruction set of processor 4 includes at least one type of load instruction, which, when its opcode is executed, causes LSU 55 to load data from data memory 22 into each ARF 26A of the thread from which the load instruction was executed. The destination location in the ARF is specified by the operands of the load instruction. Another operand of the load instruction specifies an address register in each MRF 26M that holds a pointer to the address in data memory 22 from which the data is loaded. The instruction set of processor 4 also includes at least one type of store instruction, which, when its opcode is executed, causes LSU 55 to store data from each ARF of the thread from which the store instruction was executed into data memory 22. The source location for the store in the ARF is specified by the operands of the store instruction. Another operand of the store instruction specifies an address register in the MRF that holds a pointer to the address in data memory 22 from which the data is stored. Generally, the instruction set may include separate load instruction types and store instruction types, and / or at least one load / store instruction type that combines load and store operations into a single instruction.

[0053] The processor instruction set also includes one or more types of arithmetic instructions for performing arithmetic operations. According to embodiments disclosed herein, these may include at least one type of arithmetic instruction that utilizes a common weight register file WRF26W. This type of instruction takes at least one operand that specifies at least one source of the corresponding arithmetic operation in each ARF26A of the thread from which the arithmetic instruction is executed. However, at least one other source of the arithmetic instruction resides in a common WRF that is common to all worker threads. In embodiments, this source is implicit within the arithmetic instruction (i.e., implicit with respect to this type of arithmetic instruction). With respect to machine code instructions, implicit means that it does not require the operand to be specified. That is, in this case, the location of the source in the WRF is specific to the opcode (pre-defined with respect to that particular opcode). Alternatively, in other embodiments, the arithmetic instruction may take an operand that specifies which set of weight registers from among several different sets in the WRF the weights are taken. However, the fact that the source of weighting is found within the WRF (as opposed to, for example, the general-purpose MRF or ARF) remains implicit.

[0054] In response to the opcode of an arithmetic instruction of the relevant type, the arithmetic unit (e.g., FPU) within the auxiliary execution unit 18A performs the arithmetic operation specified by the opcode, which includes performing operations on the values ​​in the specified source registers within each thread's ARF and the values ​​in the source registers within the WRF. The arithmetic unit also outputs the result of the arithmetic operation to the destination register within each thread's ARF, which is explicitly specified by the destination operand of the arithmetic instruction.

[0055] Exemplary types of arithmetic instructions that may draw from a common source within the WRF26W include one or more vector multiplication instruction types, one or more matrix multiplication instruction types, one or more cumulative vector multiplication instruction types and / or cumulative matrix multiplication instruction types (which accumulate the result of a multiplication from one instance of an instruction to the next instance) and / or one or more convolution instruction types. For example, a vector multiplication instruction type may multiply an explicit input vector from the ARF26A by a given vector of weights from the WRF, or a matrix multiplication instruction type may multiply an explicit input vector from the ARF by a given matrix of weights from the WRF. As another example, a convolution instruction type may convolve an input matrix from the ARF by a given matrix from the WRF. Having a common shared weight register file and WRF for multiple threads allows each thread to multiply or convolve a common kernel with its own respective data. This is useful because it is a scenario that often occurs in machine learning applications, for example, where each thread represents a different node in a neural network, and a common kernel represents the feature being searched or trained on (e.g., an area or volume of graphic data, an edge, or a specific shape).

[0056] In this embodiment, values ​​in WRF26W may be written by a supervisor thread. The supervisor (in this embodiment, starting by running in all slots S0···SM) first executes a series of put instructions to write some common weight values ​​to a designated location in the WRF. Next, the supervisor executes run instructions (or run-all instructions) to start up each worker in some or all of slots S0···SJ-1. At this time, each worker includes one or more instances of one or more arithmetic instructions of the type discussed above, but using the common weights written to WRF26W by the supervisor, to perform corresponding arithmetic operations on its own respective input data loaded into its ARF26A. Upon completion of its respective task, each thread executes a terminate instruction to return its slot to the supervisor. Once all started threads have completed their respective tasks, the supervisor may write new values ​​to the WRF and start up a new set of threads (or start up a new set to continue using the existing values ​​in the WRF).

[0057] It should be noted that the labels “primary,” “auxiliary,” and “weighting” are not necessarily limiting. In embodiments, these may be any first register file (per worker context), a second register file (per worker context), and a shared third register file (e.g., a portion of the supervisor context, but accessible to all workers). The ARF26A and auxiliary execution unit 18 may also be called the arithmetic register file and arithmetic execution unit, as they are used for arithmetic instructions (or at least floating-point operations). The MRF26M and auxiliary execution unit 18 may also be called the memory address register file and arithmetic execution unit, as one of its uses is for accessing memory. The weight register file (WRF) 26W is so named because it is used to hold multiplication weights used in one or more specific types of arithmetic instructions, which will be discussed in more detail below. For example, these may be used to represent the weights of nodes in a neural network. Alternatively, the MRF can be called an integer register file because it is used to hold integer operands, while the ARF can be called a floating-point register file because it is used to hold floating-point operands. In embodiments where instructions are executed in bundles of two, the MRF is the register file used by the main pipeline, and the ARF is the register file used by the auxiliary pipeline.

[0058] However, it should be noted that in alternative embodiments, the register space 26 is not necessarily divided into these separate register files for these various purposes. Instead, instructions executed via the main and auxiliary execution units may specify several registers from the same shared register file (one register file per context in the case of a multithreaded processor). The pipeline 13 also does not necessarily have parallel configuration pipelines (e.g., auxiliary and main pipelines) for concurrently executing bundles of instructions.

[0059] The processor 4 may also include memory 11 and exchange interfaces 51 for exchanging data with one or more other resources, such as other instances of the processor and / or other instances of external devices such as a network interface or network-attached storage (NAS) device. As shown in Figure 3, in the embodiment, the processor 4 may form one of an array 6 of interconnected processor tiles, each tile executing a portion of a broader program. Thus, individual processors 4 (tiles) form part of a broader processor or processing system 6. The tiles 4 may be connected together via interconnect subsystems 34 connected via their respective exchange interfaces 51. The tiles 4 may be mounted on the same chip (i.e., die) or different chips or a combination thereof (i.e., an array may be formed from multiple chips, each containing multiple tiles 4). Thus, the interconnect system 34 and exchange interfaces 51 may include internal (on-chip) interconnection mechanisms and / or external (inter-chip) exchange mechanisms accordingly.

[0060] In one exemplary application of a multithreaded and / or multitile processor or system, program execution across multiple threads and / or tiles 4 includes an algorithm configured to train a machine intelligence algorithm, such as a neural network, and / or perform inference based on the neural network. In such an embodiment, each worker thread, or a portion of program execution on each tile, or each worker thread on each tile is used to represent a different node 102 in a neural network (a type of graph), and accordingly, communication between threads and / or tiles represents an edge 104 between the nodes 102 in the graph. This is illustrated in Figure 4.

[0061] Machine intelligence begins with a learning phase in which the machine intelligence algorithm learns a knowledge model. The model includes a graph of interconnected nodes (i.e., vertices) 102 and edges (i.e., links) 104. Each node 102 in the graph has one or more input edges and one or more output edges. Some of the input edges of some nodes of node 102 are some of the other output edges of node 102, thereby connecting the nodes together to form the graph. Furthermore, one or more of the input edges of one or more nodes of node 102 form the inputs to the graph as a whole, and one or more of the output edges of one or more nodes of node 102 form the outputs of the graph as a whole. At any given time, a node may also have all of the inputs to the graph, outputs from the graph, and connections to other nodes. Each edge 104 carries values ​​or more frequently tensors (n-dimensional matrices), which form the inputs and outputs provided to and from the nodes 102 on each of the input and output edges.

[0062] Each node 102 represents a function of one or more inputs received on its input edge, and the result of this function is the output provided on the output edge. Each function is parameterized by one or more parameters (sometimes called weights, but not necessarily multiplicative weights). In general, functions represented by various nodes 102 can be functions of various forms and / or can be parameterized by various parameters.

[0063] In the learning phase, the algorithm receives empirical data, i.e., multiple data points, representing various possible combinations of inputs to the graph. As more empirical data is received, the algorithm gradually tunes the parameters of various nodes 102 in the graph based on the empirical data in an attempt to minimize the parameter error. The goal is to find parameter values ​​such that the output of the graph comes as close as possible to the desired output of a given input. When the graph as a whole tends toward this state, the graph is said to converge. After a suitable degree of convergence, the graph can then be used to make predictions or inferences, i.e., to predict the outcome of a given input or to infer the cause of a given output.

[0064] The learning stage can take many different forms. For example, in supervised methods, the input experience data takes the form of training data, i.e., inputs corresponding to known outputs. For each data point, the algorithm can tune its parameters so that the output more closely matches the known output of a given input. In the subsequent prediction stage, the graph can then be used to map input queries to approximate predicted outputs (and vice versa when performing inference). Other methods are also possible. For example, in unsupervised methods, there is no concept of a baseline outcome per input data; instead, the machine intelligence algorithm is tasked with identifying its own structure within the output data. Or, in reinforcement methods, the algorithm tries at least one possible output for each data point in the input experience data and is told whether this output is positive or negative (and to what extent it is potentially positive or negative), e.g., win or lose, or reward or cost, etc. Over many tries, the algorithm can gradually tune the parameters of the graph so that it can predict inputs that will produce a positive outcome. Various methods and algorithms for learning graphs will become known to those skilled in machine learning.

[0065] One common algorithm used when training machine intelligence models is gradient descent. This relates to a neural network containing 102 node parameters that are tuned to empirical data as described above. In gradient descent, this tuning is done by defining a loss or error function and updating the model parameters in the negative direction of the gradient of the loss function; in other words, updating the parameters to minimize the defined loss function. Several types of gradient descent algorithms have been developed, and these will be common to those skilled in machine learning.

[0066] When training machine intelligence models based on empirical data, depending on the task, it may be necessary to process very large amounts of data at once. One way to improve processing efficiency is to employ multithreading and concurrent parallelism, as described above, which allows pipeline latency to be hidden so that multiple threads run simultaneously. Another way to improve processing speed and efficiency is to change the format of the data being processed so that more individual data elements can be processed simultaneously, and more data elements can be stored in a given capacity of memory at any given time during processing, so that each data value in the network is stored using less memory.

[0067] Typically, the parameters and values ​​of a neural network, as well as its gradients, are stored in floating-point format. "Single-precision" floating-point values ​​have 32 bits and can be called 32-bit floating-point format or float32, while "double-precision" numbers or "doubles" are represented by 64 bits. Similarly, 16-bit floating-point numbers are called "1 / 2 precision," and 8-bit floating-point numbers, though less widely used, can be called "1 / 4 precision."

[0068] Floating-point representations consist of three distinct components: a sign component, a mantissa component, and an exponent component. In single-precision (i.e., 32-bit) floating-point representations according to the IEEE 754 standard, the sign component consists of a single bit, the exponent consists of 8 bits, and the mantissa consists of 23 bits. In standard half-precision (i.e., 16-bit) floating-point representations, the sign component consists of a single bit, the mantissa consists of 10 bits, and the exponent consists of 5 bits. In most cases, a number is given by these three components using the following formula:

number

[0069] The "offset" displayed for an exponent depends on the number of bits used to represent the exponent, which in turn depends on the level of precision. In single-precision representation, the offset is typically equal to 127. In half-precision representation, the offset is typically equal to 15. However, it should be noted that various numbers of exponent bits can be defined for a given precision or total number of bits in the floating-point representation, and bit assignments other than those specified in the above standards are possible.

[0070] "I" is the implicit bit derived from the exponent. If the exponent bit sequence consists of all zeros or any other number other than all ones, the implicit bit is equal to 1, and the number is known as the "norm". In this case, the floating-point number is given by the following formula:

number

[0071] If the exponent bit sequence consists entirely of zeros, the implicit bits are equal to zero, and the number is a denormalized number, also referred to herein as the “denorm,” “denormalized,” or “denormalized number.” The terms “denormalized” and “denorm” or “denormalized number” may be used interchangeably herein to refer to a number between the least norm and zero. In this case, the floating-point number is given by the following formula:

number

[0072] The dinorm is useful for representing numbers smaller than those that would otherwise be representable by a limited number of exponential bits.

[0073] When a number is too small to be represented by the given number of mantissa and exponent bits in its floating-point representation, this is called underflow. Methods for reducing underflow when training machine intelligence models will be discussed later. Non-zero values ​​below the threshold of representable values ​​can be treated as "denormalized" numbers, which are floating-point values ​​that satisfy the underflow gap between the lowest "normalized" representable number and zero. Denormalized numbers can be represented by using implicit bits based on the value of the exponent bit, as will be explained below.

[0074] According to embodiments of the present invention, the execution unit 18 is configured to execute a new machine code instruction. A histogram instruction, also referred to herein as a “hist instruction,” takes two operands: The first operand defines a set of multibins by the range of values ​​that each bin extends to, as well as the count of values ​​within each bin; and the second operand defines a vector of multiple values. The hist instruction is executed to determine which bin of the histogram each value belongs to and to increment the histogram count of that bin by 1 for each value that falls within the range.

[0075] As understood in the field of computer processor architecture, machine code instructions are represented as opcodes that define the operation of the instruction and as one or more operands, and include data to which the operation is applied. A histogram instruction takes as operands (a) register indices of a set of registers containing state information for a set of bins, (b) state information including bin ends and counts, and (c) a vector of values ​​to be added to the histogram, as described in detail below, adding any value that falls within the limits of each bin to the count of that bin. A histogram instruction can be defined in two or more numeric forms; for example, a histogram instruction can be defined not only for 32-bit floating-point numbers but also for vectors of 16-bit floating-point numbers and 8-bit floating-point numbers. Both the histogram bin state information and the value vectors are stored in their respective registers defined within the ARF26A.

[0076] The hist instruction can be applied to vectors of values ​​having 32-bit, 16-bit, or 8-bit representations, and each form has a different instruction opcode because the function of the instruction differs slightly from form to form, as will be explained in detail below. The operand of a hist instruction that defines the vector of values ​​to be binned is a set of register indices that hold a defined number of bits. In other words, if a given register or set of registers referenced in the operand holds a vector of four 32-bit values ​​totaling 128 bins, then if the values ​​being processed are instead represented in 16 bits, one or more registers may hold twice as many values, i.e., a vector of eight values. For 8-bit values, the same register referenced by a given operand may hold 16 values ​​totaling 128 bits.

[0077] The nomenclature used herein for hist instructions is "f32v4hist", where "f32" refers to the floating-point format of the input and "v4" refers to the size of the vector. As mentioned above, with respect to lower precision representations, the number of vector values ​​that can be processed within a single instruction increases. This results in several hist instructions, including "f32v4hist", "f16v8hist" which bins a vector of eight values ​​represented in 1 / 2 precision or floating-point 16 format, and "f8v16hist" which bins a vector of sixteen values ​​represented in 1 / 4 precision or floating-point 8 format. Note that this list of instructions is not exhaustive. For representations of vectors with various numbers of bits and / or various sizes, other instructions with the same functionality as described below may be defined.

[0078] Here, we describe the 32-bit histogram instruction "f32v4hist". This instruction causes the execution unit 18 to count a vector of four single-precision (i.e., 32-bit) elements into a histogram of four buckets (bins). In this example, each register is a 32-bit register such that the second operand defines an array of four 32-bit values. However, in other embodiments, the registers may have different capacities, e.g., 16-bit or 64-bit, and the size of the vector and / or the precision of the values ​​may differ. As will be described later, with respect to a given register capacity, various instances of the histogram instruction may be defined to handle vectors with lower-precision representations such that the relationship between values ​​and registers is not necessarily one-to-one. Note that while this example describes a histogram of four bins, histograms of any number of bins may be used. The instruction takes, as the first operand, histogram state information including several fields which will be described in more detail below, and as the second operand, a vector of values. The instruction is configured to control the execution unit 18 to update the count field for each bin of the histogram before returning the updated histogram value to its corresponding register. This is illustrated below with reference to Figure 5.

[0079] The instruction has the following syntax: f32v4hist $aSrcDst0:aSrcDst0+3 $aSrc0:aSrc0+3, where f32v4hist is the instruction identifier as described above, and the operand identifies the register index range by identifying a set of four registers (SrcDst0~SrcDst0+3) of the ARF26A, respectively, where "SrcDst" is used to indicate that both registers are the source and destination registers of the hist instruction. This operand defines the state information of the histogram, including the count and range of the histogram bins, and when the instruction is executed, the updated bin count is stored in the registers of the same range. Only the bin count is affected by the hist instruction; other state information remains fixed.

[0080] Each bin is defined by an exponent [base, base + range] within a range. In one exemplary embodiment, each element of the first operand has 32 bits and takes the following form: • [17:0]BIN_COUNT: The 18 bits of each bin in the first operand are allocated to the saturated count value (in other words, once a given maximum count is reached, the instruction no longer updates the count). • [25:18] THRESH_EXP: These 8 bits represent the lower threshold exponential configuration value that defines the limit of a given bin. This field is not modified by the execution of the hist instruction. ·[29:26]THRESH_RANGE: These four bits represent the exponent range configuration value. These bits have two special values. If THRESH_EXP is not all equal to 1, a THRESH_RANGE of 0 (also written as 0b0000) means that all exponents less than or equal to the current bin's THRESH_EXP should be counted, and a THRESH_RANGE of 0b1111 means that all exponents greater than or equal to the given bin's THRESH_EXP should be counted. • [31:30] SIGNC: These two bits determine how the sign is handled when adding new values ​​to the histogram. 0b00 or 0b01 means the sign is ignored, and both positive and negative values ​​are counted if they fall within a given range of the bins. 0b means only positive values ​​are counted. 0b11 means only negative values ​​are counted.

[0081] Figure 5 shows an exemplary logic circuit implemented within the execution unit 18 for executing the hist instruction process. As a floating-point instruction, the circuit for executing the hist instruction is implemented within the FPU of the auxiliary execution unit 18A. Note that while Figure 5 shows only the circuit necessary for executing the hist instruction, the FPU of the auxiliary execution unit 18A further includes circuitry for performing other types of floating-point operations.

[0082] As described above, the instruction takes register indices as operands that define a register 704 holding the bin state information of the histogram and another register holding the values ​​of the vector 702 to be added to the histogram, both of which are held in the arithmetic (or auxiliary) register file 26A. The bin state information is passed to a bin check circuit 704 that checks the bin state information, and the threshold range bits and the threshold indices of a given bin are checked with respect to the special values ​​THRESH_EXP and THRESH_RANGE, as described above.

[0083] In embodiments further described below with reference to Figure 6, the denormalized number is treated as 0, and no special value of THRESH_EXP is used. However, in other embodiments, as described below with reference to Figure 7, a special value of THRESH_EXP, 0b11111111, is defined which defines a given bin as containing only 0 or the dinorm, and the value of THRESH_RANGE is used to determine whether the bin is 0 (if THRESH_RANGE=0) or the dinorm (with respect to any other value of THRESH_RANGE). Special values ​​of THRESH_RANGE when THRESH_EXP is not 0b11111111 are described above.

[0084] The special cases described above indicate various modes of operation for the instruction and are referred to herein as “mode indicators.” For example, if THRESH_EXP is not 0b11111111 and THRESH_RANGE is 0, the hist instruction for that bin operates in a mode that counts values ​​below the threshold, instead of the default mode that counts values ​​within a range with upper and lower limits. The bin verifier circuit verifies not only each of these specific cases but also the bin’s SIGNC bit to determine how to handle the sign of the value. The bin verifier circuit then outputs one or more signals indicating whether any particular case can be flagged by the histogram state, and if so, which one it is. This is passed to the comparator circuit 708, which processes the value along with the bin state information to determine whether the value falls within the range defined for that bin. Note that each value in the vector is processed for each of the bins defined by the instruction. However, Figures 6 and 7 are simplified for clarity and show only the processing of a single value for each bin.

[0085] The bin verification circuit 706, shown as a single component in Figure 5, may include multiple logic components to verify both the threshold index and threshold range, and to direct the state information to the comparison circuit 708. If the bin verification circuit 706 determines that the threshold index and threshold range do not have special values, the threshold index and threshold range are processed by logic in the comparison circuit, which takes the value 702 of a vector and determines whether it falls within the range of a given bin, i.e., whether the value 702 of the vector is greater than or equal to the bin's threshold index and less than the upper limit given by the threshold index and threshold range. The comparison circuit also includes logic for a specific case, as explained with reference to Figure 7. If the bin verification circuit 206 determines that the threshold index takes the special value 0b11111111 and the threshold range is 0, only 0 is counted, and the value 702 is passed to logic to verify that the mantissa and exponent of the value are 0. Logic to verify whether the value 702 is a dinorm is also implemented, and the bin verification circuit 706 determines, as described above, that the threshold index and range are set to their respective values ​​to indicate this specific case. Again, although the comparator circuit 708 is shown as a single component, it may include many logic circuits to handle several specific cases, which will be further described below with reference to Figures 6 and 7 above. If THRESH_EXP and THRESH_RANGE do not take any of the special values ​​described above, the instruction returns to the default mode of counting values ​​whose exponents fall within the range [THRESH_EXP, THRESH_EXP + THRESH_RANGE]. For all cases, the comparator circuit 708 includes logic to compare the value to either a bin state or a special value (such as 0), and to issue a signal to the increment circuit only if the value satisfies the conditions of the comparison logic.

[0086] The increment circuit 710 increments the bin count of bin state 704 by 1 when the corresponding signal is received from the comparator circuit 708. The updated count is written back to the register containing the bin state in ARF26A, as indicated by the arrow from the increment circuit 710. The above description refers only to the processing of a single value of a single-bin vector, but it should be noted that the above process is repeated for each given value and bin. The writing back of the bin count to the register in ARF26A can only occur after all values ​​and bins of the instruction have been processed (where the vector of counts for all bins is used to maintain the updated count value while the instruction is executing) and before the count is written back to the register containing the bin state information.

[0087] Figure 6 is a flowchart illustrating the process of executing a hist instruction when denormalized values ​​are treated as 0. Before instruction calculation begins, the execution unit 18 prepares the operand data to extract the individual elements of each bin and vector value so that each operand is represented as an array of values ​​in a given numerical representation, which in this example is a 32-bit floating-point number.

[0088] In step S600, the bin index is set to 0, representing the first bin of the histogram. The schematic diagrams shown in Figures 6 and 7 illustrate the steps taken to process a single value for a single bin, which may be the first bin. However, as noted above, multiple bins may be processed in parallel, and in such embodiments, the initialization step S600 is not required to set an initial bin that will be sufficiently processed before the execution unit considers the next bin, and instead some or all of the bins may be processed simultaneously by steps S602-S614. The bin count is BIN_COUNT, read from the BIN_COUNT bit of that bin in the operand. The count may be added to an array "count" indexed by the bin, which gives the count of the values ​​for each bin in the histogram. The "threshRange" and "threshExp" variables are also defined as the THRESH_RANGE and THRESH_EXP bits, respectively, of the operand of a given bin.

[0089] The auxiliary execution unit 18A processes each value of the second operand, i.e., each value of the vector assigned to the bins of the histogram, within a loop. Processing of multiple values ​​is not shown in Figure 6 or 7. However, as noted above, the processing of bins and values ​​can be performed in parallel by the execution units, and there is no requirement to process each value of the vector in sequence. For brevity, these values ​​will be referred to as "input values" in the following description. In step S602, the execution unit 18A checks whether the count has saturated with respect to the current bin, i.e., whether the count has reached the maximum count value, and this step is performed by using the bin checking circuit 706. If the count is at its maximum value, the loop is interrupted, the count does not change, and the value is not processed. Otherwise, the instruction proceeds to process the input values ​​sequentially. For each input value, step S604 checks whether the sign bit of the bin matches the sign of the input value based on the value given above SIGNC. If the signs do not match, the value is not added to the given bin, and the count remains the same.

[0090] If the signs match, the bin verification circuit 706 performs S606, which is the first verification regarding the special value of threshRange. If threshRange is zero, verification (S608) is performed by the comparison circuit 708 to determine whether the input value has an exponent less than or equal to the threshold exponent of the current bin. As described above, any value having an exponent less than or equal to the threshold exponent threshExp is added to the count of the given bin by the increment circuit 7. If threshRange is 0 but the exponent of the input value is greater than the threshold exponent value threshExp, the comparison circuit does not output a signal to instruct the increment circuit to increment the count, and the count remains the same.

[0091] If threshRange is not equal to 0, in step S610, the bin verification circuit 706 checks whether threshRange is 0b1111, and a second special value in which all values ​​are greater than or equal to the threshold is added to the current bin count. In step S612, the exponent of the input value is compared with the threshold exponent by the comparison circuit, and if it is greater than or equal to the threshold exponent, the comparison circuit 708 outputs a signal to the increment circuit 710, and the value is added to the current bin count; otherwise, the count remains the same.

[0092] Finally, if threshRange does not have any of the above special values, verification is performed by the comparator circuit 708 to determine whether the input value exponent is within the defined exponent range of the current bin (in step S614). The comparator circuit checks whether the input value satisfies the following conditions: index ≥ threshExp and index < (threshExp + threshRange). If the exponent is within this range, the comparator circuit 708 outputs a signal to the increment circuit 710 and the count is increased by 1 for the current bin. Otherwise, the count for the current bin remains the same.

[0093] Although not shown in Figure 6, the loops for each value of the input vector are performed for each input value of the vector. While the loops for each bin and floating-point value of the vector have been described, there is no requirement that the bins and values ​​be processed sequentially or in any particular order. In fact, since the increment of a bin for a given value has no dependency on other bins and values, the comparison of the input value with the range of values ​​defined for each bin can be performed in parallel by the hardware. In parallel, two or more bins are processed simultaneously in relation to one or more input values. All that is required of all bins and values ​​is that the state information for each bin is checked to determine whether any particular case applies, and the state information of each bin is compared with each value of the input. Once the vector has been sufficiently processed with respect to a given bin of the histogram, the execution unit 18A proceeds to step S616, where it checks whether all bins have been processed according to the instructions. If not processed, the next bin is selected for processing, and the relevant steps S604-S614 are repeated for each input vector value with respect to the next histogram bin. Once all bins have been processed in this manner, the updated count array based on the input vector values ​​is set as the new BIN_COUNT value for each bin in the operand and written back to the original register of the first operand.

[0094] Note that the special values ​​threshRange=0 and threshRange=0b1111 enable the generation of cumulative histograms. When threshRange=0, all values ​​with an index equal to or lower than the threshold index of the current bin are counted in the current bin, thus accumulating "all values ​​in lower bins" + "values ​​within the range of the current bin". Conversely, when threshRange=0b1111, values ​​with an index greater than or equal to the threshold index of that bin are counted, meaning that the bin with the lowest threshold contains all values ​​within that bin + all values ​​within the range of higher bins.

[0095] The above example illustrates the processing of 32-bit vector values ​​by the execution unit 18A according to the 32-bit version of the hist instruction. However, this is not limiting, and the process described above can be applied to add any floating-point precision value to the histogram. In one exemplary embodiment, denormalized 32-bit floating-point numbers are treated as zero, while low-precision floating-point denormalized numbers may be counted as denormes separately from 0 by the hist instruction. When processing values ​​that may contain denormes, an additional step may be taken to allow counting of 0s or denormes only for a given bin. This is shown in Figure 7, where the step is identical to that in Figure 6, with the addition of step S605 for checking the value of threshExp.

[0096] In step S605, if threshExp has the special value 0b11111111, confirmation S607 is performed by the bin confirmation circuit 706 to determine whether threshRange is 0. If threshRange is 0, the count of the given bin is incremented only with respect to the value of 0, as determined in step S611, and the comparison circuit 708 checks whether the exponent and mantissa of the input value are 0, and if so, signals the increment circuit to increment the count of the current bin by 1. If threshRange takes any other value, in S609, confirmation is performed by the comparison circuit 708 to determine whether the input value is a denorm. This circuit checks whether the exponent is 0 and the mantissa is not 0, in which case the value is a denorm. If both the exponent and mantissa are 0 as described above, the value is 0, and this is not added to the denorm count in this special mode given by the special value of threshExp. If the value is a dinorm, a signal is sent to the increment circuit 710 to increment the count of a given bin by 1; otherwise, the count does not change.

[0097] If the threshold exponent can take any value other than 0b11111111, this is processed as the "default mode," and the processing then continues from step S606 as described above with respect to Figure 6. In some embodiments, as described above, the f32 denormalization number is processed as 0 and added to the count of bins whose range includes 0. With respect to the low-precision form, if the threshold exponent does not take the value 0b11111111 (i.e., the default case), the dinorm is not processed as 0, but since the bins are determined based only on the value of the exponent, the dinorm and 0 are added to the count of the same bins and are therefore processed in the same way.

[0098] When the instruction is executed as described above for the low-precision form, the execution unit 18 prepares the vector values ​​before processing by applying a function to the register data of the second operand to extract 16-bit values ​​from a set of 32-bit registers. For 16-bit floating-point numbers, the function "pickHalf," which takes 0 or 1 as an argument along with the register index, returns either the first or second half of the bit string at each register index as a separate 16-bit value. This can be done by both halves of a given 32-bit register to extract two 16-bit floating-point numbers from a 32-bit operand register. Similarly, for 8-bit floating-point values, the function "pickQuart" is done by 0, 1, 2, 3, and 4 respectively as arguments to extract various quarters of the 32-bit operand register data to be used as the 8-bit value of the vector. This conversion process allows 16-element f8 vectors and 8-element f16 vectors to be processed within a single instruction as defined above, with four 32-bit registers provided as the second operand. Other versions of the hist instruction may be defined by a different number of registers and / or registers of varying capacities specified by the second operand, and it should be noted that the above implementation is not limited to four 32-bit registers.

[0099] The execution of the above histogram instruction allows multiple values ​​to be added to a multi-binned histogram within a single machine code instruction, enabling faster and more efficient calculations compared to standard arithmetic instructions.

[0100] Collecting histograms of the data being processed is useful for generating statistics or cumulative metrics of the data being processed by a computer system. One exemplary application of histograms in the context of machine intelligence applications is automatic loss scaling of gradients when training neural networks. As briefly mentioned earlier, various floating-point formats can be used to represent values ​​in neural network training, and scaling factors can be used to reduce numerical underflow in the low-precision format. This works by scaling up the gradient by a certain factor, processing the scaled gradient in the low-precision format for more complex calculations such as matrix multiplication and convolution, and then rescaling back by the same factor if the complex calculation is complete and the high-precision format is used instead. One potential problem with loss scaling is that if the scaling factor is too large, the scaled gradient may increase beyond an upper threshold of representable values ​​available in a given format. Automatic loss scaling works by calculating the statistical distribution of the gradient to determine which parts of the gradient are above and / or below a given threshold, and increasing or decreasing the loss scaling factor in response to the determined parts. A histogram of the gradient can be generated to represent this statistical distribution of the gradient. In its simplest example, a histogram with two bins may be used, and the threshold index of the histogram defines an index close to the upper limit of the representable values ​​within a given numerical precision. This example is shown in Figure 8.

[0101] Figure 8 illustrates how the loss scaling factor affects gradient statistics and prevents numerical overflow. On the left side of Figure 8, a simplified distribution of gradient values ​​is shown. This distribution shape is illustrative and not intended to represent a realistic distribution of gradient values. A threshold T is shown above which a small number of gradients are located. The same distribution is represented in quantized form as a histogram 802 with two bins, where the first bin h1 gives the count of all gradients below threshold T, and the second bin h2 gives the gradients of all counts above threshold T. The hist instruction described above, performed for the pair of bins, can be used to generate the histogram 802 by processing the network's gradients as a set of vectors of size n, where n is determined based on the register capacity and the precision of the gradient representation. The total number of gradients to be processed can be very large, and the gradients can be loaded at once as a vector of n elements in the operand register for processing by the hist instruction.

[0102] Histogram 802 represents the number of values ​​within a defined range for each bin; therefore, the threshold exponent and threshold range bits do not take any special values, and the count is based on whether the exponent of the value falls within the exponent range defined for each bin of the histogram, which is predefined by the user.

[0103] The threshold index can be chosen as an index close to the maximum representable value within a given numerical representation. A useful statistic for avoiding overflow is to determine what percentage of the gradient exceeds a value close to the maximum representable value in a given numerical form. For example, with respect to FP16, the threshold can be chosen as 33762, which is half the maximum number representable by FP16.

[0104] The percentage of gradients that exceeds this threshold provides an indicator of the extent of overflow occurring within the network. This percentage can be directly obtained from the count determined by the hist instruction described above, for the entire set of gradients, using the following formula:

number

[0105] A minimum percentage f can be set and reduced at which the loss scaling factor is deemed too high and causes too many gradients to overflow. An example percentage f for FP16 is, for example, 10 -6 It can be selected as such. Note that the histogram in Figure 8 is not proportional to the actual size.

[0106] If it is determined that a proportion of the gradient greater than f is above the threshold T, the loss scaling coefficient may be reduced by a coefficient s. This has the effect of shifting the gradient distribution downward so that a small proportion of the gradient exceeds the threshold T when the gradient is scaled by this coefficient. If the loss scaling coefficient is below the threshold, either an algorithm that increases the loss scaling coefficient in each of multiple loss scaling coefficient update steps or an algorithm that updates the loss scaling coefficient only after many consecutive update steps where the proportion exceeding the threshold is less than the critical coefficient f can be applied.

[0107] A gradient histogram containing three or more bins can be calculated. In this case, the hist instruction simply defines a set of n bins as the first operand, where n=4 in the explanation of Figures 6 and 7 and n=2 in the exemplary application of Figure 8. If the gradient histogram contains three or more bins with bin edges {b_1, b_2, ..., b_(n-1)} and bin counts {h_1, h_2, ..., h_n}, after M consecutive optimal processes with respect to a given threshold T, the loss scaling coefficient L is increased only if the proportion of the total count of all bins whose edges are greater than or equal to the threshold T does not exceed a user-defined proportion f. That is,

number

[0108] As explained with reference to Figure 6, with respect to the count calculated by the hist instruction, this condition can be described as follows:

number

[0109] The loss scaling factor is reduced when the proportion of values ​​in a bin that exceed the threshold exceeds the critical ratio f.

[0110] This allows for the flexible use of low-precision forms to represent the gradients of neural networks, enabling more efficient computation, especially for computationally intensive operations such as matrix multiplication and convolution.

[0111] Histograms of values ​​can be useful within machine intelligence models and for any other application, and it should be noted that the histogram instruction is broadly applicable to determining the distribution of floating-point numbers for any given set. For example, the number of denorms in a given dataset may indicate the degree of underflow when processing the dataset, and this can also be used to indicate the numerical format used to represent the values ​​for this processing. In the extended version of loss scaling described above, both denorms (or values ​​below a predetermined lower threshold near the minimum representable value) and values ​​above a threshold near the maximum representable value may be monitored to most effectively tune the scaling coefficient and give the best possible trade-off between the ends of the dynamic range representable in a given format. For example, if most values ​​fall within a limited range in a higher-precision representation such as FP32, it may be determined that representing the data in a lower-precision format such as FP16 would be more efficient. Conversely, if there are many denormalized values ​​and many values ​​above a predetermined threshold near the upper end of the representable value, this indicates that the format used to represent the set of values ​​does not adequately cover the range of values ​​and a higher-precision representation may be required. This is merely one example of a range of applications for the collected statistics of processed data. Other variations or uses of the disclosed technology may become apparent to those skilled in the art once the disclosure herein is given. The scope of this disclosure is not limited by the embodiments described, but only by the appended claims.

Claims

1. A plurality of operand registers, wherein a first subset of the operand registers is configured to store state information for a plurality of bins, and for each of the plurality of bins, the state information includes a range of values ​​associated with the bin and a bin count, and a second subset of the operand registers is configured to store a vector of floating-point values, A first instruction is executed that takes the state information of the plurality of bins and the vector of floating-point values ​​as operands, and in response to the execution of the first instruction, for each of the floating-point values, Based on the exponent of the floating-point value, identify the bin among the plurality of bins in which the floating-point value falls within the range of the associated value. Increment the bin count associated with the bin identified for the floating-point value among the plurality of bins, An execution unit configured to perform the following: A processing device that includes this.

2. The processing device according to claim 1, wherein identifying a bin within the range of values ​​to which each floating-point value is associated comprises selecting each of the plurality of bins and, for each bin, using a comparison circuit, comparing the exponent of each floating-point value with a condition defining the range of values ​​associated with each bin.

3. The processing device according to claim 2, wherein the execution unit is configured to perform in parallel the determination of whether each of a plurality of floating-point values ​​is within the range of values ​​associated with each of a plurality of bins, in response to the execution of the first instruction.

4. For each of the bins, the state information includes a threshold index field. The processing device according to claim 1, wherein the execution unit is configured to identify the bin for each of the floating-point values ​​depending on the value of the threshold exponent field of the bin.

5. The state information for each bin includes a bin code indicator. The execution unit further includes a sign verification circuit that compares the sign of each floating-point value with the sign indicator of each bin, The processing device according to claim 2, wherein the range of values ​​associated with each bin includes only values ​​that match the sign of that bin.

6. The processing device according to claim 1, wherein the state information for each bin includes at least one mode indicator, and the execution unit includes a bin confirmation circuit configured to determine a range of values ​​associated with the bin based on the value of the mode indicator for the bin.

7. The processing device according to claim 1, wherein, in response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit is further configured to identify a bin among the plurality of bins that is within the range of values ​​associated with 0, and to increment the bin count associated with the identified bin.

8. The processing device according to claim 1, wherein, in response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit is further configured to identify a bin among the plurality of bins that falls within the range of the value to which the denormalized value is associated, and to increment the bin count associated with the identified bin.

9. The processing device according to claim 6, wherein the at least one mode indicator includes at least one of a threshold exponent field and a threshold range field.

10. The processing device according to claim 9, wherein when the at least one mode indicator indicates a default mode, the lower limit of the range of values ​​is the value of the threshold exponent field, and the upper limit of the range of values ​​is the sum of the value of the threshold exponent field and the value of the threshold range field.

11. The processing device according to claim 6, wherein when the mode indicator indicates a first special mode, the range of values ​​associated with the bin includes one of the ranges of 0 and a denormalized value.

12. The processing device according to claim 11, wherein the mode indicator includes a threshold index field, and the first special mode is indicated by a predefined special value of the threshold index field.

13. The processing device according to claim 6, wherein when the mode indicator indicates a second special mode, the range of values ​​associated with the bin includes all values ​​less than or equal to the threshold of the bin, and when the mode indicator indicates a third special mode, the range of values ​​associated with the bin includes all values ​​greater than or equal to the threshold of the bin.

14. The processing device according to claim 13, wherein the mode indicator includes a threshold range field, and the second special mode and the third special mode are each indicated by a special value of the threshold range field.

15. The processing device according to claim 11, wherein the mode indicator further includes a threshold range field, and when the threshold range value takes a special value of 0, the range of values ​​associated with the bin includes only 0, and when the threshold range field is not 0, the range of values ​​associated with the bin includes the range of the denormalized value.

16. The processing device according to claim 1, wherein the execution unit is configured to process the gradient of a machine intelligence application scaled by a scaling factor, the first instruction is executed by the execution unit on a plurality of vectors of gradients to generate a histogram containing a plurality of bins, and the loss scaling factor is calculated as the ratio of the count of bins that exceeds a predetermined threshold to the sum of the counts of all bins, and increases or decreases according to the ratio of the gradient that exceeds the predetermined threshold.

17. The processing device according to claim 1, wherein the first subset and the second subset of the operand registers are registers in an arithmetic register file.

18. The aforementioned floating-point value is, 32-bit representation, 16-bit representation, or 8-bit representation The processing device according to claim 1, provided as such.

19. The processing device according to claim 1, wherein the first subset and the second subset each include four 32-bit registers.

20. A computer program comprising code configured to run on a processing device according to any one of claims 1 to 19, wherein the code comprises one or more instances of an instruction whose operands are state information including the bin count of each of at least a plurality of bins and a vector of floating-point values, When the aforementioned code is executed, the processing device will, for each of the floating-point values, Based on the exponent of the floating-point value, identify the bin among the plurality of bins in which the floating-point value falls within the range of the associated value. Increment the bin count associated with the bin identified for the floating-point value among the plurality of bins, A computer program that executes something.

21. A method for operating a processing device configured according to any one of claims 1 to 19, Execute a first instruction that takes state information including the bin count of each of the multiple bins and a vector of floating-point values ​​as operands, The execution unit of the processing device, in response to the execution of the first instruction, performs the following for each of the floating-point values: The execution unit identifies, based on the exponent of the floating-point value, one of the plurality of bins in which the floating-point value falls within the range of values ​​associated with that floating-point value. The execution unit increments the bin count associated with the bin identified for the floating-point value among the plurality of bins, A method that includes this.

Images