Machine Code Instructions

Abstract

1. A processing device comprising: a plurality of operand registers, a first subset of the operand registers configured to store state information for a plurality of bins including a bin count and a range of values ​​associated with each bin, and a second subset of the operand registers configured to store a vector of floating-point values; and an execution unit configured to execute a first instruction taking as operands the state information for the plurality of bins and the vector of floating-point values, and to perform, in response to execution of the first instruction, the following: for each floating-point value, identify a bin among the plurality of bins within which the floating-point value falls in an associated value range based on an exponent of the floating-point value; and increment a bin count associated with the identified bin.

Description

[Technical field]

[0001] The present disclosure relates to machine code instructions for processing vectors. [Background technology]

[0002] There has been an increasing interest in developing processors designed for specific applications, such as graphic processing units (GPUs) and digital signal processors (DSPs). Another type of application-specific processor that has recently gained interest is specialized for machine intelligence applications, referred to by the applicant as "IPUs" (intelligent processing units). These may be employed, for example, as accelerator processors arranged to perform tasks assigned to them by a host, such as training or assisting in the training of knowledge models, such as neural networks, or making or assisting in the performance of predictions or inferences based on such models.

[0003] A processor designed for machine intelligence applications may include dedicated instructions within its instruction set (an instruction set is a base set of machine code instruction types that the processor's execution unit is configured to recognize, each type defined by a respective opcode and zero or more operands) for performing arithmetic operations commonly employed in machine intelligence applications.

[0004] Machine intelligence models are often trained on large amounts of data, which can be expensive in terms of computational resources. To reduce the memory usage required to process large amounts of data, recent methods using low-precision representations of the data and / or model parameters have been developed to improve the efficiency of such training. Recent training methodologies for neural networks use mixed-precision training, where some values ​​of the network have high-precision representations, such as 32-bit floating point, while other values ​​are represented in a low-precision format, such as 16-bit floating point. The low-precision format has a narrower range of representation and is therefore more vulnerable to numerical underflow and overflow. One method used to overcome this challenge is known as "loss scaling", where the loss function used in training the neural network model is scaled to prevent the gradient of the loss function from falling below a threshold of representable values ​​for the selected low-precision format. Summary of the Invention [Problem to be solved by the invention]

[0005] Histograms are used to approximate the distribution of various types of values. Floating-point numbers processed by a computer may be added to a histogram to generate approximate statistics for a set of values ​​processed within a computer program. As described in more detail herein, this may be useful for collecting data related to the distribution of values ​​of a given floating-point representation, such as to identify the degree of underflow and / or overflow occurring when processing a given data set, which may be reduced by adjusting the representation of the values ​​(e.g., adjusting the skew). [Means for solving the problem]

[0006] Described herein are machine instructions defined within a processor's instruction set architecture, configured to assign floating-point values ​​of a vector to bins of a histogram, the machine instructions being defined for multiple numeric representations including 8-bit, 16-bit and 32-bit floating-point formats. This allows a histogram of a set of values ​​having floating-point representations to be collected that can be used to gain insight into the distribution of data and allow underflows and overflows to be reduced. A particular advantage of the instructions described herein is that they allow the generation and updating of the histogram with fewer instructions than is allowed by standard arithmetic instructions, thus allowing the collection of such statistics with low overhead and high code density.

[0007] A first aspect disclosed herein provides a processing device, the processing device including: a plurality of operand registers, a first subset of the operand registers configured to store state information for a plurality of bins, where for each of the plurality of bins, the state information includes a range of values ​​and a bin count associated with the bin, and a second subset of the operand registers configured to store a vector of floating-point values; and an execution unit configured to execute a first instruction taking as operands the state information for the plurality of bins and the vector of floating-point values, and in response to executing the first instruction, identify, for each floating-point value, a bin among the plurality of bins within which the floating-point value is within an associated range of values ​​based on an exponent of the floating-point value; and increment a bin count associated with the bin among the plurality of bins identified for the floating-point value.

[0008] Identifying a bin within the range of values ​​associated with each floating-point value may include selecting each bin of a plurality of bins and, for each bin, comparing, using a comparison circuit, an exponent of each floating-point value to a condition defining a range of values ​​associated with each bin.

[0009] The execution unit may be configured to perform in parallel, in response to execution of the first instruction, determining whether each of the plurality of floating-point values ​​falls within a range of values ​​associated with each of the plurality of bins.

[0010] For each of the bins, the state information may include a threshold exponent field, and the execution unit is configured to identify a bin for each of the floating-point values ​​in dependence on the value of the bin's threshold exponent field.

[0011] A threshold bin count saturation value may be defined. The execution unit is configured to, in response to execution of the first instruction, perform the following: for each bin, compare the bin count to the threshold bin count saturation value; and identify, of a non-saturated subset of the bins, bins within a range of values ​​associated with each of the floating-point values. The non-saturated subset of the bins includes bins whose bin counts are less than the threshold bin count saturation value.

[0012] The state information for each bin may include a sign indicator for the bin. The execution unit further includes a sign checking circuit for comparing the sign of each floating-point value with the sign indicator for each bin. The range of values ​​associated with each bin includes only values ​​that match the sign of that bin.

[0013] The state information for each bin may include at least one mode indicator, and the execution unit includes a bin validation circuit configured to identify a range of values ​​associated with each bin based on a value of the mode indicator for that bin.

[0014] In response to execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit may be further configured to identify a bin of the plurality of bins that is within a range of values ​​associated with 0 and increment a bin count associated with the identified bin. Alternatively, in response to execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit may be further configured to identify a bin of the plurality of bins that is within a range of values ​​associated with the denormalized value and increment a bin count associated with the identified bin.

[0015] The at least one mode indicator may include at least one of a threshold exponent field and a threshold range field.

[0016] If at least one mode indicator indicates the default mode, the lower limit of the value range may be the value of the threshold exponent field, and the upper limit of the value range may be the sum of the value of the threshold exponent field and the value of the threshold range field.

[0017] If the mode indicator indicates a first special mode, the range of values ​​associated with the bin may include one of: (i) 0 and (ii) a range of unnormalized values.

[0018] The mode indicator may include a threshold exponent field, where a first special mode is indicated by a predefined special value of the threshold exponent field.

[0019] If the mode indicator indicates a second special mode, the range of values ​​associated with a bin may include all values ​​below the threshold of that bin, and if the mode indicator indicates a third special mode, the associated range of that bin includes all values ​​above the threshold of that bin.

[0020] The mode indicator may include a threshold range field, where the second special mode and the third special mode are each indicated by a special value of the threshold range field.

[0021] The mode indicator may further include a threshold range field. If the threshold range value takes the special value of 0, then the range of values ​​associated with the bin includes only 0. If the threshold range field is not 0, then the range of values ​​associated with the bin includes the range of denormalized values.

[0022] The execution unit can be configured to process gradients of the machine intelligence application scaled by a scaling factor. A first instruction is executed by the execution unit on a plurality of vectors of gradients to generate a histogram including a plurality of bins, and a loss scaling factor is adjusted relative to a total count of all the bins based on a relative count of a predetermined set of the plurality of bins.

[0023] The first and second subsets of the operand registers may be registers of an arithmetic register file.

[0024] Floating-point values ​​may be provided as (i) a 32-bit representation, (ii) a 16-bit representation, or (iii) an 8-bit representation.

[0025] The first and second subsets of registers may each include four 32-bit registers.

[0026] A second aspect disclosed herein provides a computer program comprising code configured to execute on a processing device of any preceding claim, the code comprising one or more instances of an instruction taking as operands state information comprising at least a bin count for each of a plurality of bins and a vector of floating point values. The code, when executed, causes the processor to: identifying a bin among a plurality of bins in which the floating point value is in an associated range of values ​​based on an exponent of the floating point value; incrementing a bin count associated with a bin among the plurality of bins identified for the floating-point value.

[0027] Another aspect disclosed herein provides a method of operating any of the processing devices disclosed herein, the method including executing a first instruction taking as operands state information including a bin count for each of a plurality of bins and a vector of floating-point values, and in response to executing the first instruction, for each of the floating-point values: identifying a bin among a plurality of bins for which the floating point value is within an associated range of values ​​based on an exponent of the floating point value; and incrementing a bin count associated with a bin identified for the floating-point value among the plurality of bins.

[0028] Identifying a bin for each floating-point value includes selecting a respective bin of the plurality of bins, and for each bin, selecting each of the floating-point values ​​of the vector, and comparing an exponent of each floating-point value to a condition defining a range of values ​​associated with the bin.

[0029] The method may include processing multiple combinations of bins and floating-point values ​​in parallel.

[0030] The state information may further include a threshold exponent field.The method may further include identifying, for each of the floating-point values, one of a plurality of bins depending on a value of the threshold exponent field for each of the plurality of bins.

[0031] A threshold bin count saturation value may be defined. The method further includes, for each bin, comparing the bin count to the threshold bin count saturation value and identifying a non-saturated subset of the bins whose respective floating-point values ​​are within an associated range of values, the non-saturated subset of the bins including bins whose bin counts are less than the threshold bin count saturation value.

[0032] The state information for each bin may include a sign indicator for the bin. The method further includes comparing the sign of each floating-point value to the sign indicator for each bin. The range of values ​​associated with each bin includes only values ​​that match the sign of the bin.

[0033] The state information for each bin may include at least one mode indicator. The method further includes performing bin validation to identify a range of values ​​associated with each bin based on the value of the bin's mode indicator. The mode indicator may include one or both of a threshold exponent field and a threshold range field.

[0034] The method may include, if each floating-point value is a non-normalized floating-point value, identifying a bin of the plurality of bins that is within a range of values ​​associated with 0 and incrementing a bin count associated with the identified bin of the plurality of bins. Alternatively, if each floating-point value is a non-normalized floating-point value, the method may include, if each floating-point value is a non-normalized floating-point value, identifying a bin of the plurality of bins that is within a range of values ​​associated with the non-normalized value and incrementing a bin count associated with the identified bin of the plurality of bins.

[0035] If at least one mode indicator indicates a default mode, the lower limit of the range of values ​​may be the value of the threshold exponent field. The upper limit of the range of values ​​is the sum of the value of the threshold exponent field and the value of the threshold range field. If at least one mode indicator indicates a first special mode, the range of values ​​associated with the bin may include one of 0 and a range of non-normalized values. The first special mode may be indicated by a predefined special value of the threshold exponent field. In the first special mode, if the threshold range value takes the special value of 0, the range of values ​​associated with the bin may include only 0. If the threshold range field is not 0, the range of values ​​associated with the bin may include the range of non-normalized values.

[0036] If the mode indicator indicates a second special mode, the range of values ​​associated with a bin may include all values ​​less than or equal to the bin threshold, and if the mode indicator indicates a third special mode, the range of values ​​associated with a bin may include all values ​​greater than or equal to the bin threshold.

[0037] The vector of values ​​may include a gradient of the machine intelligence algorithm. The method may further include scaling the gradient of the machine intelligence algorithm by a scaling factor, the scaling factor being adjusted based on the relative count of a predetermined subset of the plurality of bins to a total count of all bins.

[0038] Another aspect disclosed herein provides a non-transitory computer readable storage medium comprising code configured to execute on a processing device as disclosed herein, the code comprising one or more instances of an instruction that takes as operands state information comprising at least a bin count for each of a plurality of bins and a vector of floating-point values, and that, when executed, causes a processor to: identifying a bin among a plurality of bins with which the floating point value is within an associated range of values ​​based on an exponent of the floating point value; incrementing a bin count associated with a bin identified for the floating-point value among a plurality of bins; Execute. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 is a schematic block diagram of an exemplary multi-threaded processor. [Diagram 2] 1 illustrates a schematic diagram of a logical block structure of an exemplary processor. [Diagram 3] 1 is a schematic block diagram of a processor including an array of constituent processors. [Figure 4] FIG. 1 is a schematic diagram of a graph used in a machine intelligence algorithm. [Diagram 5] FIG. 2 is a schematic block diagram of logic circuitry for implementing the hist instruction. [Figure 6] 11 is a flow diagram showing the logic of the HIST instruction. [Figure 7] 11 is a flow diagram showing the logic of a second HIST instruction. [Figure 8]13 illustrates generally how a histogram generated by a hist command is used to adjust a scale factor for training a machine intelligence model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] Figure 1 illustrates an example of a processor 4 according to an embodiment of the present disclosure. The architectures of Figures 1 and 2 are example architectures in which the present invention may be implemented. As will be apparent to those skilled in the art of computer architecture, the present invention may be implemented in a variety of processors having a variety of architectures. This architecture is described in further detail in U.S. Patent Application Publication No. 16 / 276,834, which is incorporated herein by reference in its entirety.

[0041] The processor 4 includes a multithreaded processing unit 10 in the form of a barrel thread processing unit, and a local memory 11 (i.e., on the same tile in the case of a multi-tile array, or on the same chip in the case of a single processor chip). A barrel thread processing unit is a type of multithreaded processing unit in which the execution time of the pipeline is divided into a repeating sequence of interleaved time slots, each of which may be occupied by a given thread. This may also be referred to as parallel execution. The memory 11 includes an instruction memory 12 and a data memory 22 (which may be implemented in different addressable memory units or in different regions of the same addressable memory unit). The instruction memory 12 stores machine code that is executed by the processing unit, while the data memory 22 stores both data that is operated on by the executable code and data that is output by the executable code (e.g., as a result of such manipulation).

[0042] Memory 12 stores multiple, various threads of a program. Each thread includes a sequence of instructions for performing one or more specific tasks. Note that an instruction as referred to herein means a machine code instruction consisting of a single opcode and zero or more operands, i.e., one instance of a basic instruction of the processor's instruction set. In an embodiment, the program includes multiple worker threads and a supervisor subprogram that may be structured as one or more supervisor threads.

[0043] Execution pipeline 13 includes a fetch stage 14, a decode stage 16, and an execution stage 18 that includes execution units that may perform arithmetic and logical operations, address calculations, load and store operations, and other operations as defined by the instruction set architecture.

[0044] The dedicated hardware includes a separate set of context registers 26 for at least each of the threads that may be executed simultaneously, i.e., one set per slot in a cycle. "Context" in discussing a multi-threaded processor refers to one program state (e.g., program counter value, state, and current operand values) of each of the threads that execute in parallel with one another. Context registers refer to the respective registers for representing this program state of the respective thread. Each set of context registers 26 includes one or more control registers for each of the threads, including at least a program counter (PC) (to keep track of the instruction address that the thread is currently executing) and in some embodiments also a set of one or more control state registers (CSRs) that record the current state of each of the threads (e.g., whether it is currently executing or paused). Each set of context register files 26 also includes a respective set of operand registers for temporarily holding the operands of the instructions executed by the thread, i.e., values ​​that, when executed, are operated on or result from values ​​defined by the opcodes of the instructions of the thread. Each set of registers 26 may be implemented in one or more register files. Note that while "operand" strictly refers to the portion of an instruction that specifies the data to be operated on, it is used more generally herein to refer to both the register indexes specified in an instruction and the data held in those registers.

[0045] Fetch stage 14 has access to a program counter (PC) for each of the contexts. For each respective thread, fetch stage 14 fetches the next instruction for that thread from the next address in program memory 12 indicated by the program counter. The program counter automatically increments each execution cycle unless a branch instruction is taken. Fetch stage 14 then passes the fetched instruction to decode stage 16 where it is decoded, which then passes a designator of the decoded instruction, along with the decoded addresses of any operand registers specified in the instruction, to execution unit 18 so that the instruction is executed. In the case of an arithmetic instruction, for example, execution unit 18 has access to operand registers and control state registers that may be used in executing the instruction based on the decoded register addresses (e.g., by adding, multiplying, subtracting or dividing values ​​in two operand registers and outputting the result to another operand register for the respective thread). Alternatively, if the instruction defines a memory access (load or store), the load / store logic of the execution unit 18 follows the instruction to load a value from the data memory into the respective thread's operand register, or store a value from the respective thread's operand register into the data memory 22.

[0046] Fetch stage 14 is connected to fetch instructions from instruction memory 12 for execution under the direction of scheduler 24. Scheduler 24 is configured to control fetch stage 14 to fetch instructions from each of a set of concurrently executing threads in turn within a repeating sequence of time slots, and thus divide the resources of pipeline 13 into a number of temporally interleaved time slots.

[0047] FIG. 2 shows details of the multi-threaded processor 4, including details of the execution units 18 and context registers 26.

[0048] The processor includes a respective instruction buffer 53 for each of the M threads that may be executed simultaneously. The context registers 26 include a respective main register file (MRF) 26M for each of the worker M context and the supervisor context. The context registers further include at least each auxiliary register file (ARF) 26A for the worker context. The context registers 26 further include a common weight register file (WRF) 26W that all currently executing worker threads may access for reading. The WRF may be associated with the supervisor context in that the supervisor thread is the only thread that may write to the WRF. The context registers 26 may also include a respective group of control status registers 26CSR for each of the supervisor context and the worker context. The execution units 18 include a main execution unit 18M and an auxiliary execution unit 18A. The main execution unit 18M includes a load / store unit (LSU) 55 and an integer arithmetic logic unit (IALU) 56. The auxiliary execution unit 18A includes at least a floating point unit (FPU).

[0049] In each of the J interleaved time slots S0...SJ-1, the scheduler 24 controls the fetch stage 14 to fetch at least one instruction of a respective thread from the instruction memory 11 into a respective one of the J instruction buffers 53 corresponding to the current time slot. In an embodiment, each time slot is one execution cycle of the processor, although other schemes (e.g., weighted round robin) are not excluded. In each execution cycle of the processor 4 (i.e., each cycle of the processor clock that times the program counter), the fetch stage 14 fetches either a single instruction or a small "instruction bundle" (e.g., a two-instruction bundle or a four-instruction bundle), depending on the implementation. Each instruction is then issued via the decode stage 16 to the LSU 55 or IALU 56 of the main execution unit 18M or to one of the FPUs of the auxiliary execution unit 18A, depending on whether the instruction is a memory access instruction, an integer arithmetic instruction, or a floating-point arithmetic instruction (according to its opcode). The LSU 55 and IALU 56 of the main execution unit 18M execute their instructions by using registers from the MRF 26M, with the particular register in the MRF 26M being specified by the operands of the instruction. The FPU of the auxiliary execution unit 18A performs operations by using registers in the ARF 26A and WRF 26W, with the particular register in the ARF being specified by the operands of the instruction. In an embodiment, the registers in the WRF may be implicit in terms of the instruction type (i.e., may be predetermined for that instruction type). The auxiliary execution unit 18A may also include circuitry in the form of logical latches internal to the auxiliary execution unit 18A for holding some internal state 57 for use in performing one or more operations of some types of floating-point arithmetic instructions.

[0050] In embodiments where instructions are fetched and executed in bundles, the individual instructions in a given instruction bundle are executed simultaneously in parallel down independent pipelines 18M, 18A (as shown in FIG. 3). In embodiments where two bundles of instructions are executed, the two instructions may be executed simultaneously across their respective auxiliary and main pipelines. In this case, the main pipeline is arranged to execute some types of instructions that use the MRF, and the auxiliary pipeline is used to execute some types of instructions that use the ARF. The pairing of instructions into suitable complementary bundles may be handled by the compiler.

[0051] Each worker thread context has its own instance of a main register file (MRF) 26M and an auxiliary register file (ARF) 26A (i.e., one MRF and one ARF for each barrel thread slot). It is understood that the functionality described herein with respect to the MRF or ARF operates on a per-context basis. However, there is a single shared weight register file (WRF) that is shared between threads. Each thread may access the MRF and ARF of only its own context 26. However, all currently executing worker threads may access a common WRF. The WRF thus provides a common set of weights for use by all worker threads. In an embodiment, only the supervisor may write to the WRF and workers may only read from the WRF.

[0052] The processor 4 instruction set includes at least one type of load instruction whose opcode, when executed, causes the LSU 55 to load data from data memory 22 into the respective ARF 26A of the thread in which the load instruction was executed. A destination location within the ARF is specified by an operand of the load instruction. Another operand of the load instruction specifies an address register within the respective MRF 26M that holds a pointer to an address within data memory 22 from which the data is to be loaded. The processor 4 instruction set also includes at least one type of store instruction whose opcode, when executed, causes the LSU 55 to store data from the respective ARF of the thread in which the store instruction was executed into data memory 22. A source location of the store within the ARF is specified by an operand of the store instruction. Another operand of the store instruction specifies an address register within the MRF that holds a pointer to an address within data memory 22 to which the data is to be stored. In general, the instruction set may include separate load and store instruction types and / or at least one load / store instruction type that combines a load operation and a store operation into a single instruction.

[0053] The processor's instruction set also includes one or more types of arithmetic instructions for performing arithmetic operations. According to embodiments disclosed herein, these may include at least one type of arithmetic instruction that utilizes a common weight register file WRF 26W. This type of instruction takes at least one operand that specifies at least one source of the corresponding arithmetic operation in the respective ARF 26A of the thread in which the arithmetic instruction is executed. However, at least one other source of the arithmetic instruction is in a common WRF common to all worker threads. In embodiments, this source is implicit in the arithmetic instruction (i.e., implicit for this type of arithmetic instruction). In terms of machine code instructions, implicit means that it does not require specifying an operand. That is, in this case, the location of the source in the WRF is specific to the opcode (predefined for that particular opcode). Alternatively, in other embodiments, the arithmetic instruction may take an operand that specifies from which set of weight registers to take the weight from among several different sets in the WRF. However, the fact that the source of the weights is found within a WRF (as opposed to, for example, a generalized MRF or ARF) is still implicit.

[0054] In response to the opcode of the associated type of arithmetic instruction, an arithmetic unit (e.g., an FPU) in auxiliary execution unit 18A performs the arithmetic operation specified by the opcode, which includes operating on values ​​in specified source registers in the thread's respective ARF and values ​​in source registers in the WRF. The arithmetic unit also outputs the result of the arithmetic operation to a destination register in the thread's respective ARF that is explicitly specified by the destination operand of the arithmetic instruction.

[0055] Exemplary types of arithmetic instructions that may employ sources within a common WRF 26W include one or more vector multiplication instruction types, one or more matrix multiplication instruction types, one or more accumulate vector multiplication instruction types and / or accumulate matrix multiplication instruction types (which accumulate the result of a multiplication from one instance of an instruction into the next instance) and / or one or more convolution instruction types. For example, a vector multiplication instruction type may multiply an explicit input vector from the ARF 26A with a predefined vector of weights from the WRF, or a matrix multiplication instruction type may multiply an explicit input vector from the ARF with a predefined matrix of weights from the WRF. As another example, a convolution instruction type may convolve an input matrix from the ARF with a predefined matrix from the WRF. Having a shared weight register file and WRF common to multiple threads allows each thread to multiply or convolve a common kernel with its own respective data. This is useful, for example, as it is a scenario that arises often in machine learning applications where each thread represents a different node in a neural network and a common kernel represents a feature to be searched for or trained on (e.g., an edge or a particular shape within an area or volume of graphical data).

[0056] In an embodiment, values ​​in WRF 26W may be written by a supervisor thread. The supervisor (which in an embodiment begins by running in all slots S0...SM) first executes a series of put instructions to write some common weighting values ​​to predetermined locations in the WRF. The supervisor then executes execute (or run-all) instructions to launch each worker in some or all of slots S0...SJ-1. Each worker then contains one or more instances of one or more arithmetic instructions of the type discussed above to perform corresponding arithmetic operations on its own respective input data loaded into its respective ARF 26A, but by using the common weightings written into WRF 26W by the supervisor. Upon finishing its respective task, each thread executes a quit instruction to return its slot to the supervisor. Once all launched threads have finished their respective tasks, the supervisor may write the new values ​​to the WRF and launch a new set of threads (or may launch a new set to continue using the existing values ​​in the WRF).

[0057] It is recognized that the labels "primary", "auxiliary" and "weighting" are not necessarily limiting. In an embodiment, these may be any first register file (per worker context), second register file (per worker context) and a shared third register file (e.g., a portion that is part of the supervisor context but accessible to all workers). The ARF 26A and auxiliary execution unit 18 may also be referred to as an arithmetic register file and arithmetic execution unit since they are used for arithmetic instructions (or at least floating point operations). The MRF 26M and auxiliary execution unit 18 may also be referred to as a memory address register file and arithmetic execution unit since one of their uses is to access memory. The weight register file (WRF) 26W is so referred to because it is used to hold multiplication weights used in one or more particular types of arithmetic instructions, discussed in more detail immediately below. For example, they may be used to represent weights of nodes in a neural network. Viewed another way, the MRF may be referred to as an integer register file since it is used to hold integer operands, while the ARF may be referred to as a floating point register file since it is used to hold floating point operands. In an embodiment that executes instructions in bundles of two, the MRF is the register file used by the main pipeline and the ARF is the register file used by the auxiliary pipeline.

[0058] However, it should be noted that in alternative embodiments, register space 26 is not necessarily divided into these separate register files for these various purposes. Instead, instructions executed via the main and auxiliary execution units may be able to define some registers from the same shared register file (one register file per context in the case of a multi-threaded processor). Pipeline 13 also does not necessarily have to have parallel constituent pipelines (e.g., auxiliary and main pipelines) for simultaneously executing bundles of instructions.

[0059] The processor 4 may also include an exchange interface 51 for exchanging data between the memory 11 and one or more other resources, e.g., other instances of the processor and / or other instances of external devices, such as a network interface or a network attached storage (NAS) device. As shown in FIG. 3, in an embodiment, the processor 4 may form one of an array 6 of interconnected processor tiles, each tile executing a part of a broader program. Thus, the individual processors 4 (tiles) form part of a broader processor or processing system 6. The tiles 4 may be connected together via an interconnection subsystem 34 that connects via respective exchange interfaces 51. The tiles 4 may be implemented on the same chip (i.e., die) or different chips, or a combination thereof (i.e., an array may be formed from multiple chips each including multiple tiles 4). Thus, the interconnection system 34 and the exchange interface 51 may include an internal (on-chip) interconnect mechanism and / or an external (inter-chip) exchange mechanism accordingly.

[0060] In one exemplary application of a multi-threaded and / or multi-tile processor or system, program execution across multiple threads and / or tiles 4 includes machine intelligence algorithms, such as algorithms configured to train a neural network and / or perform inference based on a neural network. In such an embodiment, each worker thread, or a portion of the program execution on each tile, or each worker thread on each tile, is used to represent a different node 102 in a neural network (a type of graph), and correspondingly, communications between threads and / or tiles represent edges 104 between the nodes 102 in the graph. This is illustrated in FIG.

[0061] Machine intelligence begins with a learning stage where a machine intelligence algorithm learns a knowledge model. The model includes a graph of interconnected nodes (i.e., vertices) 102 and edges (i.e., links) 104. Each node 102 in the graph has one or more input edges and one or more output edges. Some of the input edges of some of the nodes 102 are output edges of some of the nodes 102, thereby connecting the nodes together to form a graph. Furthermore, one or more of the input edges of one or more of the nodes 102 collectively form the inputs to the graph, and one or more of the output edges of one or more of the nodes 102 collectively form the outputs of the graph. Sometimes a given node may also have all of the inputs to the graph, outputs from the graph, and connections to other nodes. Each edge 104 carries a value or, more frequently, a tensor (an n-dimensional matrix), which forms the inputs and outputs provided to and from the node 102 on the input and output edges, respectively.

[0062] Each node 102 represents a function of one or more inputs received on its input edges, and the result of this function is an output provided on its output edges. Each function is parameterized by one or more respective parameters (sometimes called weights, but not necessarily multiplicative weights). In general, the functions represented by the various nodes 102 may be functions of different types and / or may be parameterized by different parameters.

[0063] In the learning stage, the algorithm receives empirical data, i.e., multiple data points representing various possible combinations of inputs to the graph. As more empirical data is received, the algorithm gradually tunes the parameters of the various nodes 102 in the graph based on the empirical data in order to try to minimize the error of the parameters. The goal is to find values ​​of the parameters such that the output of the graph is as close as possible to the desired output for the given inputs. When the graph as a whole tends towards such a state, the graph is said to converge. After a suitable degree of convergence, the graph can then be used to make predictions or inferences, i.e., to predict the outcome of a given input or to infer the cause of a given output.

[0064] The learning stage can take many different possible forms. For example, in a supervised approach, the input experience data takes the form of training data, i.e., inputs that correspond to known outputs. With each data point, the algorithm can tune the parameters so that the output matches more closely to the known output for the given input. In a subsequent prediction stage, the graph can then be used to map the input query to an approximate predicted output (and vice versa when making inference). Other approaches are possible. For example, in an unsupervised approach, there is no notion of a baseline outcome per input data, and instead the machine intelligence algorithm is left to identify its own structure in the output data. Or, in a reinforcement approach, the algorithm tries at least one possible output for each data point in the input experience data, and is told whether this output is positive or negative (and the extent to which it is potentially positive or negative), e.g., win or lose, or reward or cost, etc. Over many attempts, the algorithm can gradually tune the parameters of the graph so that it can predict inputs that will produce positive outcomes. Various techniques and algorithms for learning graphs will be known to those skilled in the art of machine learning.

[0065] One common algorithm used in training machine intelligence models is gradient descent, which involves a neural network with the parameters of the nodes 102 being tuned as described above against empirical data. In gradient descent, this tuning is done by defining a loss or error function and updating the parameters of the model in the negative direction of the gradient of the loss function, in other words, updating the parameters to minimize the defined loss function. Several types of gradient descent algorithms have been developed and will be familiar to those skilled in the art of machine learning.

[0066] When training a machine intelligence model based on empirical data, depending on the task, it may be necessary to process very large amounts of data at once. One way to improve the efficiency of processing is to employ multi-threading and concurrency as described above, which allows pipeline latencies to be hidden so that multiple threads run simultaneously. Another way to improve the speed and efficiency of processing is to change the format of the data being processed so that more individual data elements can be processed simultaneously and more data elements can be stored in a given amount of memory at any given time during processing, so that each of the data values ​​of the network is stored using less memory.

[0067] Typically, neural network parameters and values, as well as gradients, are stored in floating-point format. A "single-precision" floating-point format value has 32 bits, which may be referred to as a 32-bit floating-point format or float32, while "double-precision" numbers or "doubles" are represented in 64 bits. Similarly, 16-bit floating-point numbers may be referred to as "half-precision" and 8-bit floating-point numbers, although less widely used, may be referred to as "quarter-precision".

[0068] A floating-point representation contains three separate components: a sign component, a mantissa component, and an exponent component. In single-precision (i.e., 32-bit) floating-point representation according to the IEEE 754 standard, the sign component consists of a single bit, the exponent consists of 8 bits, and the mantissa consists of 23 bits. In standard 1 / 2-precision (i.e., 16-bit) floating-point representation, the sign component consists of a single bit, the mantissa consists of 10 bits, and the exponent consists of 5 bits. In most cases, a number is given by these three components by the following formula:

number

[0069] The "offset" displayed for the exponent depends on the number of bits used to represent the exponent, which in turn depends on the precision level. In single precision representations, the offset is usually equal to 127. In half precision formats, the offset is usually equal to 15. However, it should be noted that a different number of exponent bits may be defined for a given precision or total number of bits in the floating-point representation, and bit allocations outside the above mentioned standard are possible.

[0070] "I" is the implied bit derived from the exponent. If the exponent bit string consists of anything other than all zeros or all ones, then the implied bit is equal to 1 and the number is known as the "norm". In this case, the floating-point number is given by the formula:

number

[0071] If the exponent bit string consists of all zeros, then the implied bit is equal to 0 and the number is a denormalized number, also referred to herein as a "dinorm," "denormal," or "denormalized number." The terms "denormalized" and "dinorm" or "denormalized number" may be used interchangeably herein to refer to numbers between the minimum norm and 0. In this case, the floating-point number is given by the following formula:

number

[0072] Denorms are useful to allow smaller numbers to be represented than would otherwise be representable with a limited number of exponent bits.

[0073] When a number is too small to be represented with the number of mantissa and exponent bits of a given floating-point representation, it is called an underflow. Methods for reducing underflow when training machine intelligence models are described later. Non-zero values ​​below a threshold of representable values ​​can be treated as "denormalized" numbers, which are floating-point values ​​that satisfy the underflow gap between the lowest "normal" representable number and zero. As described below, it is possible to represent denormalized numbers by using an implied bit based on the value of the exponent bits.

[0074] According to an embodiment of the present invention, execution unit 18 is configured to execute novel machine code instructions. Histogram instructions, also referred to herein as "hist instructions," take two operands: a first operand defines a set of multi-bins with a range of values ​​that each bin spans, as well as a count of the values ​​in each bin, and a second operand defines a vector of values. The hist instruction is executed to determine which bin in the histogram each value falls into, and to increment that bin's histogram count by one for each value that falls within the range.

[0075] As understood in the field of computer processor architecture, a machine code instruction is expressed as an opcode that defines the operation of the instruction and as one or more operands, including the data to which the operation is applied. A hist instruction, as described in detail below, takes as operands (a) a register index to a set of registers that contains state information for a set of bins, (b) state information including bin ends and counts, and (c) a vector of values ​​to be added to the histogram, and adds any value that falls within the limits of each bin to the count of that bin. A hist instruction may be defined in more than one numeric format, for example, a hist instruction may be defined for vectors of 16-bit and 8-bit floating point numbers as well as 32-bit floating point numbers. Both the histogram bin state information and the vector of values ​​are stored in respective registers defined in the ARF 26A.

[0076] The hist instructions may apply to vectors of values ​​having 32-bit, 16-bit or 8-bit representations, with each format having a different instruction opcode since the function of the instruction differs slightly from one format to the next, as explained in more detail below. The operands of the hist instructions, which specify the vector of values ​​to be binned, are sets of register indices holding a defined number of bits. In other words, if a given register or set of registers referenced in an operand holds a vector of four 32-bit values ​​totaling 128 bins, one or more registers may hold a vector of twice as many values, i.e., eight values, if the values ​​being processed are instead represented in 16 bits. For 8-bit values, the same registers referenced by a given operand may hold 16 values ​​totaling 128 bits.

[0077] The nomenclature used herein for the hist instructions is "f32v4hist", where "f32" refers to the floating-point format of the input and "v4" refers to the size of the vector. As mentioned above, for lower precision representations, the number of values ​​of a vector that can be processed within one instruction increases. This results in multiple hist instructions with the names "f32v4hist", "f16v8hist" which bins a vector of 8 values ​​represented in 1 / 2 precision or floating-point 16 format, and "f8v16hist" which bins a vector of 16 values ​​represented in "1 / 4 precision" or floating-point 8 format. Note that this list of instructions is not exhaustive. Other instructions with the same functionality described below may be defined for representations of vectors of different numbers of bits and / or different sizes.

[0078] We now describe the 32-bit hist instruction "f32v4hist." This instruction causes execution unit 18 to count a vector of four single-precision (i.e., 32-bit) elements into a histogram of four buckets (bins). In this example, each register is a 32-bit register such that the second operand defines an array of four 32-bit values. However, in other embodiments, the registers may have a different capacity, e.g., 16-bit or 64-bit, and the size of the vector and / or the precision of the values ​​may differ. As will be explained later, for a given register capacity, various instances of the hist instruction may be defined to process vectors having lower precision representations, such that the relationship between values ​​and registers is not necessarily one-to-one. Note that while this example describes a four-bin histogram, any multi-bin histogram may be used. The instruction takes as a first operand the state information of the histogram, including several fields that will be explained in more detail below, and as a second operand a vector of values. The instructions are configured to control execution unit 18 to update the count field of each bin of the histogram before storing the updated histogram value back in its corresponding register, as described below with reference to FIG.

[0079] The instruction has the following syntax: f32v4hist $aSrcDst0:aSrcDst0+3 $aSrc0:aSrc0+3, where f32v4hist is the instruction identifier as described above, the operands identify the register index range by identifying a set of four registers (SrcDst0 to SrcDst0+3) in ARF 26A, respectively, and "SrcDst" is used to indicate that the registers are both source and destination registers for the hist instruction. This operand defines the state information for the histogram, including the count and range of the histogram bins, and when the instruction is executed, the updated bin counts are stored in the registers of the same range. Only the bin counts are affected by the hist instruction, the other state information is fixed.

[0080] Each bin is defined by a range of exponents [base, base+range]. In one exemplary embodiment, each element of the first operand has 32 bits and takes the following format: [17:0] BIN_COUNT: 18 bits of each bin of the first operand are assigned to a saturation count value (in other words, once a predefined maximum count is reached, the instruction no longer updates the count). [25:18] THRESH_EXP: These 8 bits represent the lower threshold exponent configuration value that defines the limit of a given bin. This field is not modified by execution of any histogram instruction. · [29:26] THRESH_RANGE: These four bits represent the exponent range configuration value. These bits have two special values. If THRESH_EXP is not equal to all ones, a THRESH_RANGE of 0 (also written as 0b0000) means that all exponents less than or equal to THRESH_EXP for the current bin should be counted, and a THRESH_RANGE of 0b1111 means that all exponents greater than or equal to THRESH_EXP for the given bin are counted. · [31:30] SIGNC: These two bits determine the sign handling when adding new values ​​to the histogram. 0b00 or 0b01 means that the sign is ignored and both positive and negative values ​​are counted if they fall within a given range of a bin. 0b means that only positive values ​​are counted. 0b11 means that only negative values ​​are counted.

[0081] Figure 5 shows exemplary logic circuitry implemented within execution unit 18 for executing the steps of a hist instruction. Circuitry for executing the hist instructions as floating point instructions is implemented within the FPU of auxiliary execution unit 18A. It should be noted that Figure 5 shows only the circuitry necessary to execute hist instructions, but the FPU of auxiliary execution unit 18A also includes circuitry for performing other types of floating point operations.

[0082] As described above, the instruction takes as operands register indexes that specify a register holding the histogram bin state information 704 and another register holding the vector 702 values ​​to be added to the histogram, both of which are held in the arithmetic (or auxiliary) register file 26 A. The bin state information is passed to a bin validation circuit 704 which performs validation on the bin state information, and the threshold range bits and threshold exponent for a given bin are validated against the special values ​​of THRESH_EXP and THRESH_RANGE, as described above.

[0083] In an embodiment described further below with reference to Figure 6, denormalized numbers are treated as 0 and no special values ​​of THRESH_EXP are used. However, in other embodiments, as described below with reference to Figure 7, a special value of 0b11111111 for THRESH_EXP is defined that defines a given bin to contain only 0s or dinors, and the value of THRESH_RANGE determines whether the bin is used to count 0s (if THRESH_RANGE=0) or dinors (for any other value of THRESH_RANGE). Special values ​​of THRESH_RANGE when THRESH_EXP is not 0b11111111 are described above.

[0084] The special cases mentioned above indicate various modes of operation of the instruction and are also referred to herein as "mode indicators." For example, if THRESH_EXP is not 0b11111111 and THRESH_RANGE is 0, then the histo instruction for that bin will operate in a mode of counting values ​​below the threshold instead of the default mode of counting values ​​within a range with an upper and lower limit. The bin validation circuitry not only checks for each of these specific cases, but also checks the SIGNC bit of the bin to determine how to handle the sign of the value. The bin validation circuitry then outputs one or more signals indicating whether any particular case is flagged by the histogram status and, if so, which one. This is passed to a comparison circuit 708 which processes the value along with the bin status information to determine whether the value falls within the range defined for that bin. Note that each value of the vector is processed for each of the bins defined by the instruction. However, Figures 6 and 7 are simplified for clarity and only show the processing of a single value for each bin.

[0085] Although the bin validation circuit 706 is shown in FIG. 5 as a single component, it may include multiple logic components to validate both the threshold exponent and the threshold range and direct the status information to the comparison circuit 708. If the bin validation circuit 706 determines that the threshold exponent and the threshold range do not have special values, then the threshold exponent and the threshold range are processed by logic in the comparison circuit that takes the value 702 of the vector and determines whether it falls within the range of the given bin, i.e., whether the value 702 of the vector is greater than or equal to the threshold exponent of the bin and less than the upper limit given by the threshold exponent and the threshold range. The comparison circuit also includes logic for the specific case described with reference to FIG. 7. If the bin validation circuit 206 determines that the threshold exponent takes the special value of 0b11111111 and the threshold range is 0, then only 0s are counted and the value 702 is passed to logic that validates that the mantissa and exponent of the value are 0. Logic is also implemented to validate whether the value 702 is a denorm, and the bin validation circuit 706 determines that the threshold exponent and the range are set to the respective values ​​to indicate this specific case, as described above. Again, the compare circuit 708 is shown as a single component but may contain many logic circuits to handle several specific cases, which are further described below with reference to Figures 6 and 7 above. If THRESH_EXP and THRESH_RANGE do not take any of the special values ​​mentioned above, the instruction reverts to a default mode of counting values ​​whose exponents fall within the range [THRESH_EXP, THRESH_EXP+THRESH_RANGE]. For all cases, the compare circuit 708 contains logic to compare the value to either a bin state or a special value (such as 0) and issue a signal to the increment circuit only if the value satisfies the condition of the compare logic.

[0086] Increment circuit 710 increments the bin count of bin status 704 by one if a corresponding signal is received from compare circuit 708. The updated count is written back to the register containing the bin status within ARF 26A, as indicated by the arrow from increment circuit 710. Note that while the above description only refers to processing a single value of a vector of a single bin, the above process is repeated for each given value and bin. Writing the bin count back to the register of ARF 26A can only occur after all values ​​and bins for an instruction have been processed (where the vector of counts for all bins is used to maintain updated count values ​​while the instruction is executing) and before the count is written back to the register of bin status information.

[0087] 6 is a flow diagram illustrating the steps of executing a hist instruction when denormalized values ​​are treated as 0. Before instruction computation begins, execution unit 18 prepares the data for the operands to extract the individual elements of each bin and each value of the vector such that each operand is represented as an array of values ​​of a given numeric representation, which in this example is 32-bit floating point.

[0088] In step S600, the bin index is set to 0, which represents the first bin of the histogram. The schematic diagrams shown in Figures 6 and 7 show the steps taken to process a single value of a single bin, which may be the first bin, for example. However, as noted above, multiple bins may be processed in parallel, and in such an embodiment, the initialization step S600 is not required to set an initial bin that is fully processed before the execution unit considers the next bin, but instead some or all of the bins may be processed simultaneously by steps S602-S614. The count of a bin is BIN_COUNT, which is read from the BIN_COUNT bits for that bin in the operand. The counts may be added to an array "count", which is indexed by the bin and gives the count of values ​​for each bin in the histogram. The "threshRange" and "threshExp" variables are also defined as the respective THRESH_RANGE and THRESH_EXP bits of the operand for a given bin.

[0089] The auxiliary execution unit 18A processes each value of the second operand, i.e., each value of the vector that is assigned to a bin of the histogram, in a loop. The processing of multiple values ​​is not shown in FIG. 6 or 7. However, as pointed out above, the processing of bins and values ​​can be performed in parallel by the execution units, and there is no requirement to process each value of the vector in sequence. These values ​​are referred to as "input values" in the following description for brevity. In step S602, the execution unit 18A checks, for the current bin, whether the count has saturated, i.e., whether the count has reached a maximum count value, which is done by using the bin check circuit 706. If the count is at the maximum value, the loop breaks, the count does not change, and no value is processed. Otherwise, the instruction proceeds to process the input values ​​in sequence. For each input value, the sign bit of the bin is checked in step S604 to see if it matches the sign of the input value based on the value provided on SIGNC. If the signs do not match, no value is added to the given bin and the count remains the same.

[0090] If the signs match, the bin validation circuit 706 performs S606 which first checks for a special value of threshRange. If threshRange is zero, a check (S608) is made by the comparison circuit 708 to determine whether the input value has an exponent less than or equal to the threshold exponent of the current bin. As described above, any value having an exponent less than or equal to the threshold exponent threshExp is added to the count of the given bin by the increment circuit 7. If threshRange is zero but the exponent of the input value is greater than the threshold exponent value threshExp, the comparison circuit does not output a signal to instruct the increment circuit to increment the count and the count remains the same.

[0091] If threshRange is not equal to 0, then in step S610, the bin validation circuit 706 checks if threshRange is 0b1111 and a second special value where all values ​​are greater than or equal to the threshold is added to the current bin's count. In step S612, the exponent of the input value is compared to the threshold exponent by the comparison circuit, and if it is greater than or equal to the threshold exponent, the comparison circuit 708 outputs a signal to the increment circuit 710 and the value is added to the current bin's count, otherwise the count remains the same.

[0092] Finally, if threshRange does not have any of the above special values, a check is made by the comparison circuit 708 to determine whether the input value exponent is within the defined exponent range of the current bin (at step S614). The comparison circuit checks whether the input value meets the following conditions: index≧threshExp and index<(threshExp+threshRange). If the exponent is within this range, the comparison circuit 708 outputs a signal to the increment circuit 710 and the count is increased by 1 for the current bin. Otherwise, the count of the current bin remains the same.

[0093] Although a loop through each value of the input vector is not shown in FIG. 6, steps S602-S624 are performed for each input value of the vector. Although a loop through each bin and floating point value of the vector has been described, there is no requirement that the processing of the bins and values ​​be performed sequentially or in any particular order. In fact, since the incrementing of a bin for a given value does not have any dependency on other bins and values, the comparison of the input value to the range of values ​​defined for each bin may be performed in parallel in hardware. In parallel, two or more bins are processed simultaneously in relation to one or more of the input values. All that is required for all bins and values ​​is that the state information is checked for each bin to determine whether any particular case applies, and the state information of each bin is compared to each value of the input. Once the vector has been fully processed for a given bin of the histogram, execution unit 18A proceeds to step S616, where it checks whether all bins have been processed according to instructions. If not, the next bin for processing is selected and the associated steps of S604-S614 are repeated for each value of the input vector for the next histogram bin. Once all bins have been processed in this manner, the updated count array based on the values ​​of the input vector is set as the new BIN_COUNT value for each bin in the operand and written back to the original register of the first operand.

[0094] Note that the special values ​​threshRange=0 and threshRange=0b1111 allow for the creation of cumulative histograms. When threshRange=0, all values ​​with an index equal to the current bin's threshold index or any lower index are counted in the current bin, thus accumulating "all values ​​in the lower bin" + "values ​​that fall within the range of the current bin". Conversely, when threshRange=0b1111, values ​​with an index equal to or greater than the bin's threshold index are counted, meaning that the bin with the lowest threshold contains all values ​​within it + values ​​that fall within the range of all higher bins.

[0095] The above example describes the processing of 32-bit vector values ​​by execution unit 18A in accordance with a 32-bit version of the hist instruction. However, this is not limiting and the above process may be applied to add values ​​of any floating point precision to the histogram. In one exemplary embodiment, denormalized 32-bit floating point numbers are treated as zero, while low precision floating point denormalized numbers may be counted as dinorms apart from 0 by the hist instruction. When processing values ​​that may contain dinorms, additional steps may be taken to allow counting of 0 or dinorms only in a given bin. This is shown in FIG. 7, where the steps are the same as those in FIG. 6, with the addition of step S605 to check the value of threshExp.

[0096] In step S605, if threshExp has the special value 0b11111111, a check S607 is made by the bin check circuit 706 to determine whether threshRange is 0. If threshRange is 0, the count of the given bin is incremented only for zero values ​​as determined in step S611, and the comparison circuit 708 checks whether the exponent and mantissa of the input value are 0, and if so, signals the increment circuit to increment the count of the current bin by 1. If threshRange takes any other value, in S609, a check is made by the comparison circuit 708 to determine whether the input value is a denorm. This circuit checks whether the exponent is 0 and whether the mantissa is not 0, in which case the value is a denorm. If the exponent and mantissa are both 0 as described above, the value is 0, which is not added to the denorm count in this special mode given by the special value of threshExp. If the value is a denorm, a signal is sent to increment circuit 710 to increment the count of the given bin by one, otherwise the count remains unchanged.

[0097] If the threshold exponent can take any value other than 0b11111111, then this is treated as the "default mode" and processing then continues from step S606 as described above with respect to Figure 6. In some embodiments, as described above, f32 denormalized numbers are treated as 0 and added to the count of bins whose range includes 0. For low precision formats, if the threshold exponent does not take the value 0b11111111 (i.e., the default case), then the dinorm is not treated as 0, but because the bins are determined based only on the value of the exponent, the dinorm and 0 are added to the count of the same bin and are therefore processed in the same manner.

[0098] When the instruction is implemented as described above for the low-precision format, the execution unit 18 prepares the values ​​of the vector before processing by applying a function to the second operand register data to extract 16-bit values ​​from a set of 32-bit registers. For 16-bit floating-point numbers, the function "pickHalf", which takes 0 or 1 as an argument along with the register index, returns either the first or second half of the bit string in each register index as a separate 16-bit value. This can be implemented with both halves of a given 32-bit register to extract two 16-bit floating-point numbers from a 32-bit operand register. Similarly, for 8-bit floating-point values, the function "pickQuart" is implemented with each of 0, 1, 2, 3, and 4 as arguments to extract various quarters of the 32-bit operand register data for use as the 8-bit values ​​of the vector. This conversion process allows 16-element f8 vectors and 8-element f16 vectors to be processed within a single instruction as defined above, with four 32-bit registers provided as the second operand. It should be noted that other versions of the HIST instruction may be defined with a different number of registers and / or registers of various capacities specified in the second operand, and the above implementation is not limited to four 32-bit registers.

[0099] Execution of the above hist instructions allows multiple values ​​to be added to a multi-binned histogram within a single machine code instruction, allowing for faster and more efficient computations compared to standard arithmetic instructions.

[0100] Collecting histograms of data being processed is useful for generating statistics or accumulated metrics of data being processed by a computer system. One exemplary application of histograms within the context of machine intelligence applications is automatic loss scaling of gradients when training neural networks. As briefly mentioned above, various floating point formats may be used to represent values ​​in neural network training, and a scaling factor may be used to reduce numerical underflow in the low precision format. This works by scaling up the gradients by a constant factor, processing the scaled gradients in the low precision format for advanced calculations such as matrix multiplication and convolution, and rescaling back by the same factor when the advanced calculations are complete and the high precision format is used instead. One potential problem with loss scaling is that if the scaling factor is too large, the scaled gradients may be increased beyond an upper threshold of representable values ​​available in a given format. Automatic loss scaling works by calculating a statistical distribution of the gradients to determine which portions of the gradients are above and / or below a predefined threshold, and increasing or decreasing the loss scaling factor in response to the determined portions. A histogram of gradients can be generated to represent this statistical distribution of gradients. In the simplest example, a two-bin histogram can be used, with the histogram's threshold index defining an index that is close to the upper limit of representable values ​​within a given numerical precision. An example of this is shown in Figure 8.

[0101] FIG. 8 illustrates how the loss scaling factor affects the gradient statistics and prevents numerical overflow. On the left side of FIG. 8, a simplified distribution of gradient values ​​is shown. This distribution shape is merely illustrative and thus not intended to represent a realistic distribution of gradient values. A threshold T above which there are a small number of gradients is shown. The same distribution is represented in quantized form as a two-bin histogram 802 below. The first bin h1 gives the count of all gradients below the threshold T, and the second bin h2 gives the gradient of all counts above the threshold T. The above-mentioned hist instruction implemented for the set of two bins can be used to generate the histogram 802 by processing the gradients of the network as a set of vectors of size n, where n is determined based on the capacity of the registers and the precision of the gradient representation. The total number of gradients to be processed can be very large, and the gradients can be loaded at once as vectors of n elements into the operand registers for processing by the hist instruction.

[0102] The histogram 802 represents the number of values ​​within a range defined for each bin, therefore the threshold index and threshold range bits do not take on any special values ​​and the count is based on whether the index of the value falls within an index range defined for each bin of the histogram, predefined by the user.

[0103] The threshold exponent may be selected as an exponent close to the maximum representable value in a given numeric representation. For purposes of avoiding overflow, a useful statistic is to determine what percentage of the gradient exceeds a value close to the maximum representable value in a given numeric format. For example, for FP16, the threshold may be selected at 1 / 2 the maximum number representable by FP16, which is 33762.

[0104] The percentage of gradients that exceed this threshold gives an indication of how much overflow is occurring in the network, which can be obtained directly from the counts determined by the hist command described above for the entire set of gradients as follows:

number

[0105] A minimum fraction f can be set where the loss scaling factor is determined to be too high and causes too many gradients to overflow, and should be reduced. An example fraction f for FP16 is, for example, 10 -6 It should be noted that the histograms in Figure 8 are not to scale.

[0106] If a greater percentage of the gradients than f are determined to be above the threshold T, then the loss scaling factor may be reduced by a factor s. This has the effect of shifting the distribution of gradients downwards such that when the gradients are scaled by this factor, a minority percentage of the gradients exceed the threshold T. If the loss scaling factor is below the threshold, then either an algorithm may be applied that increases the loss scaling factor at each of multiple loss scaling factor update steps, or that only updates the loss scaling factor after a number of successive update steps where the percentage above the threshold is less than a critical percentage f.

[0107] Gradient histograms containing more than two bins may be computed. In this case, the hist instruction simply defines a set of n bins as the first operand, where n=4 in the illustrations of Figs. 6 and 7 and n=2 in the example application of Fig. 8. If the gradient histogram contains more than two bins with bin edges {b_1, b_2,..., b_(n-1)} and bin counts {h_1, h_2,..., h_n}, after M successive optimization steps for a given threshold T, the loss scaling factor L is increased if and only if the fraction of the total counts of all bins whose edges are greater than or equal to the threshold T does not exceed a user-defined fraction f. That is,

number

[0108] In terms of the counts calculated by the histo instruction, as described with reference to FIG.

number

[0109] The loss scaling factor is reduced when the fraction of values ​​in a bin that exceed the threshold exceeds a critical fraction f.

[0110] This allows for the flexible use of lower precision formats to represent gradients of neural networks in a lower precision format, allowing for more efficient computations, especially for computationally intensive operations such as matrix multiplication and convolution.

[0111] It should be noted that histograms of values ​​may be useful for other applications within machine intelligence models and elsewhere, and the hist command is broadly applicable to determining the distribution of any given set of floating-point numbers. For example, the number of dinors in a given data set may indicate the degree of underflow when processing the data set, which may also be used to inform the numeric format used to represent the values ​​for this processing. In an extended version of the lossy scaling described above, both the dinors (or values ​​below some predefined lower threshold close to the minimum representable value) and values ​​above a threshold close to the maximum representable value may be monitored to most effectively tune the scaling coefficients to give the best possible tradeoff between the two ends of the dynamic range representable in a given format. For example, if most values ​​fall within the limits of a higher precision representation such as FP32, it may be determined that it would be more efficient to represent the data in a lower precision format such as FP16. Conversely, if the number of denormalized values ​​and the number of values ​​above a predefined threshold near the upper end of representable values ​​are high, this indicates that the format used to represent the set of values ​​does not adequately cover the range of values ​​and a high precision representation may be required. This is just one example of a range of applications for collected statistics of processed data. Other variations or use cases of the disclosed technology may become apparent to those of ordinary skill in the art given the disclosure herein. The scope of the present disclosure is not limited by the described embodiments, but only by the appended claims.

Claims

1. a plurality of operand registers, a first subset of the operand registers configured to store state information for a plurality of bins, for each of the plurality of bins, the state information including a range of values ​​associated with that bin and a bin count, and a second subset of the operand registers configured to store a vector of floating point values; execute a first instruction that takes as operands the state information for the plurality of bins and the vector of floating-point values, and in response to execution of the first instruction, for each of the floating-point values: identifying a bin of the plurality of bins with which the floating point value is within the associated range of values ​​based on an exponent of the floating point value; incrementing the bin count associated with a bin in the plurality of bins identified for the floating-point value; an execution unit configured to execute a processing device including

2. 2. The processing device of claim 1, wherein identifying a bin within the range of values ​​with which each floating-point value is associated comprises: selecting each bin of the plurality of bins; and, for each bin, using a comparison circuit, comparing an exponent of each floating-point value with a condition defining a range of values ​​associated with each bin.

3. 3. The processing device of claim 2, wherein the execution unit is configured to perform in parallel, in response to execution of the first instruction, determining whether each of a plurality of floating-point values ​​is within the range of values ​​associated with each of a plurality of bins.

4. For each of the bins, the state information includes a threshold exponent field; 2. The processing device of claim 1, wherein the execution unit is configured to determine the bin for each of the floating-point values ​​in dependence on a value of the threshold exponent field for that bin.

5. A threshold bin count saturation value is defined, the execution unit is configured to, in response to execution of the first instruction, for each bin, compare the bin count to the threshold bin count saturation value; and identify bins of a non-saturated subset of bins whose respective floating-point values ​​are within an associated value range; A processing device according to any preceding claim, wherein the non-saturated subset of bins comprises bins whose bin counts are less than the threshold bin count saturation value.

6. the state information for each bin includes a sign designator for the bin; the execution unit further includes a sign checking circuit for comparing the sign of each floating-point value with the sign indicator for each bin; The processing device of claim 2 , wherein the range of values ​​associated with each bin includes only values ​​that match the code of that bin.

7. 7. The processing device of claim 1, wherein the state information for each bin includes at least one mode indicator, and the execution unit includes bin identification circuitry configured to identify the range of values ​​associated with a bin based on the value of the mode indicator for that bin.

8. 8. The processing device of claim 1, wherein, in response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit is further configured to identify a bin of the plurality of bins that is within a range of values ​​associated with 0, and to increment a bin count associated with the identified bin.

9. 8. The processing device of claim 1, wherein, in response to the execution of the first instruction, if the floating-point value is a denormalized floating-point value, the execution unit is further configured to identify a bin of the plurality of bins that is within the range of values ​​to which a denormalized value is associated, and to increment a bin count associated with the identified bin.

10. The processing device of claim 7 , wherein the at least one mode indicator includes at least one of a threshold exponent field and a threshold range field.

11. 8. A processing device according to claim 7 or any one of claims dependent on claim 7, wherein if the at least one mode indicator indicates a default mode, a lower limit of the range of values ​​is the value of the threshold exponent field and an upper limit of the range of values ​​is the sum of the value of the threshold exponent field and the value of the threshold range field.

12. 8. A processing device according to claim 7 or any claim dependent on claim 7, wherein if the mode indicator indicates a first special mode, a range of values ​​associated with the bins includes one of 0 and a range of non-normalized values.

13. 13. The processing device of claim 12 when dependent on claim 10, wherein the mode indicator includes the threshold exponent field, and the first special mode is indicated by a predefined special value of the threshold exponent field.

14. 8. A processing device as claimed in claim 7 or any claim dependent on claim 7, wherein, when the mode indicator indicates a second special mode, the range of values ​​associated with the bin includes all values ​​below the threshold of that bin, and when the mode indicator indicates a third special mode, the range of values ​​associated with the bin includes all values ​​above the threshold of that bin.

15. 15. The processing device of claim 14 when dependent on claim 10, wherein the mode indicator includes the threshold range field, and the second special mode and the third special mode are each indicated by a special value of the threshold range field.

16. The processing device of claim 12 when dependent on claim 10, wherein the mode indicator further includes a threshold range field, and when the threshold range value has a special value of 0, the range of values ​​associated with the bin includes only 0, and when the threshold range field is not 0, the range of values ​​associated with the bin includes the range of non-normalized values.

17. 17. The processing device of claim 1, wherein the execution unit is configured to process gradients of a machine intelligence application scaled by a scaling factor, the first instructions being executed by the execution unit on a plurality of vectors of gradients to generate a histogram comprising a plurality of bins, and the loss scaling factor is adjusted relative to a total count of all bins based on a relative count of a predetermined set of the plurality of bins.

18. A processing device according to any preceding claim, wherein the first and second subsets of operand registers are registers of an arithmetic register file.

19. The floating point value is 32-bit representation, 16-bit representation, or ・8-bit representation The processing device according to any one of claims 1 to 18, provided as follows:

20. The processing device of any one of claims 1 to 19, wherein the first and second subsets each comprise four 32-bit registers.

21. 21. A computer program comprising code configured to execute on a processing device according to any one of claims 1 to 20, the code comprising one or more instances of an instruction taking as operands state information comprising at least a bin count for each of a plurality of bins and a vector of floating point values; The code, when executed, causes the processor to, for each of the floating-point values: identifying a bin of the plurality of bins for which the floating-point value is in an associated range of values ​​based on an exponent of the floating-point value; incrementing the bin count associated with a bin in the plurality of bins identified for the floating-point value; A computer program that executes the following:

22. A method of operating a processor configured in accordance with any one of claims 1 to 21, comprising the steps of: Executing a first instruction that takes as operands state information including a bin count for each of a plurality of bins and a vector of floating point values; responsive to execution of said first instruction, for each of said floating point values: identifying a bin of the plurality of bins within a range of values ​​to which the floating-point value is associated based on an exponent of the floating-point value; incrementing the bin count associated with a bin in the plurality of bins identified for the floating-point value; The method includes:

Images