Matrix multiplication in dynamically spatially and temporally divisible architectures

Abstract

The data processing device includes an input circuit that receives a matrix having values ​​in a first format. An output circuit outputs a matrix having values ​​in a second format, while a correction circuit performs a modification of the matrix from the first format to the second format. The second format has first and second vector registers, both configured to be dynamically partitioned spatially and dynamically temporally, and is computationally continuous with respect to the data processing device that performs matrix multiplication.

Description

[Technical Field]

[0001] This disclosure relates to data processing. [Background technology]

[0002] It is desirable to improve the efficiency of matrix multiplication. [Overview of the project]

[0003] Viewed from a first exemplary configuration, a data processing device is provided, comprising: an input circuit configured to receive a matrix containing multiple values ​​in a first format; an output circuit configured to output a matrix containing multiple values ​​in a second format; and a reconciliation circuit configured to perform a modification of the matrix from the first format to the second format, wherein the second format is computationally continuous with respect to a data processing device that performs matrix multiplication using the matrix, and has first and second vector registers configured to both be dynamically divided spatially and dynamically temporally.

[0004] Viewed from a second exemplary configuration, a data processing method is provided which includes receiving a matrix containing multiple weights in a first format, outputting a matrix containing multiple weights in a second format, and performing a matrix modification from the first format to the second format, wherein the second format has a first vector register and a second vector register, both configured to be dynamically partitioned spatially and dynamically temporally, and is computationally continuous with respect to a data processing device that performs matrix multiplication.

[0005] From the perspective of the third exemplary configuration, a computer program for controlling a host data processing device to provide an instruction execution environment, comprising an input logic configured to receive a matrix including a plurality of values in a first format, an output logic configured to output a matrix including a plurality of values in a second format, and an adjustment logic configured to perform a modification of a model matrix from the first format to the second format, wherein the second format has a first vector register and a second vector register configured to be both dynamically spatially and dynamically temporally divided, and is computationally continuous with respect to a data processing device that performs matrix multiplication using the matrix, is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present invention will be further described by way of example only with reference to those embodiments shown in the accompanying drawings. [Figure 1] Schematically shows an example of a data processing device. [Figure 2] Shows the range of instruction duplication. [Figure 3] Shows a data processing device in the form of a preprocessor. [Figure 4A] Shows the effect of the data processing device in the preprocessing and execution of matrix multiplication instructions. [Figure 4B] Shows the effect of preprocessing. [Figure 4C] Shows the effect of further preprocessing. [Figure 5] Shows in more detail the multiplication performed by the multiplication circuit and the relationship between the inputs. [Figure 6] Shows a configuration in which each weight is only 1 bit. [Figure 7] Shows a modified instruction having different bit widths for weights for compression. [Figure 8] Shows how the compression of a matrix (or vector) operates. [Figure 9] Shows in more detail the operation of the multiplexer circuit. [Figure 10]An example where a set of weights is multiplied by a set of activations is shown. [Figure 11] Different possible combinations for the multiplexer inputs are shown. [Figure 12] A configuration in which a 3-to-1 multiplexer is used is shown. [Figure 13A] How matrix multiplication can be performed in a situation having four channels is shown. [Figure 13B] How the compressed data in the form of a sparsity mask can be rearranged is shown. [Figure 13C] A further method by which the compressed data can be rearranged is shown. [Figure 13D] Yet another method by which the compressed data can be rearranged is shown. [Figure 14A] For example, in a matrix multiplication instruction, how the opcode space can be saved is shown. [Figure 14B] For example, in a matrix multiplication instruction, how the opcode space can be saved is shown. [Figure 15] A flowchart showing an example of the present technology is shown. [Figure 16] A flowchart showing the present technology that can be applied by a preprocessor is shown. [Figure 17A] An implementation of a simulator that can be used is shown. [Figure 17B] An implementation of a simulator that can be used is shown.

Best Mode for Carrying Out the Invention

[0007] A neural network can consist of several layers, where a set of inputs is passed to "neurons" (there may be several per layer), and the output of a neuron in one layer is passed as input to a neuron in the next layer. Each neuron receives several inputs (activations), each of which is then multiplied by (potentially different) scaling factors (weights), and the results are added together to produce an output. The weights and connections of these neurons together constitute a model, often recorded as a matrix. The process of training a neural network involves adjusting the weights in each neuron so that the final output value produced by the final layer can be used as a measure (e.g., regarding whether the input represents data of a particular category). Thus, the process of training or using a neural network involves a number of operations of multiplying one vector by another and adding the results (e.g., in dot product multiplication and accumulation operations). Where there is reuse where vectors are multiplied together, there is an opportunity to use matrix multiplication and accumulation operations.

[0008] Before examining embodiments with reference to the attached drawings, the following embodiments will be described.

[0009] In some examples, a data processing device is provided, comprising a first vector register and a second vector register, both configured to be dynamically divided spatially and dynamically temporally; and a decoding circuit configured to receive one or more matrix multiplication instructions including indications of a first set of elements in the first vector register and a second set of elements in the second vector register, and configured to generate a matrix multiplication operation in response to the receipt of a matrix multiplication instruction, wherein the matrix multiplication operation is configured to cause one or more execution units to perform a matrix multiplication of the first set of elements and the second set of elements, and the average bit width of the first elements is different from the average bit width of the second elements.

[0010] It will be understood that representing sets of weights and sets of activations can result in extremely large data structures. For example, in a neural network containing tens to hundreds of layers, each with tens of thousands of neurons using tens or hundreds of activations, the entire data structure can contain millions or billions of activations. If each activation is represented by an 8-bit number, this can result in a very large dynamic data structure to represent a single input (to store the activations). One option is to reduce the size (bit width) of the weights or activations, for example, so that each weight and / or activation occupies only 4 bits. In practice, this can work. However, the inventors of this technique have found that reducing the bit width of both weights and activations degrades performance, while reducing only the average bit width of the weights may result in only a slight performance degradation for a significant reduction in the size of the model data structure (for storing weights). A further advantage of reducing the size (average bit width) of the weights is that, in certain architectures, it is possible to improve computational throughput. This is because more weights can be input at once (because each occupies fewer bits). This technique achieves this by performing matrix multiplication between activations and weights, meaning that at least one of the elements being multiplied is two-dimensional, rather than both elements being one-dimensional vectors. Note that here we are referring to the average bit width; that is, the number of bits used to reference all the weights (for example, in one of the vector registers) divided by the number of weights represented by that data (which may include implicit values). Of course, this technique is not limited to multiplication between activations and weights, but is applicable to other matrix multiplications as well. This technique is particularly well suited to beat-based architectures where vector registers can be dynamically partitioned in space and / or time, so that the size of the elements retrieved from the vector registers can be dynamically controlled, and the rate at which vector instruction "pipelining" occurs can be dynamically controlled.By using a beat-based architecture in combination with matrix multiplication, an increased number of weighted activated multiplications can be achieved without repeatedly loading / unloading values. Spatially partitioned registers involve the ability of a particular processor microarchitecture implementation to change the logical partitioning of bits to represent distinct values ​​of different sizes depending on the value, and to change this statically or dynamically at runtime based on the boot-time configuration. Temporally partitioned registers involve the ability of a particular processor microarchitecture implementation to support beats of different sizes (a variable number of bits in the vector processed per cycle), and to change this statically or dynamically at runtime based on the boot-time configuration.

[0011] In some examples, the first element contains a single row of n activations, the second element contains a pair of columns, each containing n weights, and the matrix multiplication instruction contains an indication of a result register configured to store a first set of bits of the result of the matrix multiplication and a second set of bits of the result of the matrix multiplication. The two columns of weights can each represent a distinct set of weights, for example, each associated with a different filter. These two columns of weights (filters) are then multiplied by the same set of input activations to produce two new output activation channels (the first and second sets of bits of the result of the matrix multiplication). In this way, two filters can be processed simultaneously, doubling the throughput. In practice, it will be understood that the pair of columns can be represented sequentially as the transpose of the columns of the (weight) matrix.

[0012] In some examples, a matrix multiplication instruction includes an indication of further result registers, where the result registers are configured to store a first set of bits of the matrix multiplication result, and the further result registers are configured to store a second set of bits of the matrix multiplication result. In these examples, one of the result registers relates to the output features (activation channels) of one filter (e.g., the filter for column 0), and the other output register relates to the output features (activation channels) of a second filter (the filter for column 1).

[0013] In some examples, matrix multiplication multiplies the number of rows in the first set of elements by the number of columns in the second set of elements. Of course, it should be understood that the use of the terms “columns” and “rows” here refers to the logical nature of the matrix multiplication being performed, and not necessarily to how the data elements are stored in hardware or represented by instructions (which may be transposed).

[0014] In some examples, matrix multiplication is the multiplication of one row of a first set of elements and two columns of a second set of elements. Here again, the use of the terms “row” and “column” refers to the logical nature of the matrix multiplication being performed, rather than how the data is actually stored or represented in the instruction.

[0015] In some examples, a matrix multiplication instruction includes an uncompressed matrix multiplication instruction where the first element contains a single group of n activations, and the second element contains m groups of n weights, where m > 1, and the bit width of the second element is 1 / m times the bit width of the first element. An uncompressed matrix multiplication instruction is an instruction where each element of the matrix exists in the representation (model or data structure) rather than being removed. For example, some forms of matrix compression can remove the value "0" and provide information about the removed zeros, which are then reinserted during processing. In these uncompressed matrix multiplication instructions, a single group of activations is multiplied by a group of multiple (m) weights, where each group of weights is associated with a different feature or channel. Since each weight is multiplied by an activation, there are n activations, and each group of weights contains n weights. The multiplication performed is a matrix multiplication, as the second element forms a two-dimensional matrix as a result of m being greater than 1. It will be understood that the average bit width of the weights is 1 / m times the bit width of the activations. In other words, the weights are smaller than the activations. By reducing the weight size to 1 / m, it becomes possible to consider m groups of weights (e.g., filters) simultaneously.

[0016] In some examples, the bit width of the second element is 4 bits or less. Therefore, the weight can be 4 bits or less, such as 4 bits, 2 bits, or 1 bit. As the bit width of the activation decreases, it becomes possible to process a relatively large number of inputs (e.g., groups of activations) at once. This refers to the elements stored in the first and second vector registers.

[0017] In some examples, the bit width of the second element is 1 bit. This refers again to the elements stored in the first and second vector registers.

[0018] In some examples, the second element is signed. The signature can be implicit (e.g., by the use of bias and scale). For example, in situations where the second element is 1 bit, this can be achieved by using the digit {-1, 1}, which can be represented by the value {1, 0} (i.e., scaled by -2 and biased by +1). By having -1 (rather than 0) as the value, no information is lost during multiplication, and therefore higher representative power is possible. By choosing {1, 0} instead of {0, 1}, the representation conforms more closely to the standard two's complement sign extension bit.

[0019] In some examples, weights are extended before matrix multiplication. Sign extension allows values ​​to be padded in a way that they fit into a larger memory circuit without changing the stored value. For positive values, this can be achieved by prepending zeros in the most significant bit position. For example, the sign extension of the positive value 0101 (5 in decimal) to an 8-bit value is 00000101. For negative values, padding is achieved by prepending one in the most significant bit position. For example, the sign extension of the negative value (in two's complement) 1001 (-7 in decimal) to an 8-bit value is 11111001. In practice, for signed values, sign extension can be achieved by repeating the most significant bit, and for unsigned values, signed extension can be achieved by repeating zeros. Sign extension is performed from the bit width in the input / model where weights are provided to the bit width of the multiplier circuit that performs the multiplication operation. Other forms of extension can be performed using scaling and biasing as described above.

[0020] In some examples, one or more matrix multiplication instructions include an indicator value, or the data processor has a selection register configured to store an indicator value, which is configured to indicate a subset of weights to be used in matrix multiplication during a particular beat of the data processor. Between each beat, parallel multiplication is available. For example, four pairs of (binary) weight subvectors may be available, and a selection register (or passed as an immediate value from the instruction) can be used to indicate which of those pairs of weight subvectors should be multiplied with activation and (extension and) multiplied. For example, if the selection register (or immediate value) contains the value 0x0, bits 0, 1, 2, and 3 are selected for one weight subvector, and bits 4, 5, 6, and 7 (from the same input operands) are selected for a second weight subvector. The value 0x1 in the selection register selects bits 8, 9, 10, and 11 for one weight subvector, and bits 12, 13, 14, and 15 for a second weight subvector, and each weight at those positions is multiplied by the four activation vectors. In the following beat, 0x1 means selecting bits 40, 41, 42, and 43 for one sub-vector of weights, and bits 44, 45, 46, and 47 for a second sub-vector of weights, with each weight at those positions being multiplied by the following vector of the four activations (bits 0-32 are related to the first beat).

[0021] In some examples, at least one bit of the indications for the first set of elements in the first vector register and the second set of elements in the second vector register is used to indicate a subset of weights to be used in matrix multiplication during a particular beat of the data processor. Therefore, rather than using a selection register or an explicit immediate value to indicate which bit to use, this can be inferred from other parameters of the instruction.

[0022] In some examples, a matrix multiplication instruction includes a compressed matrix multiplication instruction that includes an indication of compressed data, where the first element contains a single row of n activations, and the second element contains m groups of n compressed weights, where m>1, and the compressed data shows how the n compressed weights are decompressed to form mn uncompressed weights. In a compressed matrix multiplication instruction, some of the matrix elements exist only implicitly; that is, some elements are excluded in the actual vector register representation. For example, a sparse matrix can exclude elements with the value "0" to create a dense matrix. In these situations, the compressed data is included to show how the compressed matrix is ​​transformed into the uncompressed matrix (e.g., to show where the removed elements are located). Explicit weights can use a full set of bits (similar to activations), while implicit weights may not use any bits. Similar to uncompressed matrix multiplication instructions, each group of (compressed) weights can be associated with a different filter or channel. As a result, it is possible to store weights for more groups / filters / channels. Since implicit values ​​do not exhaust bits, the average bit width of all (implicit and explicit) weights is smaller than what can be achieved with an uncompressed matrix multiplication instruction.

[0023] In some examples, compressed data contains multiple parts, each applicable to one of several matrix multiplication instructions, including a compressed matrix multiplication instruction, and the compressed matrix multiplication instruction includes a compressed data selector configured to select the portion of the compressed data to be applied to form n uncompressed weights. If the compressed data is smaller than the decompressed weights, the contents of the registers storing the compressed data may relate to several different matrix multiplication operations. Thus, the selector is used to choose which portion of the compressed data to use in the current operation.

[0024] In some examples, compressed data is applicable to multiple matrix multiplication instructions, at least some of which indicate a second element distinct from each other, and the compressed data contains several items. Numerous matrix multiplication operations may be performed, each directed to different weights on the same channel. In these situations, if the compressed data is sufficiently small, it can be applicable to multiple matrix multiplication operations and therefore does not need to be reloaded after each operation. Each item could be, for example, a single sparsity mask related to a pair of weights showing how two weights can be expanded to four weights by adding a given value to the indicated position.

[0025] In some examples, compressed data is applicable to more than m groups of n weights. A single matrix multiplication operation can only be performed on m groups of n weights, but compressed data may relate to more than m groups of n weights. There are several ways to store compressed data for some matrix multiplication operations. However, ordering matrix multiplication operations can reduce register pressure (e.g., the number of registers required to store compressed data) by prioritizing storing more weights for a larger number of channels, as opposed to storing more weights for a smaller number of channels, while also requiring the compressed data to be reloaded less frequently.

[0026] In some examples, items are ordered within the compressed data according to the beats used in multiple matrix multiplication operations. For example, items used in the first beat of any of the matrix multiplication operations are clustered together, then items used in the second beat of any of the matrix multiplication operations are clustered together, and so on.

[0027] In some examples, items are ordered within the compressed data so that items used in the same beat of the same single matrix multiplication operation are adjacent to each other. Thus, a single fetch of sequential data from the compressed data can be performed for each beat of each operation.

[0028] In some examples, the compressed data selector is the least significant bit of another parameter in a compressed matrix multiplication instruction. The least significant bit of one of the instruction's parameters can be used to indicate the portion of the compressed data used in that instruction. Therefore, it is possible to save bits required to encode the instruction, as the bits needed to specify the mask selector do not need to be encoded as a separate parameter.

[0029] In some examples, the compressed data selector is the least significant bit of the address of the second element. Since the second element corresponds to the (compressed) weights and the compressed data is used to decompress the weights, it is appropriate that the least significant bit of the weight's location corresponds to the information used to specify which data is used to decompress the particular weights being used by the operation.

[0030] In some examples, the compressed data selector can be combined with the stub address of the first element to form the address of the first element, and the compressed data selector can be combined with the stub address of the result register to form the address of the result register, which stores at least a portion of the results of the matrix multiplication. The stub address can be concatenated with the compressed data selector bits (located in the least significant bit position) to form the addresses of the first element and the result register(s).

[0031] In some examples, the data processing device includes a multiplexer circuit configured to select from among activations to match non-zero uncompressed weights to provide as input to matrix multiplication. Compressed data can take the form of a sparsity mask which can be used to indicate where in the matrix the provided weights belong. For example, consider a situation where two weights are provided and it is known that two "0" values ​​are removed to form the compressed weights within a set of four positions. In this situation, two 2-bit values ​​can be used to indicate which of the four positions each value belongs to. For example, "00" and "10" may mean that the first weight belongs to the first position and the second weight belongs to the third position, thus giving the sequence w0, 0, w1, 0. Similarly, "10" and "11" may mean that the first weight belongs to the third position and the second weight belongs to the fourth position, thereby giving the sequence 0, 0, w0, w1. Therefore, a multiplexer circuit consists of a set of multiplexers (one for each possible position, four in these examples) that can drop the accumulated activations multiplied by zero values, and only the activations multiplied by the compressed weights in an extended form require processing, so bits from a sparsity mask can be used as inputs to indicate which inputs should go into each position. In other examples, bit fields may be used to indicate where non-zero values ​​belong, for example, field 1100 means that non-zero values ​​belong to the first and second positions (of the four).

[0032] In some examples, a multiplexer circuit is configured to select from a subset of activations to match uncompressed weights that are non-zero. Depending on the form of compression used, there are situations where it is not possible for all weights to be present in all possible positions. For example, consider again the previous example where two non-zero weights are retained from an initial set of four values. In this situation, the weights can superficially belong to any of the four positions. However, certain combinations of positions are impossible. For example, if the first weight belongs to the fourth and last position, there is no position for the second weight. Similarly, if the second weight belongs to the first position, there is no position for the first weight. By recognizing this, it is possible to limit the number of inputs that the multiplexer circuit considers, and thus the multiplexer circuit can select only from a subset of activations for a particular position.

[0033] In some examples, a data processing device is provided, comprising an input circuit configured to receive a matrix containing multiple values ​​in a first format, an output circuit configured to output a matrix containing multiple values ​​in a second format, and a reconciliation circuit configured to perform a modification of the matrix from the first format to the second format, wherein the second format has a first vector register and a second vector register, both configured to be dynamically divided spatially and dynamically temporally, and is computationally continuous with respect to a data processing device that performs matrix multiplication using the matrix.

[0034] To rapidly input and output data to the aforementioned data processing device, which features dynamically spatially and temporally divisible vector registers, it may be necessary to provide data in a given format. This allows matrix multiplication instructions to be executed efficiently by providing the data to the matrix multiplication instruction in a way that enables the operation to proceed without the need to scan the data and parse it into a format that can be executed in a parallel manner. This can be achieved by a further data processing device that performs a preprocessing step of converting the data (e.g., matrix or model values ​​or weights) from a first format to a more suitable second format. The second format is such that it is computationally continuous with respect to the data processing device having the first and second vector registers. This enables computation without the need for parsing at runtime. In other words, the elements of the model in the second format can be retrieved and processed (computed) without being rearranged.

[0035] In some examples, the values ​​include a first set and a second set, each of which will be multiplied by a single set of activations in matrix multiplication, in the first format, the first set forms a first consecutive group, the second set forms a second consecutive group, and the tuning circuit is configured to produce the second format by interleaving the values ​​of each of the first and second sets by performing a "z" shaped scan across the first and second sets. Thus, the values ​​(e.g., weights) are modified so that all the values ​​of one input (e.g., a model) are not together but are divided, and possibly repeatedly divided. For example, in the second model, the weights of the first set may be followed by the weights of the second set, then the weights of the first set, alternating. If the first and second sets are represented in the form of a matrix (with columns and rows), the "scan" performed therefore forms a "z" shape. By presenting the weights in this way, the weights can be input to the processing circuit so that one group of activations can be applied to the elements of the first and second sets (at once).

[0036] In some examples, the first and second sets are interleaved in the second format by blocks of size p, where p is equal to the number of activations multiplied simultaneously in the matrix multiplication, and p > 1. The interleaving does not have to alternate between the first and second sets, i.e., one value after another. In some examples, a block of elements from the first set is followed by a block of elements from the second set, and then some elements from the first set. The number of elements may be the same each time, for example, p elements from the first set followed by p elements from the second set. The value p can be chosen to correspond to the number of multiplications that can occur simultaneously in the matrix multiplication. For example, if p = 4, then four values ​​from the first set are followed by four values ​​from the second set, and so on, with all values ​​rearranged in this manner.

[0037] In some examples, the tuning circuit is configured to interleave the first and second sets into a block of size p by performing a two-layer interleaving, in which a "z" shaped scan is performed p times across the first and second sets. For example, a "z" shaped scan can take the values ​​of the first group from the first set and the values ​​of the second group from the second set. This can be followed by the values ​​of another set from the first set at position 1+q and another set from the second set at position 1+q, then the values ​​of another set from position 1+2q in the first set and another set from position 1+2q in the second set, then the values ​​of another set from position 1+3q in the first set and another set from position 1+3qth in the second set. Collectively, these form the first "z" shaped scan. The next "z" shape scan can include a set of values ​​of similar size from position r of the first set, followed by r of the second set, followed by r+q of the first set, followed by r+q of the second set, followed by r+2q of the first set, followed by r+2q of the second set, followed by r+3q of the first set, followed by r+3q of the second set, and so on. A third "z" shape scan can then be performed, and so on. Thus, each "z" takes a value (e.g., a weight) multiplied by the same subset (e.g., activations), and the value "q" represents the number of activations present.

[0038] In some examples, in the second format, values ​​are grouped according to their positions in the first and second sets within q consecutive blocks of size p. The first set of values ​​and the second set of values ​​are interleaved to form q blocks of size p. Across the q blocks, the values ​​from each set maintain the order they had in the first format relative to the other values ​​in that set. This allows the generation of the predicate mask to proceed in the same way as other instructions. In particular, since the relative ordering within each set is not disrupted, a loop counter can be used to compare the number of elements of a vector (or matrix) executed so far with the total number of elements in that vector (or matrix). This can be used to generate a predicate mask indicating the elements to be executed in the next iteration. It is expected that all values ​​(e.g., weights) in the q blocks of size p will be processed within a single beat of a data processing device (having a first vector register and a second vector register). * p values ​​can be processed within a single beat.

[0039] In some examples, each weight has an index j within set i, and in the second format, within each q blocks of size p, the values ​​are ordered first according to their index j, and then according to their set i. A value (e.g., a weight) may also be defined as having a position (i,j), where i is the set number or channel number to which the value (e.g., a weight) belongs, and j is the index number to that set number or channel. In the second format, in a given q blocks of size p, all values ​​with j=0 are first grouped together and ordered (among themselves) in the order of i. Then all values ​​with j=1 are grouped together and again ordered (among themselves) in the order of i, and so on.

[0040] In some examples, the adjustment circuit is configured to perform the correction in a single stage; that is, no intermediate format is generated between the first and second formats. Instead, all sorting is performed at once. In other embodiments, intermediate formats may be generated. For example, one form of interleaving can be performed (e.g., a "z" shaped scan), and then, after generating the ordering of values, further sorting can be performed within each q block of size p.

[0041] In some examples, the values ​​include compressed weights, and the output circuit is configured to output compressed data, which includes several items indicating how the compressed weights can be decompressed into decompressed weights. The weights in the model in the first format may already be compressed weights, in which case the compressed data is provided with the model. Alternatively, the weights in the model in the first format may be decompressed weights. In this case, the data processing device may perform compression and generate compressed data.

[0042] In some examples, compressed data is applicable to multiple matrix multiplication instructions, and at least some of these instructions use different compressed weights. This makes compressed data applicable to a large number of different matrix multiplication instructions.

[0043] In some cases, compressed data can be applied to sets of more than two compressed weights. If the compressed data of a weight is smaller than the weight itself, it is possible to store compressed data for multiple matrix multiplication instructions, each of which may be associated with a different weight. There are two ways to use this fact to store large amounts of compressed data. One is to store a larger number of weights for a smaller number of channels (sets of weights). The other is to store a smaller number of weights for a larger number of channels. By storing the latter, i.e., by storing compressed data in relation to more than two channels, it is possible to reduce register pressure. That is, it is possible to reduce the number of registers required to store compressed data while reducing the number of times the compressed data is loaded into registers.

[0044] In some examples, items are ordered within the compressed data according to the beats used within multiple matrix multiplication instructions. For example, items used in the first beat of any matrix multiplication operation are clustered together, then items used in the second beat of any matrix multiplication operation are clustered together, and so on.

[0045] In some examples, items are ordered within the compressed data so that items used in the same beat of the same single matrix instruction operation are adjacent to each other. Thus, a single fetch of sequential data from the compressed data can be performed for each beat of each operation.

[0046] Here, a specific embodiment will be described with reference to the drawings.

[0047] Figure 1 schematically shows an example of a data processing device 2 that supports the processing of vector instructions. This is a simplified diagram for ease of explanation, and it should be understood that in reality, the device may have many elements not shown in Figure 1 for the sake of brevity. The device 2 comprises a processing circuit 4 for performing data processing in response to instructions decoded by an instruction decoder 6. Program instructions are fetched from the memory system 8, decoded by the instruction decoder to generate control signals that control the processing circuit 4 to process the instructions in an architecture-defined manner. For example, the decoder 6 may interpret the opcode of the decoded instruction and any additional control fields of the instruction to generate control signals that cause the processing circuit 4 to invoke the appropriate hardware unit to perform an operation such as an arithmetic operation, a load / storage operation, or a logical operation. The device has a set of registers 10 for storing data values ​​to be processed by the processing circuit 4 and control information to constitute the operation of the processing circuit. In response to an arithmetic or logical instruction, the processing circuit 4 reads operands from the registers 10 and writes the result of the instruction back to the registers 10. In response to a load / save instruction, data values ​​are transferred between register 10 and memory system 8 via processing circuitry. Memory system 8 may include one or more levels of data cache and main memory.

[0048] Register 10 includes a scalar register file 12 which contains multiple scalar registers for storing scalar values ​​containing a single data element. Several instructions supported by the instruction decoder 6 and processing circuit 4 are scalar instructions that process a scalar operand read from scalar register 12 and produce a scalar result that is written back to the scalar register.

[0049] Register 10 also includes a vector register file 14 containing several vector registers for storing vector values, each containing multiple data elements. In response to a vector instruction, the instruction decoder 6 controls the processing circuit 4 to execute several lanes of vector processing for each element of a vector operand read from one of the vector registers 14 to generate either a scalar result to be written to a scalar register 12 or a further vector result to be written to a vector register 14. Some vector instructions may generate a vector result from one or more scalar operands, or perform additional scalar operations on scalar operands in the scalar register file, and execute lanes of vector processing on vector operands read from the vector register file 14. Thus, some instructions may be mixed-scalar vector instructions, where at least one of the one or more source and destination registers of the instruction is a vector register 14, and the other of the one or more source and destination registers is a scalar register 12. Vector instructions may also include vector load / storage instructions that transfer data values ​​between a vector register 14 and a location in the memory system 8. The load / storage instructions may include contiguous load / storage instructions where the location in memory corresponds to a contiguous range of addresses, or distributed / aggregated type vector load / storage instructions that specify several discrete addresses and control the processing circuit 4 to load data from each of those addresses into each element of a vector register, or to store data from each element of a vector register into discrete addresses.

[0050] The processing circuit 4 can support the processing of vectors having a range of different data element sizes. For example, the 128-bit vector register 14 can be divided into, for example, 16 eight-bit data elements, 8 sixteen-bit data elements, 4 three-thirds-bit data elements, or 2 sixty-four-bit data elements. The control registers in the register bank 10 may specify the currently used data element size, or alternatively, they may be parameters of a given vector instruction to be executed. Thus, it can be said that the registers are dynamically and spatially partitioned.

[0051] Register 10 also includes several control registers for controlling the processing of the processing circuit 4. For example, these may include a program counter register 16 for storing a program counter address indicating the address of the instruction corresponding to the current execution point being processed, a link register 18 for storing the return address to which processing is directed after the processing of a function call, a stack pointer register 20 indicating the location of the stack data structure in the memory system 8, and a beat status register 22 for storing beat status information, which will be described in more detail below. These are only some of the types of control information that can be stored, and in fact, it will be understood that a given instruction set of the architecture can store many other control parameters as defined by the architecture. For example, a control register may specify the full width of a vector register, or the current data element size used for a given instance of vector processing.

[0052] The architecture can support the duplication of vector instructions. For example, the load instruction VLDR and the vector multiply accumulate (VMLA) instruction can be executed simultaneously, even if there is a data dependency between them. This is because each vector instruction executes on only a subset of values ​​at a time, and once the first instruction has worked on the first subset of values, the second instruction can use those values ​​and therefore begin executing VMLA before the execution of VLDR has finished. By duplication of instructions, high-value execution blocks, such as multipliers, can be kept active for longer periods of time.

[0053] As shown in the example in Figure 2, there may be a range of instruction overlap. The execution of a vector instruction is divided into parts called "beats," each beat corresponding to the processing of a portion of a vector of a given size. A beat is a minute portion of a vector instruction that is either fully executed or not executed at all; it cannot be partially executed. The size of the portion of the vector processed in a single beat is defined by the architecture and can be any fraction of the vector. In the example in Figure 2, beats are defined as atomic processing of 1 / 4 of the vector length lane, so that there are four beats for each vector instruction. Obviously, this is just one example, and other architectures can use different beat lane widths, e.g., 64-bit lanes, or the number of beats, e.g., two or eight. The portion of the vector corresponding to a single beat may be the same size as the data element size of the vector being processed, and may be larger or smaller. Thus, even if the element size varies from implementation to implementation or between executions of different instructions, a beat is a specific fixed width of the vector processing. If the portion of the vector processed in a single beat contains multiple data elements, the carry signal can be disabled at the boundaries between each element to ensure that each element is processed independently. If the portion of the vector processed in one beat corresponds to only a part of the elements, and the hardware is insufficient to compute several beats in parallel, the carry output generated during one beat of processing can be input as a carry input to the next beat of processing, so that the results of the two beats together form a data element.

[0054] As shown in Figure 2, different microarchitectural implementations of the processing circuit 4 may execute different numbers of beats in a single “tick” of the abstract architecture clock, where “tick” corresponds to a unit of progress of the architecture state (for example, in a simple architecture, each tick may correspond to an instance that updates all architecture state associated with the execution of an instruction, including updating the program counter to point to the next instruction). It will be understood by those skilled in the art that known microarchitectural techniques, such as pipeline, may require multiple clock cycles to execute a single tick at the hardware level, and that in fact, a single clock cycle at the hardware level may process multiple parts of multiple instructions. However, such microarchitectural techniques are invisible to software because the ticks are minute at the architecture level. For the sake of brevity, such microarchitectural techniques will be ignored for the time being during further description of this disclosure.

[0055] As shown in the lower example of Figure 2, some implementations can schedule all four beats of a vector instruction within the same tick by providing sufficient hardware resources to process all beats in parallel within a single tick. This may be suitable for higher-performance implementations. In this case, the entire instruction can be completed in one tick, so there is no need for instruction duplication at the architectural level.

[0056] On the other hand, a more area-efficient implementation can provide a narrower processing unit that can handle only two beats per tick, and as shown in the central example of Figure 2, instruction execution can overlap with the first and second beats of a second vector instruction, which is executed in parallel with the third or fourth beat of a first instruction, and these instructions are executed on different execution units within the processing circuit (for example, in Figure 2, the first instruction is a load instruction executed using the load / storage unit 200, and the second instruction is a multiply-accumulate instruction executed using MAC 202).

[0057] Furthermore, more energy- and area-efficient implementations can provide a smaller hardware unit capable of processing only one beat at a time, in which case one beat may be processed per tick, and instruction execution is overlapped and shifted by only one beat, as shown in the upper example of Figure 2.

[0058] The overlaps shown in Figure 2 are just a few examples, and it should be understood that other implementations are possible. For example, some implementations of processing circuit 4 can support the parallel issuance of multiple instructions in the same tick, thereby improving instruction throughput. In this case, two or more vector instructions that start together in one cycle may have several beats that overlap with two or more vector instructions that start in the next cycle.

[0059] Not only can the amount of overlap be varied from implementation to implementation to scale to different performance points, but the amount of overlap between vector instructions can also be varied at runtime between different instances of vector instruction execution within a program. Therefore, the processing circuit 4 may be provided with a beat control circuit 30, as shown in Figure 1, to dynamically control time division, for example, the timing at which a given instruction is executed relative to the previous instruction. This gives the microarchitecture the freedom to choose not to overlap instructions in certain troublesome cases where implementation is more difficult, or depending on the resources available for the instructions. For example, if there are consecutive instructions of a given type (e.g., multiplication and accumulation) that require the same resources, and all available MAC or ALU resources are already being used by another instruction, there may not be enough free resources to start the execution of the next instruction, and therefore, instead of overlapping, the issuance of the second instruction can wait until the first instruction is completed.

[0060] Figure 3 shows a data processing unit 100 in the form of a preprocessor. The preprocessor 100 acquires a model at a first input received by the input circuit 120. The model includes a set of weights associated with each input to be tested by, for example, a neural network. From there, the tuning circuit 130 modifies the model to generate a model in a second format output by the output circuit 140. In the second format, the weights are rearranged and the model is thus tuned so that it can be inserted into the second data processing unit 2 and processed without reanalyzing the model or accessing it "randomly" (rather than sequentially). Thus, the second format is said to be computationally continuous with respect to the second data processing unit 2.

[0061] In the second data processing device 2, the model is received and a first set of vector registers or several vector registers R in bank 14A are received. A1 , R A2 , R A3 It is stored in register R. B1 , R B2 , R B3 The second set, or bank 14B, stores the corresponding set of activations. The decoding circuit 6 receives a matrix multiplication instruction and generates a matrix multiplication operation from it. The neural network operates by performing multiplication between weights and activations. Typically, in a single neuron, the set of weights (vector) is multiplied by the set of activations (vector).

[0062] In the second format model, the weights and activations each have different bit widths. For example, each weight may be 4 bits and each activation may be 8 bits. By using smaller bit widths for the weights, the model can be made smaller. However, by keeping the activations larger, it is still possible to produce good results from the neural network. Also, the data processing unit 2 can use smaller weights to improve throughput. In particular, by carefully arranging the weights of the two channels of the model (as achieved by the tuning circuit 130), it is possible to process both channels simultaneously or in parallel. This process is shown in the following figure.

[0063] Figure 4A shows the effect of the data processing unit on the preprocessing and execution of a matrix multiplication instruction. Here, the matrix multiplication instruction is in the following form: VMLAMAV.S8 RdaA, RdaB, Qn, Qm

[0064] Here, "S8" indicates that the activation is 8 bits, "RdaA" is the location of the scalar register where the first set of result bits is placed, "RdaB" is the location of the scalar register where the second set of result bits is placed, "Qn" is the location of the vector register where the activation is placed in a series of processing lanes, and "Qm" is the location of the vector register where the weights are placed in the processing lanes.

[0065] Preprocessing is performed by the preprocessing data processing unit 100. The input channel is shown as input B. Each channel consists of 16 weights, each of 4 bits. There are two models in the input, and therefore a total of 32 weights. Another input A is a set of 16 activations. Each activation is 8 bits. Thus, the nodes of the neural network multiply each activation by the weight corresponding to it (for example, multiply weight 1 by activation 1, multiply weight 2 by activation 2, etc.). The results of each multiplication for the model weights are added together. Thus, for the two channels, there are two results (either 16 bits or 32 bits) generated in a pair of scalar registers in C (corresponding to RdaA and RdaB in the instruction definition above).

[0066] In this technique, weights belonging to two channels are interleaved to generate a channel in a second format. This is achieved by a "z" shaped scan across the weights. The "z" shape is derived from the fact that the scan passes through a first group of contiguous weights from the first channel, then a second group of contiguous weights in the second channel, then a third group of contiguous weights in the first channel (following immediately after the first group of contiguous weights), then a fourth group of contiguous weights (following immediately after the second group of contiguous weights), and so on. Thus, the "z" shaped scan can be considered an ordered interleaving of blocks of values ​​within a channel. In this example, each group consists of four weights that lead to a "striped" join set of channel 400, represented by 128 bits. The term "p" refers to the number of weights in each block. This also corresponds to the number of activations multiplied simultaneously. The term "q," on the other hand, is used to indicate the number of channels through which entries are processed simultaneously.

[0067] In the first beat (beat 0), the set of multipliers 410 multiplies each weight of the first and second channels by an activation. The results of the multiplications for the first channels are added together and stored in a scalar register. The results of the multiplications for the second channels are added together and stored in a second scalar register (in some embodiments, a single register can be used to store both multiplication accumulators). Since the weights of the first and second channels are interleaved, it is possible to use the same set of activations for both sets of multiplications without requiring data beyond the 32-bit lane boundary associated with beat 0. This is important because the activations have a higher bit width than the weights (8 bits vs. 4 bits), and therefore fewer activations can be stored in the vector register compared to the number of weights. Across a series of four beats, each of the weights of the two channels is multiplied by the set of activations, with half of the multiplications in each beat directed to one channel and the other half directed to the second channel.

[0068] Since the same set of activations is used for each set of multiplications, the number of load operations can be kept low. In fact, in some situations, the ratio of load operations to calculation operations can be kept to 1:1 or close to 1:1.

[0069] Therefore, it will be understood that the valid operation performed is matrix multiplication. This is because each of the weights of the two channels is multiplied by a single set of activations, and the results are added together. In other words, in the example in Figure 4A, a (1×16) matrix is ​​multiplied by a (16×2) matrix (or vector). The second of these inputs is two-dimensional, and therefore the operation in question is matrix multiplication. However, the device shown in Figure 4A achieves matrix multiplication by interleaving (2×16) inputs. This makes it possible to proceed with the multiplication efficiently by avoiding lane boundary crossing in a beat-based context.

[0070] In vector processing, a "predicate mask" can be used to indicate specific entries in a vector that should not be processed. For example, in Figure 4A, it is assumed that the matrix multiplication parameters fit exactly within the beat. Instead, if the multiplication to be performed is a 1x17 matrix x 17x1 matrix, this may not fit entirely within the beat. Therefore, the operation is looped so that in the first iteration, the first set of elements is multiplied, and in the second iteration, the second set of elements is multiplied. This is fine if the total number of elements fits perfectly into the register within a multiple of the register size and fills the register. However, if not, it is appropriate to provide a mechanism to indicate that certain elements in the beat should not be processed. Without this, the still-present "junk" data will be computed and affect the final result.

[0071] Figure 4A shows four 32-bit beats. This allows for the multiplication of a 16x2 matrix of 4-bit weights. However, if the weights form a 17x2 matrix, this must be performed over two iterations, the second iteration of which only multiplies a 1x2 matrix of weights. In this situation, we use a predicate mask to indicate that the second iteration should process only one portion of the beats (the portion holding 2x1 matrix elements).

[0072] A predicate mask can be generated by a simple loop counter, since the number of elements processed per operation is known a priori, and the total number of elements processed is known. Therefore, it can maintain an execution count of the remaining elements to be processed. This can be used to generate the predicate mask in each iteration. In these examples, each multiplication by activation is performed twice (once for each channel), and therefore each entry in the predicate mask refers to a pair of values. Thus, the predicate mask in the previous example could be 1 followed by 15 zeros (one entry for each element multiplied twice, i.e., one entry for each activation).

[0073] However, the reshaping of the beat width 2x4 matrix "z" shown in Figure 4A rearranges the elements, and therefore the weights no longer appear in a consecutive order. Thus, further steps are taken to enable the generation of a predicate mask using a loop counter.

[0074] The first method is shown with respect to Figure 4B. In this technique, each channel of the model is "padding" with null entries (0) that do not affect the computation performed. The padding is performed so that each channel expands to a multiple of the total beat width (32 × 4 in the case of Figure 4A). The predicate mask generated for these instructions is exactly 1 for these instructions because there are no element "overhangs". That is, the complete set of elements or weights fits exactly to the beat over the entire number of iterations.

[0075] The drawback of this approach is that the channel-based model becomes bloated with extraneous and unnecessary data. Furthermore, it leads to null calculations that need to be performed unnecessarily. These null calculations still require some processing, even if they do not affect the final result and are easy to perform.

[0076] Figure 4C shows an alternative example in which in-beat reformation is performed, for example, by the pre-processor 100, in order to avoid the expansion of the model and unnecessary calculations performed in Figure 4B. In Figure 4C, this in-beat reformation is shown as being performed on the data structure generated as a result of the "z" shaped reformation (q = 2, p = 4). However, it is of course also possible to perform both reforms in a single step. Figure 4C is provided mainly to show the nature of the in-beat reformation that occurs. In particular, the in-beat reformation rearranges the elements within each beat as determined by the "z" shaped reformation. In these examples, eight weights (elements) are processed together within a beat, and thus the in-beat reformation is performed within each block of eight weights. The in-beat reformation groups the elements within a beat together based on their order within each channel. In other words, the weights W i,j (where i is the channel index and j is the weight index within that channel) are first sorted according to j and then according to i. Thus, as shown in Figure 4C, the elements W 0,0 、W 0,1 、W 0,2 、W 0,3 、W 1,0 、W 1,1 、W 1,2 、W 1,3 processed within the first beat are sorted to W 0,0 、W 1,0 、W 0,1 、W 1,1 、W 0,2 、W 1,2 、W 0,3 、W 1,3 .

[0077] The generation of the predicate mask can be advanced in the manner described above using a loop counter. If there are three weights for each "overhanging" channel (for example, W 0,0 、W 0,1 、W 0,2 、W 1,0 、W 1,1 、W 1,2The predicate mask can be set to three ones followed by twelve zeros. As mentioned earlier, each entry in the predicate mask corresponds to a number of weights equal to the number of channels, since each of the corresponding weights for each channel is multiplied by the same activation. Note that without inter-beat reformation (i.e., looking only at the structure generated after a rescan of the "z" shape), there are undesirable weights that would be multiplied. For example, weight W 0,3 It is multiplied (this does not apply to overhangs), and the weight W 1,2 It is not multiplied (this is in the overhang).

[0078] Figure 5 shows in more detail the multiplication performed by the multiplication circuit 410 and the relationship between the inputs. In this example, eight 4-bit weights, totaling 32 bits, are accessed in one beat from the first vector register Qm. Also in the same beat, four 8-bit activations, totaling 32 bits, are accessed from the second vector register Qn. The eight weights consist of four weights from the first channel and four weights from the second channel. Therefore, the four activations are multiplied by each of the first set of four weights and each of the second set of four weights. The results of the multiplications are added together. That is, the four multiplications of the first set are added by the first adder 500, and the four multiplications of the second set are added by the second adder 510. The results are stored in the first scalar register RdaA and the second scalar register RdaB, respectively.

[0079] In each subsequent beat, the activations and weights of different sets are multiplied together. The weights belonging to the first channel are added by the same adder 500, and the weights belonging to the second channel are added by the same adder 510. In each case, the result of the addition is added to the result held in the scalar registers RdaA and RdaB. Thus, by the end of the 4 beats, the scalar registers RdaA and RdaB store the result of the matrix multiplication.

[0080] Since weights are smaller in size than activations (4 bits and 8 bits respectively in this example), it is possible to store more weights in the same space. By organizing the weights of multiple channels in free space, and by carefully organizing the weights, it is possible to process additional weights simultaneously.

[0081] In this example, data is output using a pair of scalar registers RdaA and RdaB. In practice, the number of connections or ports required to implement a second scalar register can be reduced by assuming that one of the accumulators / adders 500 always outputs data to even register numbers and the other accumulator / adder 510 always outputs to odd register numbers, and therefore they always map to different memory banks within the register file (when the register file is microarchitecturally divided into even / odd banks).

[0082] This technology is applied to a beat-based architecture in which registers can be divided temporally and spatially as needed, making it possible to process either larger or smaller elements as desired using the same hardware.

[0083] Figure 6 shows a similar configuration where each weight is only 1 bit. Thus, in this example, there are a total of 128 bits associated with 8 different channels (16 bits per channel). For input B, these are arranged as a (64 × 2) matrix.

[0084] In this example, the preprocessing step achieves two-layer interleaving by repeatedly performing a "z" shaped scan to generate a striped output 600. Similar to the example in Figure 5, each "z" shaped scan takes a group of weights from each channel. However, in the example in Figure 6, due to the size of each weight and the number of channels, the "z" shaped scan repeatedly jumps. That is, after taking four weights from the first channel (the first row of the matrix) and four weights from the second channel (the second row of the matrix) to generate output 600, the scan jumps to the first four weights of the third channel, which is further down the first row. Thus, the scan is discontinuous in that some entries are skipped within a particular iteration of the scan. The second iteration then returns near the top of the first column for the next four entries (in fact, this group of entries could start from entry 5 of the first column, where the first group of entries initially taken from the first column ended).

[0085] During execution by the execution data processing device 2, there are more individual (and narrower) B inputs than when processing the 4-bit matrix held in B in Figure 4A. The multiplier circuit 610 is expected to be similar to or identical to the multiplier circuit 410 from Figure 4A for hardware area efficiency reasons, and therefore 610 also consists of eight multipliers, expected to process 2-channel data and store the results in two output scalars. This means that each instance of instruction execution must select 8 of the 1-bit inputs from B of the 32 available 1-bit values. The selection register imm 620 is used to store which consecutive 8 bits (four from the two channels) will be processed by the multiplier circuit 610 for a particular beat. As in the example in Figure 5, the bits are loaded into the multiplier circuit 610, multiplied by activation, and the results are provided to scalar registers RdaA and RdaB.

[0086] Each weight, when loaded into the multiplication circuit 610, is expanded (for example, to 4 bits). That is, for unsigned or signed positive numbers, the most significant bit is filled with 0, and for signed negative numbers, the most significant bit is filled with 1. If the second element is 1 bit and the digit {-1,1} is represented by the value {1,0}, then 1 is expanded to 0b1111 and 0 is expanded to 0b0001.

[0087] In this example, it will be understood that only a pair of scalar registers are still provided. This is because this operation effectively handles the vector-matrix multiplication of 16 8-bit values ​​from A (1x16) and 32 1-bit values ​​from B (16x2), each having two result terms (1x2). Again, this 16x2 from B is only a portion of the available 64x2 terms held internally. In a typical actual use, we see four vector registers loaded with consecutive vectors so that a virtual 1x64 vector (spread across these four vector registers) is available for multiplication of all the contents of the 64x2 held in B. The operation then executes the instruction four times using different A registers (in consecutive order) and different matching immediate terms (0x0, 0x1, 0x2, 0x3) to appropriately multiply each of the larger virtual 1x64 1x16s with the corresponding 16x2s in B. This four-instruction sequence completes the vector matrix product of 64 8-bit values ​​(1×64) and 128 1-bit values ​​(64×2) in B, which has two cumulative result terms (1×2). As shown with respect to Figures 14A and 14B, other parameters of the instruction (such as the address of register A) can be used to determine the immediate term.

[0088] Note that inter-beat reformation is equally applicable to the situation in Figure 6, where the size of each weight is only 2 bits.

[0089] Figure 7 shows a modified instruction with different weighted bit widths for compression. Here, the instruction takes the following form: VMLADBBAV.S8 RdaA,RdaB,Qn,Qm,Q0,# <sel>

[0090] The parameters correspond to the instructions shown earlier. "Q0" stores the corresponding compressed data in the form of a sparsity mask or mux indexing. "sel" (an example of a requested compressed data selector) is an immediate value used to indicate which sparsity mask is taken from Q0. In particular, as shown in Figures 13A to 13D, Q0 can store compressed data (e.g., sparsity masks) for multiple operations due to the fact that the compressed data is significantly smaller than the weight data. For this reason, the sel parameter is used to indicate which part of the compressed data is needed for the current operation. The compressed data can be stored in registers 730 and 740 (e.g., one for each channel decompressed per beat).

[0091] A matrix is ​​sometimes said to be sparse if some of its values ​​(e.g., half) are zero rather than non-zero. However, for the purposes of matrix multiplication, zero values ​​have no effect; that is, the result of any multiplication by zero is always zero, and there is no result to add that result to a series of other multiplications (x+0=x). Therefore, to some extent, zero values ​​in a matrix can be ignored, as long as the relative positions of the other values ​​are correctly preserved. Thus, one way to compress a matrix is ​​to eliminate the zero values, compress the remaining values ​​together, and provide a sparsity mask to understand the true positions of the remaining values.

[0092] Therefore, in the example in Figure 7, input A also consists of 16 8-bit activations. Input B, on the other hand, consists of 32 weights, each 8 bits, with 16 weights belonging to each of the two channels. However, these weights are compressed into 16 8-bit weights and 8 4-bit sparsity masks, which illustrate how the compressed form is converted back to the uncompressed form. In this example, a compression ratio of 1.6x is achieved. That is, half of the entries in the input channel are zero (2x), to which the overhead of the decompression sparsity mask is added. As a result, the average number of bits used to represent activations is reduced because the number of bits used to represent zero is zero (assuming the sparsity mask itself is not included).

[0093] As in the example in Figure 4A, a "z" shaped scan is performed to interleave the weights of the two channels into a single stripe format 700 (in other words, the model is placed in the second format). As in Figure 4A, each group is 16 bits. That is, 16 bits are taken from the first channel, followed by 16 bits from the second channel, then the next 16 bits are taken from the first channel, and so on. The execution is again performed over 4 beats. In each beat, the four weights (two from each channel) are multiplied by 4 activations. In practice, for the sake of compression, this is equivalent to the four weights being multiplied by 4 activations, with two of the weights being zero (and therefore no multiplication being necessary).

[0094] A series of multiplexers 710 are used to rearrange activations to compensate for the removed zero weights. This rearrangement is achieved using sparsity mask information, which is shown in more detail with respect to Figures 8 and 9.

[0095] Once the position is determined, multiplication is performed using the multiplication circuit 720, and the result is provided to the pair of scalar registers RdaA and RdaB, as described above.

[0096] Since the same set of activations is used for each set of multiplications, the number of load operations can be kept low. In fact, in some situations, the ratio of load operations to computation operations can be kept to 1:1 or close to 1:1, requiring only one or more additional loads for the sparsity mask that indicates how the compression was performed.

[0097] With respect to the aforementioned instructions, if padding is performed (in the same manner as in Figure 4B) or in-beat reshaping is performed (in the same manner as in Figure 4C) before the weights are compressed in the preprocessor 110, predicate generation can proceed in the normal manner.

[0098] Figure 8 illustrates how matrix (or vector) compression can work. In this case, the values ​​"x" and "y" are considered non-zero. Therefore, for any of the six different (1×4) matrices shown in Figure 8, a single (1×2) matrix can be obtained through compression by removing zero values.

[0099] Therefore, compressed data can take the form of a (sparseness) mask indicating the index of the location where non-zero values ​​are located. For example, compressed data 0010 may mean that the first non-zero is located at position 00 (i.e., 0) and the second non-zero is located at position 10 (i.e., 2). Assuming a right-to-left numbering scheme, this would therefore correspond to the second form shown in Figure 8.

[0100] There are many other forms that compressed data can take. For example, data can be compressed as <value><position> pairs. For data in a particular (known) format, further compression can be performed using knowledge of the format. For example, if it is known that the first position is always zero, the compressed data can be reduced to 2 bits to indicate where the second zero is located. Other techniques are known to those skilled in the art.

[0101] Figure 9 shows the operation of the multiplexer circuit 710 in more detail. In this example, Qm contains the weights (compressed in this case to remove zero values), and Qn contains the activations. Then, sparsity mask data from register Q0 is included to indicate where the removed zeros should be located so that the remaining non-zero weights are multiplied by the correct activations. It should also be recalled that the weights are associated with two different channels. The first two weights belong to the first channel, and the second two weights belong to the second channel. The multiplexer 920 is provided to select the sparsity mask associated with the current beat (or subbeat). That is, in this example, only a 32-bit register Q0, i.e., 8 bits per beat, is needed to store the sparsity information.

[0102] In this example, each multiplexer in the multiplexing circuit 710 receives a 2-bit switching signal to indicate which activation from Qn (available in the current beat) it should take. Activations that are not taken are those multiplied by a zero weight (revealed by the sparsity mask). Thus, the 8-bit sparsity mask is divided into four 2-bit lots, each of which acts as a switching signal to one of the multiplexers to determine which activation to take. Since the sparsity mask can be different for each of the two channels, it is possible to provide each multiplexer with completely different values.

[0103] As before, the results of the multiplication are added by adders 900 and 910 and stored in scalar registers RdaA and RdaB.

[0104] Figure 10 shows an example where the set of weights (y,0,x,0) belonging to the first channel is multiplied by the set of activations (a,b,c,d). Theoretically, the multiplication performed is (ya) + (0b) + (xc) + (0d). Therefore, this is equivalent to ya + xc. In another example, the second model has the set of weights (0,y,x,0) belonging to the first model, and these are multiplied by the same set of activations (a,b,c,d). Here, the multiplication performed is (0a) + (yb) + (xc) + (0d). This is equivalent to yb + xc. Therefore, any four activations can be taken for any one channel, but only two of them are taken for each channel. A sparsity mask is used to indicate which activations are taken for each channel.

[0105] Figure 9 shows a 4:1 multiplexer, but it will be understood that such complexity is not necessary. This is because not every permutation of activations is possible in every instance. For example, Figure 11 shows different possible combinations. Note that there are only three positions that weight "x" can occupy and only three positions that weight "y" can occupy. This is because if weight "x" is placed in the fourth (leftmost) position in Figure 11, there is no space to place weight "x" (the weight order is preserved). Similarly, if weight "y" is placed in the rightmost position, there is no space to occupy weight "x" if the weight ordering should be preserved as presented in the compressed data.

[0106] In general, if there are n positions for an uncompressed weight and m non-zero weights, there are n+1-m positions that each weight can occupy, and therefore, an (n+1-m)-1 multiplexer is needed for each position.

[0107] Therefore, Figure 12 shows a configuration in which a 3:1 multiplexer is used in the multiplexer circuit 1100.

[0108] It will be understood that the amount of memory required for compressed data is likely to be considerably less than the amount of data required for the weights. For example, when using sparsity mask data, four fields of 1 bit each can be used to represent the non-zero positions in the unpacked set of four weights. If each of those weights is 8 bits, the total of 4 bits for position coding plus 16 bits for the weight values ​​(2 x 8 bits assuming 50% sparsity) will decompress into 32 bits of data, resulting in a compression ratio of 32:20, or 1.6:1.

[0109] In practice, within a memory system, each mask for a given channel may be provided one at a time. That is, if there are four channels, each with N weights, the mask associated with the N weights of the first channel may be stored before the N weights of the second channel, then before the N weights of the third channel, and so on. In practice, loading compressed data in such a format for matrix multiplication requires either multiple registers to store data belonging to multiple channels, or the mask data for different channels to be repeatedly loaded into a single register.

[0110] For example, Figure 13A shows the ordering of the load and calculation operations performed in matrix multiplication. Here, the term Axx is used to refer to the vector register that stores the activations, and the term Byy is used to refer to the vector register that stores the weights. Recall that by reducing the weight size (e.g., from 8 bits to 4 bits), each weight-storing vector register can store weights for two channels. Thus, Czz refers to a pair of scalar registers (e.g., RdaA and RdaB) that each store the result of multiplying the activation of one activation register by the weights belonging to one channel in one weight-storing vector register.

[0111] Figure 13A shows the order in which weights and activations are loaded into their respective registers. Furthermore, Figure 13A shows the order in which multiplication is performed.

[0112] In the ordering shown in Figure 13A, in order for a total of four channels to be processed, the operation is completed for the set of weights on a pair of channels (for example, as shown in Figure 7) before moving on to the next set of weights on an adjacent pair of channels. That is, each operation is performed on the weights of the four channels held in B00 and B01 before operating on the next set of weights further down the four channels held in B10 and B11. Similarly, A00 and A10 are consumed before moving further across the activation vector to A01 and A11. This continues with B20, B21, and A02, A12 (not shown), etc., until the entire vector of n elements has been processed.

[0113] Figure 7, mentioned above, illustrates one of these operations, for example, the multiplication behavior of A00 in register Q0 and B00 in register Q4.

[0114] If sparsity data for all the weights of two channels is stored in a single register, it will be understood that this would require either two registers to store all the mask data for four channels, or a single register where the mask data is repeatedly loaded and unloaded between two pairs of channels. For example, after each operation, the mask data related to the other two channels needs to be loaded.

[0115] Note that in these examples, a selection register is not required. Instead, an immediate value "sel" is passed through each instruction to indicate which portion of the compressed data is needed for the instruction. As will be considered with respect to Figures 14A and 14B, there is a one-to-one mapping between one of the source registers (holding the compressed matrix data) and the immediate value, so this explicit specification of the immediate value may be omitted.

[0116] Figure 13B shows how the natural memory layout of the mask can be reordered by the preprocessor 100 to reduce register pressure. Here, we assume that there are four channels, and each channel contains 64 weights w. i,j refers to the j-th weight of channel i. In this case, the mask is sorted by the preprocessor 100 so that the mask for the first 32 weights of each of the four channels is stored before the next 32 weights of each of the four channels are stored. In other words, the mask is sorted so that as many (equal) weights as possible from each channel are stored in the vector register 14 of the processor 2.

[0117] Therefore, if one of the vector registers 14 is used to store the sparsity mask data, a reload of the mask data is required after performing eight of the operations shown in Figure 13A, that is, until the mask data is needed for register B20.

[0118] Alternatively, note that a single vector register can store all 64 weights from two channels. However, as shown in Figure 13A, the second operation dealing with channels 3 and 4 requires loading the mask data for channels 3 and 4, then the third operation requires loading the mask data for channels 1 and 2, then the fourth operation again requires loading the mask data for channels 3 and 4, and so on. Therefore, one or more extra vector registers would be required, or extra loading operations would be performed. The rearrangement shown in Figure 13B prevents this from happening.

[0119] Figure 13C shows further sorting that can be performed after the sorting performed in Figure 13B. Here, the masks in the register are reordered based on which beat they are used in; that is, the masks used in beat 0 are grouped together, followed by the masks used in beat 1, followed by the masks used in beat 2, and so on. The grouping can take several actions into consideration. For example, the masks presented at the first beat boundary are the masks required for the weights used in the first beat of each of the first eight actions in Figure 13A. Similarly, the masks presented at the second beat boundary are the masks required for the weights used in the second beat of each of the first eight actions in Figure 13A.

[0120] Figure 13D shows further sorting that can be performed after the sorting performed in Figure 13B or Figure 13C. The sorting performed in Figure 13D is similar to the sorting performed in Figure 4C. In particular, the masks within each beat boundary are paired together so that a single 8-bit fetch (of the two masks) can be made relating to all the weights processed within a single beat of a single operation. Thus, in Figure 13D, the masks are ordered by the weight index followed by the channel number (within each beat). As shown in Figure 7, the first beat of the first operation is weight W 0,0 ~W 0,4 (i.e., the first two elements of the first weight sequence in Figure 7) and W 1,0 ~W 1,4 A mask is needed for (i.e., the first two elements of the second weight sequence in Figure 7). Thus, the remaining elements of the first beat are reserved for other actions. In the second beat of the first action, weight W 0,4 ~W 0,7 (i.e., the third and fourth elements of the first weight sequence in Figure 7) and W 1,4 ~W 1,7 The masks required are (i.e., the third and fourth elements of the second weight column in Figure 7). The previously shown sel parameter can be used to indicate which masks in the register are used for a particular operation. The sel parameter indicates the required masks, and since the required masks depend on the weights being used, the same sel parameter is used for each weight register. In fact, the register numbers shown for register B in Figure 13A can be directly converted to sel numbers. Therefore, the indices of the required masks are as follows: <Number of beats> x 8+ <sel>x 2 and (<Number of beats> x 8+ <sel>x 2)+1

[0121] Note that the above diagram shows an example of compression where a sparsity mask in the form of a bit field is used to indicate the positions of non-zero values ​​across the entire matrix. Other forms of sparsity masks or compressions can also be used. For example, non-zero values ​​and the index of where these non-zero values ​​are located within a block can be stored. Thus, the values ​​"000110" and "11" could mean that the value 6 belongs to position 3.

[0122] Figures 13A to 13D illustrate several separate steps for rearranging compressed / sparse mask data, but it will be understood that all of these steps may be performed in a single rearrangement operation, or only a subset of the steps may be performed.

[0123] Figure 14A illustrates a technique that can be used to reduce the number of bits required to encode instructions such as matrix multiplication instructions. This technique is similarly applicable to other forms of instructions. In this example, the least significant bit of the address of one of the source registers (for example, the register used to store weights) is used to determine the addresses of the other source registers, the destination register, and the sel value. In the example in Figure 14B, the least significant bit is 09 (hexadecimal). These bits are then combined with the stub address, which is the bit provided for the destination register (0x111111), to form the full address of the first destination register (0x11111109). Similarly, the bits of the second destination register (0x222222) form the second stub address, which, combined with the least significant bit 09, forms the full address of the second destination register (0x22222209). Similarly, the bits provided to another source register (0x333333) form a third stub address, which, combined with the least significant bit 09, forms the full address of the other source register, for example, the one holding activation (0x33333309). Finally, bit 09 directly corresponds to the sel value.

[0124] In this example, 8 bits are represented in the encoding of each of the two source registers, the destination register, and the sel value, saving a total of 24 bits (32 bits are represented using only 8 bits, saving 32-8=24 bits). Other encoding schemes may save a different number of bits. This technique assumes that the relationship between the offset from the base source register and the other source and destination registers is the same. Note that in different embodiments, this technique may be used for any combination of source registers, destination registers, and sel values. In different embodiments, the bits used may originate from any of these different registers. For example, one bit from a destination register may be used to determine the addresses of other destination registers, source registers, and sel values. In other embodiments, the base values ​​of certain types of registers may be hardcoded, and a single offset may be provided for some or each of the destination and source registers, which can also be used as the sel value.

[0125] Note that in the above example, the load width may be greater than the processor width; that is, the register may be larger than the number of bits that can be processed across the entire beat. In these situations, it is possible to load the values ​​associated with two adjacent matrix multiplication instructions into a vector register. The matrix multiplication instruction can then be directed to either the upper or lower half of the bits, as appropriate.

[0126] Another optimization is the implicit use of specific registers. For example, in the example in Figure 7, register R0 was specified as part of the instruction. In practice, of course, the register used to store the sparsity mask can be fixed, thereby reducing the number of bits required to encode the instruction.

[0127] Figure 15 shows a flowchart 1600 illustrating an example of this technology, particularly in relation to the data processing device 2. Here, in step 1610, a matrix multiplication instruction is received. This refers to a set of first elements (e.g., activations) in a first vector register and a set of second elements (e.g., weights) in a second vector register. The first and second vector registers can be dynamically partitioned temporally and spatially (as necessary). Next, in step 1620, a matrix multiplication operation is generated, causing the execution unit to perform the multiplication of the first and second elements. The first and second elements have different bit widths.

[0128] Figure 16 shows a flowchart 1700 illustrating the technique as it may be applied by the preprocessor 100. In step 1710, a model containing multiple weights in a first format is received. In step 1720, the model is converted from the first format to a second format. In step 1730, a model containing weights in the second format is output. The second format has a first vector register 14A and a second vector register 14B, both configured to be dynamically partitioned spatially and dynamically temporally, and is computationally continuous with respect to a data processing device 2 that performs matrix multiplication using the model.

[0129] The concepts described herein may be embodied in computer-readable code for the fabrication of devices that embody the described concepts. For example, the computer-readable code can be used in one or more stages of a semiconductor design and fabrication process, including an electronic design automation (EDA) stage, to fabricate an integrated circuit comprising a device that embodies the concepts. The computer-readable code described herein may additionally or alternatively enable the definition, modeling, simulation, verification, and / or testing of devices that embody the concepts described herein.

[0130] Figures 17A and 17B illustrate possible simulator implementations. While the above embodiments implement the present invention in terms of devices and methods for operating specific processing hardware that supports the technology, it is also possible to provide an instruction execution environment according to the embodiments described herein, which are implemented by the use of a computer program. Such a computer program is often referred to as a simulator, insofar as it provides a software-based implementation of a hardware architecture. Various simulator computer programs include emulators, virtual machines, models, and binary translators, including dynamic binary translators. Typically, a simulator implementation may run on a host processor 1830, optionally running on a host operating system 1820, and supporting simulator programs 1810A and 1810B. In some configurations, there may be multiple layers of simulation between the hardware and the provided instruction execution environment, and / or multiple different instruction execution environments may be provided on the same host processor. Historically, powerful processors have been required to provide simulator implementations that run at reasonable speeds, but such techniques may be justified in certain circumstances, such as when it is desirable to run native code on a different processor for reasons of compatibility or reuse. For example, a simulator implementation may provide an instruction execution environment with additional functionality not supported by the host processor hardware, or it may provide an instruction execution environment typically associated with a different hardware architecture. An overview of the simulation is provided in "Some Efficient Architecture Simulation Techniques," Robert Bedichek, Winter 1990 USENIX Conference, pages 53-63.

[0131] While embodiments have been described so far with reference to specific hardware components or features, in simulated embodiments, equivalent functionality may be provided by suitable software components or features. For example, certain circuits may be implemented as computer program logic in simulated embodiments. Similarly, memory hardware such as registers or caches may be implemented as software data structures in simulated embodiments. In arrangements where one or more of the hardware elements referenced in the embodiments described above reside on host hardware (e.g., host processor 1830), some simulated embodiments may, if preferred, utilize the host hardware.

[0132] Simulator programs 1810A and 1810B can be stored on a computer-readable storage medium (which may be a non-temporary medium) and provide a program interface (instruction execution environment) to target code 1800 (which may include an application, operating system, and hypervisor) that is the same as the interface of the hardware architecture modeled by simulator program 1810. Thus, program instructions of target code 1800 can be executed from within the instruction execution environment using simulator program 1810, and for this reason, a host computer 1830 that does not actually possess the hardware features of the aforementioned device can emulate these features.

[0133] The simulator program 1810A corresponds to the aforementioned preprocessor unit 100 and includes an input logic 1805 whose behavior corresponds to the behavior of the input circuit 120. An output logic 1815 is provided that behaves in accordance with the behavior of the output circuit 140, and an adjustment logic 1825 is provided that behaves in accordance with the behavior of the adjustment circuit 140.

[0134] The simulator program 1810B corresponds to the aforementioned data processing device 2 and includes a decoding logic 1835 whose behavior corresponds to the behavior of the decoding circuit 6. A first data structure 1845 is provided that corresponds to the first bank 14A of the vector register, and a second data structure 1855 is provided that corresponds to the second bank 14B of the vector register.

[0135] The concepts described herein may be embodied in computer-readable code for the fabrication of devices that embody the described concepts. For example, computer-readable code can be used in one or more stages of a semiconductor design and fabrication process, including an electronic design automation (EDA) stage, to fabricate an integrated circuit comprising a device that embodies the concepts. The computer-readable code described herein may additionally or alternatively enable the definition, modeling, simulation, verification, and / or testing of devices that embody the concepts described herein.

[0136] For example, computer-readable code for fabricating a device embodying a concept described herein may be embodyed by code that defines a hardware description language (HDL) representation of the concept. For example, the code may define register-transfer-level (RTL) abstractions of one or more logic circuits for defining a device embodying the concept. The code may define HDL representations of one or more logic circuits embodying the device in Verilog, SystemVerilog, Chisel, or an intermediate representation such as Very High-Speed ​​Integrated Circuit Hardware Description Language (VHDL) and FIRRTL. The computer-readable code may provide definitions that embody the concept using system-level modeling languages ​​such as SystemC and SystemVerilog, or other behavioral representations of the concept that may be interpreted by a computer to enable simulation, functional and / or formal verification, and testing of the concept.

[0137] Additionally or alternatively, computer-readable code may define low-level descriptions of integrated circuit components that embody the concepts described herein, such as one or more netlists or integrated circuit layout definitions, including representations such as GDSII. One or more netlists or other computer-readable representations of integrated circuit components may be generated by applying one or more logic synthesis processes to the RTL representations to generate definitions used for fabricating devices that embody the invention. Alternatively or additionally, one or more logic synthesis processes can generate bitstreams from computer-readable code that are loaded into a field-programmable gate array (FPGA) to configure the FPGA to embody the concepts described herein. The FPGA may be deployed for the purpose of verifying and testing the concept before fabrication in an integrated circuit, or the FPGA may be deployed directly into a product.

[0138] Computer-readable code may include a mixture of code representations for the fabrication of a device, for example, a mixture of one or more RTL representations, netlist representations, or other computer-readable definitions used in semiconductor design and fabrication processes for fabricating a device embodying the present invention. Alternatively or additionally, the concept may be defined as a combination of computer-readable definitions used in semiconductor design and fabrication processes for fabricating a device and computer-readable code that defines instructions to be executed by the device once it is fabricated.

[0139] Such computer-readable code may be placed on any known temporary computer-readable medium (such as wired or wireless transmission of code over a network) or on a non-temporary computer-readable medium such as a semiconductor, magnetic disk, or optical disk. Integrated circuits fabricated using computer-readable code may include components such as a central processing unit, a graphics processing unit, a neural processing unit, a digital signal processor, or one or more other components that individually or collectively embody a concept.

[0140] In this application, the phrase "configured to..." is used to mean that an element of the device has a configuration that enables it to perform a defined operation. In this context, "configuration" means the arrangement or interconnection of hardware or software. For example, the device may have dedicated hardware to provide the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not mean that an element of the device must be modified in any way to provide the defined operation.

[0141] While exemplary embodiments of the present invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the present invention is not limited to those exact embodiments, and that various changes, additions, and modifications may be made by those skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims. For example, various combinations of the features of the dependent claims may be made together with the features of the independent claims without departing from the scope of the present invention.< / sel> < / sel> < / sel>

Claims

1. A data processing device, A first vector register and a second vector register, both configured to be dynamically divided spatially and temporally, The system includes a decoding circuit configured to receive one or more matrix multiplication instructions that include indications of a first set of elements in a first vector register and a second set of elements in a second vector register, and to generate a matrix multiplication operation in response to the receipt of the matrix multiplication instructions, The matrix multiplication operation is configured to cause one or more execution units to perform matrix multiplication of the first set of elements and the second set of elements, A data processing device in which the average bit width of the first element is different from the average bit width of the second element.

2. The first element comprises a single row of n activations, and the second element comprises a pair of columns, each containing n weights. The data processing apparatus according to claim 1, wherein the matrix multiplication instruction includes an indication of a result register configured to store a first set of bits of the result of the matrix multiplication and a second set of bits of the result of the matrix multiplication.

3. The matrix multiplication instruction includes further indication of result registers, The result register is configured to store the first set of bits of the result of the matrix multiplication, The data processing apparatus according to claim 2, wherein the further result register is configured to store the second set of bits of the result of the matrix multiplication.

4. The data processing apparatus according to any one of claims 1 to 3, wherein the matrix multiplication is performed by multiplying the number of rows of the first set of elements by a number of rows less than the number of columns of the second set of elements.

5. The data processing apparatus according to any one of claims 1 to 4, wherein the matrix multiplication is a matrix multiplication of one row of the first set of elements and two columns of the second set of elements.

6. The matrix multiplication instruction includes an uncompressed matrix multiplication instruction, The first element includes a single group of n activations, The second element contains m groups of n weights, where m > 1. The data processing apparatus according to any one of claims 1 to 5, wherein the bit width of the second element is 1 / m times the bit width of the first element.

7. The data processing device according to any one of claims 1 to 6, wherein the bit width of the second element is 4 bits or less.

8. The data processing apparatus according to any one of claims 1 to 7, wherein the bit width of the second element is 1 bit.

9. The data processing apparatus according to any one of claims 1 to 8, wherein the second element is labeled.

10. The data processing apparatus according to any one of claims 1 to 9, wherein the weights are extended before the matrix multiplication.

11. The one or more matrix multiplication instructions include an indicator value, or the data processing device includes a selection register configured to store the indicator value, The data processing apparatus according to any one of claims 8 to 10, wherein the indicator value is configured to indicate a subset of the weights used in the matrix multiplication during a particular beat of the data processing apparatus.

12. The data processing device according to any one of claims 8 to 10, wherein at least one bit of the indication of the first set of elements in the first vector register and the second set of elements in the second vector register is used to indicate a subset of the weights used in the matrix multiplication during a particular beat of the data processing device.

13. The matrix multiplication instruction includes a compressed matrix multiplication instruction that includes the indication of compressed data, The first element includes a single row of n activations, The second element includes m groups of n compressed weights, where m > 1. The data processing apparatus according to any one of claims 1 to 12, wherein the compressed data shows how the n compressed weights are decompressed to form mn uncompressed weights.

14. The compressed data includes a plurality of parts, each applicable to one of a plurality of matrix multiplication instructions, including the compressed matrix multiplication instruction. The data processing apparatus according to claim 13, comprising a compressed data selector configured to select one of the portions of the compressed data to be applied to form the n uncompressed weights.

15. The compressed data is applicable to multiple matrix multiplication instructions, At least some of the matrix multiplication instructions indicate two distinct elements, The data processing apparatus according to claim 14, wherein the compressed data includes several items.

16. The data processing device according to claim 15, wherein the compressed data is applicable to groups of more than m weights of n weights.

17. The data processing apparatus according to claim 15 or 16, wherein the items are ordered in the compressed data according to beats used in a plurality of matrix multiplication operations.

18. The data processing apparatus according to any one of claims 15 to 17, wherein the items are ordered in the compressed data such that items used in the same beat of the same single matrix multiplication operation are adjacent to each other.

19. The data processing apparatus according to any one of claims 14 to 18, wherein the compressed data selector is the least significant bit of another parameter of the compressed matrix multiplication instruction.

20. The data processing apparatus according to any one of claims 14 to 19, wherein the compressed data selector is the least significant bit of the address of the second element.

21. The compressed data selector is capable of being coupled with the stub address of the first element to form the address of the first element, The data processing apparatus according to claim 20, wherein the compressed data selector is coupled with a stub address of a result register to form an address of a result register in which at least a portion of the result of the matrix multiplication is stored.

22. A data processing apparatus according to any one of claims 14 to 21, comprising a multiplexer circuit configured to select from among the activations to match the non-zero uncompressed weights, for use as input to the matrix multiplication.

23. The data processing apparatus according to claim 22, wherein the multiplexer circuit is configured to select from among a subset of the activations to match the non-zero uncompressed weights.

24. A data processing method, Receiving one or more matrix multiplication instructions including indications of a first set of elements in a first vector register and a second set of elements in a second vector register, wherein each of the first and second vector registers is configured to be dynamically divided spatially and dynamically temporally. This includes generating a matrix multiplication operation in response to receiving the aforementioned matrix multiplication instruction, The matrix multiplication operation causes one or more execution units to perform matrix multiplication of the first set of elements and the second set of elements. A data processing method in which the average bit width of the first element is different from the average bit width of the second element.

25. A computer program for controlling a host data processing device to provide an instruction execution environment, A first data structure and a second data structure, both configured to be dynamically partitioned spatially and temporally, A decoding logic configured to receive one or more matrix multiplication instructions including indications of a first set of elements in a first data structure and a second set of elements in a second data structure, and to generate a matrix multiplication operation in response to the receipt of the matrix multiplication instructions, The matrix multiplication operation is configured to cause the execution logic to perform matrix multiplication between the first set of elements and the second set of elements. A computer program in which the average bit width of the first element is different from the average bit width of the second element.

26. A data processing device, An input circuit configured to receive a matrix containing multiple values ​​in a first format, An output circuit configured to output the matrix containing the plurality of values ​​in a second format, The system comprises an adjustment circuit configured to perform a modification of the matrix from the first format to the second format, A data processing device having a first vector register and a second vector register, both configured to be dynamically divided spatially and dynamically temporally, and which is computationally continuous with respect to a data processing device that performs matrix multiplication using the matrix.

27. The aforementioned values ​​include a first set and a second set, and each of the first set and the second set is multiplied by a single set of activations in the matrix multiplication. In the first format, the first set forms a first consecutive group, and the second set forms a second consecutive group. The data processing apparatus according to claim 26, wherein the adjustment circuit is configured to interleave the values ​​of the first set and the second set in order to generate the second format by performing a "z" shaped scan on the first set and the second set.

28. The first set and the second set are interleaved in blocks of size p in the second format. The data processing apparatus according to claim 26, wherein p is equal to the number of activations that are multiplied simultaneously in the matrix multiplication, and p > 1.

29. The data processing apparatus according to claim 26, wherein the adjustment circuit is configured to interleave the first set and the second set into a block of size p by performing a two-layer interleaving, in which the "z" shape scan is performed p times for the first set and the second set.

30. The data processing apparatus according to any one of claims 26 to 29, wherein in the second format, the values ​​are grouped within q consecutive blocks of size p according to their positions in the first set and the second set.

31. Each weight has an index j within set i, The data processing apparatus according to any one of claims 26 to 30, wherein in the second format, within each q blocks of size p, the values ​​are ordered first according to their index j and then according to their set i.

32. The data processing apparatus according to any one of claims 26 to 31, wherein the adjustment circuit is configured to perform the modification in a single step.

33. The aforementioned value includes compressed weights, The output circuit is configured to output compressed data, The data processing apparatus according to any one of claims 26 to 32, wherein the compressed data includes a plurality of items, and the plurality of items indicate how the compressed weights can be decompressed into decompressed weights.

34. The data processing apparatus according to claim 33, wherein the compressed data is applicable to a plurality of matrix multiplication instructions, and at least some of the matrix multiplication instructions indicate different compressed weights.

35. The data processing apparatus according to claim 34, wherein the compressed data is applicable to a set of more than two compressed weights.

36. The data processing apparatus according to claim 34 or 35, wherein the items are ordered in the compressed data according to beats used in the plurality of matrix multiplication instructions.

37. The data processing apparatus according to any one of claims 34 to 36, wherein the items are ordered in the compressed data such that items used in the same beat of the same single matrix instruction operation are adjacent to each other.

38. A data processing method, The first format involves receiving a matrix containing multiple weights, Outputting the matrix containing the multiple weights in a second format, This includes performing a modification of the matrix from the first format to the second format, A data processing method wherein the second format has a first vector register and a second vector register, both configured to be dynamically divided spatially and dynamically temporally, and is computationally continuous with respect to a data processing device that performs matrix multiplication.

39. A computer program for controlling a host data processing device to provide an instruction execution environment, Input logic configured to receive a matrix containing multiple values ​​in a first format, Output logic configured to output the matrix containing the plurality of values ​​in a second format, The system comprises: adjustment logic configured to perform a modification of the model matrix from the first format to the second format, A computer program having a first vector register and a second vector register, both configured to be dynamically divided spatially and dynamically temporally, and having computationally continuous data processing equipment that performs matrix multiplication using the matrix.

Images