Matrix multiplication in dynamically spatially and dynamically temporally divisible architectures

Abstract

A data processing apparatus includes a first vector register and a second vector register, both of which are dynamically spatially and dynamically temporally divisible. A decode circuit receives one or more matrix multiplication instructions indicating a set of first elements in the first vector register and a set of second elements in the second vector register, and generates a matrix multiplication operation in response to receiving the matrix multiplication instructions. The matrix multiplication operation causes one or more execution units to perform a matrix multiplication of the set of first elements and the set of second elements, and an average bit width of the first elements is different from an average bit width of the second elements.

Description

Technical Field

[0001] This disclosure relates to data processing. Background Technology

[0002] Increasing the efficiency that matrix multiplication might bring is worthwhile. Summary of the Invention

[0003] From the first example configuration, a data processing apparatus is provided, comprising: a first vector register and a second vector register configured to be partitioned in dynamic space and dynamic time; a decoding circuit configured to receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including an indication of a set of first elements in the first vector register and a set of second elements in the second vector register, and to perform a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein the matrix multiplication operation is configured to cause one or more execution units to perform a matrix multiplication of the set of first elements and the set of second elements; and the average bit width of the first elements is different from the average bit width of the second elements.

[0004] From the second example configuration, a data processing method is provided, comprising: receiving one or more matrix multiplication instructions, the one or more matrix multiplication instructions including indications to a set of first elements in a first vector register and a set of second elements in a second vector register, wherein each of the first vector register and the second vector register is configured to be partitioned in dynamic space and dynamic time; generating a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein the matrix multiplication operation causes one or more execution units to perform a matrix multiplication of the set of first elements with the set of second elements; and the average bit width of the first elements is different from the average bit width of the second elements.

[0005] From the third example configuration, a computer program is provided for controlling a host data processing device to provide an instruction execution environment. The computer program includes: a first data structure and a second data structure configured to be partitioned in dynamic space and dynamic time; decoding logic configured to receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including indications to a set of first elements in the first data structure and a set of second elements in the second data structure, and to perform a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein the matrix multiplication operation is configured to cause the execution logic to perform a matrix multiplication of the set of first elements and the set of second elements; and the average bit width of the first elements is different from the average bit width of the second elements. Attached Figure Description

[0006] The present invention will be further described by way of example only, with reference to embodiments of the invention illustrated in the accompanying drawings, wherein:

[0007] Figure 1 An example of a data processing device is illustrated schematically;

[0008] Figure 2 The extent of overlapping instructions is shown;

[0009] Figure 3 A data processing device in the form of a preprocessor is shown;

[0010] Figure 4A The effects of the data processing device on preprocessing and executing matrix multiplication instructions are illustrated.

[0011] Figure 4B The effect of preprocessing is illustrated;

[0012] Figure 4C The effects of further preprocessing are illustrated;

[0013] Figure 5 The multiplication performed by the multiplication circuit and the relationship between the inputs are shown in more detail;

[0014] Figure 6 The arrangement in which each weight is only 1 bit is shown;

[0015] Figure 7 Examples of variant instructions are shown where the weights have different bit widths due to compression;

[0016] Figure 8 This illustrates how matrix (or vector) compression works;

[0017] Figure 9 The operation of the multiplexer circuit is shown in more detail;

[0018] Figure 10 An example is shown where one set of weights is multiplied by a set of activation values;

[0019] Figure 11 This illustrates the different possible combinations of the multiplexer inputs;

[0020] Figure 12 The arrangement using a 3-to-1 multiplexer is shown;

[0021] Figure 13A It demonstrates how matrix multiplication can be performed in a scenario with four channels;

[0022] Figure 13B This demonstrates how compressed data in the form of a sparse mask can be rearranged;

[0023] Figure 13C This demonstrates another way to rearrange compressed data;

[0024] Figure 13D This demonstrates yet another way to rearrange compressed data;

[0025] Figure 14A and Figure 14B This demonstrates how opcode space can be stored in, for example, matrix multiplication instructions;

[0026] Figure 15 A flowchart illustrating an example of this technology is shown;

[0027] Figure 16 A flowchart illustrating this technique, as exemplified by what can be applied by a preprocessor, is shown; and

[0028] Figure 17A and Figure 17B The specific implementation of the available simulator is illustrated. Detailed Implementation

[0029] A neural network can consist of multiple layers, where a set of inputs is passed to “neurons” (each layer can have several neurons), and the output of a neuron in one layer is passed as input to a neuron in the next layer. Each neuron receives multiple inputs (activation values), and each of these inputs is then multiplied by a (potentially different) scaling factor (weights), and the results are summed to produce an output. All these neuron weights and connections constitute the model and are typically recorded as a matrix. The process of training a neural network involves adjusting the weights at each neuron such that the final output value produced by the final layer can be used as a metric (e.g., whether the input represents data of a particular class). Therefore, the process of training or using a neural network involves many operations in which one vector is multiplied by another and the results are summed (e.g., in a dot product summation operation). Matrix multiplication summation operations can be used when there is a reuse where vectors are multiplied by each other.

[0030] Before discussing the implementation scheme with reference to the accompanying drawings, the following description of the implementation scheme is provided.

[0031] In some examples, a data processing apparatus is provided, comprising: a first vector register and a second vector register configured to be partitioned in dynamic space and dynamic time; a decoding circuit configured to receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including an indication of a set of first elements in the first vector register and a set of second elements in the second vector register, and to perform a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein the matrix multiplication operation is configured to cause one or more execution units to perform a matrix multiplication of the set of first elements and the set of second elements; and the average bit width of the first elements is different from the average bit width of the second elements.

[0032] It should be understood that representing weighted sets and activation value groups can result in extremely large data structures. For example, in neural networks with tens to hundreds of layers containing tens of thousands of neurons, each neuron uses tens or hundreds of activation values, and the overall data structure may involve millions or billions of activation values. If each activation value is represented by 8 bits, this can result in very large dynamic data structures (to store the activation values) to represent a single input. One option is to reduce the size (bit width) of the weights or activation values, for example, so that each weight and / or activation value occupies only four bits. In practice, this may be feasible. However, the inventors of this technique have determined that reducing the bit width of both the weights and activation values ​​in order to significantly reduce the size of the model data structure (to store the weights) leads to particularly poor performance, while reducing the average bit width of the weights alone may result in only a small performance drop. Another advantage of reducing the size (average bit width) of the weights is that, in some architectures, it is possible to increase computational throughput. This is because a larger number of weights can be input at once (because each weight occupies fewer bits). This technique achieves this by performing a matrix multiplication between the activation value and the weight, meaning that at least one element of the multiplied elements is two-dimensional, rather than both elements being one-dimensional vectors. Note that this refers to the average bit width. That is, the number of bits used to represent (e.g., in a vector register within a vector register) all the weights divided by the number of weights represented by that data (which may include implicit values). Of course, this technique is not limited to multiplication between activation values ​​and weights and is equally applicable to other matrix multiplications. This technique is particularly well-suited to tick-based architectures where the vector register can be dynamically partitioned spatially and / or temporally, allowing dynamic control over the size of the elements taken from the vector register and the rate at which the "pipelining" of vector instructions occurs. The combination of tick-based architecture and matrix multiplication enables an increasing number of weight-activation value multiplications without repeatedly loading / unloading values. Spatially partitioned registers encompass the ability of a processor microarchitecture-specific implementation to change the logical partitioning of bits to represent different values ​​of different sizes, and the ability to change this statically or dynamically at runtime based on startup-time configuration. The time-divided registers encompass the ability of a specific processor microarchitecture implementation to support different sized ticks (variable bit lengths of the vector processed per loop), and the ability to change this statically or dynamically at runtime based on startup time configuration.

[0033] In some examples, the first element comprises a single row of n activation values, and the second element comprises a pair of columns, each containing n weights; and the matrix multiplication instruction includes an instruction to a result register configured to store a first set of bits and a second set of bits of the result of the matrix multiplication. The two weight columns can each represent a separate set of weights, for example, each weight associated with a different filter. These two weight columns (filters) are then multiplied by the same set of input activation values ​​to produce two new output activation value channels (the first and second sets of bits of the result of the matrix multiplication). In this way, two filters can be processed simultaneously, doubling the throughput. It should be understood that, in practice, the pair of columns can be represented sequentially as the transpose of the columns of a (weight) matrix.

[0034] In some examples, the matrix multiplication instruction includes an instruction to an additional result register; this result register is configured to store the first set of bits of the result of the matrix multiplication; and the additional result register is configured to store the second set of bits of the result of the matrix multiplication. In these examples, one of the result registers is associated with the output feature (activation value channel) of a filter (e.g., the filter for column 0), and the other output register is associated with the output feature (activation value channel) of a second filter (the filter for column 1).

[0035] In some examples, the matrix multiplication multiplied by the first group of elements has fewer rows than the second group of elements. Of course, it should be understood that the terms "column" and "row" used in this document refer to the logical nature of the matrix multiplication being performed, and not necessarily to how the data elements are stored in the hardware or represented in the instructions (which may be transposed).

[0036] In some examples, the matrix multiplication has one row for the first set of elements and two columns for the second set of elements. Again, the use of the terms "column" and "row" refers to the logical nature of the matrix multiplication being performed, not how the data is actually stored or represented in the instructions.

[0037] In some examples, the matrix multiplication instruction includes an uncompressed matrix multiplication instruction; the first element comprises n activation values ​​of a single group; the second element comprises n weights of m groups, where m > 1; and the bit width of the second element is 1 / m of the bit width of the first element. An uncompressed matrix multiplication instruction is one in which each element of the matrix is ​​present in the representation (model or data structure) rather than being removed. For example, some forms of matrix compression might eliminate the value "0" and provide information on the removed zeros to be reinserted at processing time. In these uncompressed matrix multiplication instructions, the activation values ​​of a single group are multiplied by the weights of multiple (m) groups, each weight associated with a different feature or channel. Since each weight is multiplied by an activation value, there are not only n activation values, but also n weights for each group. As a result of m being greater than one, since the second element forms a two-dimensional matrix, the multiplication that occurs is matrix multiplication. It should be understood that the average bit width of the weight is 1 / m of the bit width of the activation value. In other words, the weight is less than the activation value. By reducing the weight by factor m, it becomes possible to consider the weights of m groups simultaneously (e.g., a filter).

[0038] In some examples, the bit width of the second element is four bits or less. Therefore, the weight can be four bits or less, such as four bits, two bits, or one bit. As the bit width of the activation value decreases, it becomes possible to process a considerable number of inputs (e.g., groups of activation values) at once. This refers to the element as stored in the first vector register and the second vector register.

[0039] In some examples, the bit width of the second element is one bit. Similarly, this refers to the element as stored in the first vector register and the second vector register.

[0040] In some examples, the second element is signed. Signification can be implicit (e.g., by using a bias and a scale). For example, in the case where the second element is a single bit, this can be achieved by using the numbers {-1, 1}, which can be represented by the values ​​{1, 0} (i.e., a scale of -2 and a bias of +1). By using -1 as the value (instead of 0), no information is lost during multiplication, and therefore higher representational power is possible. By choosing {1, 0} instead of {0, 1}, the representation more closely matches the standard two's complement sign extension bits.

[0041] In some examples, the weights are expanded before the matrix multiplication. Sign expansion allows values ​​to be padded in a way that fits within larger storage circuitry without altering the stored value. For positive values, this is achieved by adding a 0 before the most significant bit. For example, the sign expansion of the positive value 0101 (decimal 5) to an eight-bit value would be 00000101. In the case of negative values, padding is achieved by adding a 1 before the most significant bit. For example, the sign expansion of the negative value (in two's complement) 1001 (decimal -7) to an eight-bit value would be 11111001. In practice, for signed values, sign expansion is achieved by repeating the most significant bit, while for unsigned values, it is achieved by repeating zeros. Sign expansion occurs where the bit width providing the weights in the input / model extends to the bit width of the multiplication circuitry performing the multiplication. Other forms of expansion can be performed using the scaling and bias described previously.

[0042] In some examples, the one or more matrix multiplication instructions include an indicator value, or the data processing device includes a selection register configured to store the indicator value; and the indicator value is configured to indicate a subset of the weights used in the matrix multiplication during a specific tick of the data processing device. Parallel multiplication is available during each tick. For example, four pairs of subvectors of (binary) weights may be available, and the selection register (or an immediate value passed in from the instruction) may be used to indicate which subvector pairs of these weights will be (expanded and) multiplied by the activation value. For example, if the selection register (or immediate value) contains the value 0x0, bits 0, 1, 2, and 3 are selected for one subvector of weights, and bits 4, 5, 6, and 7 (from the same input operand) are selected for a second subvector of weights. The value 0x1 in the select register will select bits 8, 9, 10, and 11 for a sub-vector of weights, and bits 12, 13, 14, and 15 for a second sub-vector of weights. Each weight at these positions is multiplied by a vector of 4 activation values. In the next beat, 0x1 now means selecting bits 40, 41, 42, and 43 for a sub-vector of weights, and bits 44, 45, 46, and 47 for a second sub-vector of weights. Each weight at these positions is multiplied by the next vector of 4 activation values ​​(bits 0 through 32 are related to the first beat).

[0043] In some examples, bits of at least one of the indications of the first set of first elements in the first vector register and the second set of second elements in the second vector register are used to indicate the subset of weights used in the matrix multiplication during a specific tick of the data processing device. Therefore, this can be inferred from other parameters of the instruction, rather than using the selection register or explicit immediate value to indicate which bits to use.

[0044] In some examples, the matrix multiplication instruction includes a compressed matrix multiplication instruction that includes an indication of compressed data; the first element includes a single row of n activation values; the second element includes n compressed weights for m groups, where m > 1; the compressed data indicates how to decompress the n compressed weights to form mn uncompressed weights. In the compressed matrix multiplication instruction, some elements of the matrix are only implicitly present. That is, some elements are excluded in the actual representation in the vector register. For example, a sparse matrix might exclude elements with "0" values, thus creating a dense matrix. In these cases, compressed data is included to indicate how the compressed matrix will be converted to an uncompressed matrix (e.g., to indicate the location of removed elements). Explicit weights can use a complete set of bits (similar to activation values), while implicit weights may not use any bits. As with the uncompressed matrix multiplication instruction, the (compressed) weights for each group can be associated with different filters or channels. Therefore, it is possible to store weights associated with more groups / filters / channels. Since implicit values ​​do not occupy any bits, the average bit width of all (implicit and explicit) weights is less than the bit width that would be achieved using uncompressed matrix multiplication instructions.

[0045] In some examples, the compressed data comprises multiple parts, each of which can be applied to one of a plurality of matrix multiplication instructions that includes the compressed matrix multiplication instruction; and the compressed matrix multiplication instruction includes a compressed data selector configured to select a portion of the compressed data to be applied to form the n uncompressed weights. When the compressed data is smaller than the decompressed weights, the contents of the register storing the compressed data can be associated with multiple different matrix multiplication operations. Therefore, the selector is used to select which portions of the compressed data are used for the current operation.

[0046] In some examples, the compressed data can be applied to multiple matrix multiplication instructions; at least some of these instructions indicate second elements that are distinct from each other; and the compressed data comprises multiple items. Multiple matrix multiplication operations may occur, each targeting different weights for the same channel. In these cases, if the compressed data is small enough, it may be possible to apply it to multiple matrix multiplication operations, and therefore reloading is not required after each multiplication operation. Each item may be, for example, a single sparse mask associated with a pair of weights, indicating how the two weights can be expanded into four weights by adding predefined values ​​at the indicated locations.

[0047] In some examples, this compressed data can be applied to n weights across more than m groups. While a single matrix multiplication operation might only operate on n weights across m groups, this compressed data allows it to be associated with n weights across more than m groups. There are various ways to store compressed data for multiple matrix multiplication operations. However, by prioritizing the storage of more weights for a larger number of channels rather than more weights for a smaller number of channels, it is possible to reduce register pressure (e.g., the number of registers required to store the compressed data), and the compressed data will also require less frequent reloading due to the order of matrix multiplication operations.

[0048] In some examples, the item is sorted in the compressed data according to the tick of the item being used within the multiple matrix multiplication operations. For example, items used in the first tick of any matrix multiplication operation within the matrix multiplication operation are grouped together, then items used in the second tick of any matrix multiplication operation within the matrix multiplication operation are grouped together, and so on.

[0049] In some examples, the items are sorted within the compressed data such that items used in the same beat of the same single matrix multiplication operation are adjacent. Therefore, continuous data can be extracted from the compressed data in a single operation for each beat of each operation.

[0050] In some examples, the compressed data selector is the least significant bit of another parameter of the compressed matrix multiplication instruction. The least significant bit of one of the parameters of the instruction can be used to indicate which part of the compressed data will be used in the instruction. Therefore, it is possible to save bits required to encode the instruction, since it is not necessary to encode the bits required to specify the mask selector as a separate parameter.

[0051] In some examples, the compressed data selector is the least significant bit of the address of the second element. Since the second element corresponds to the (compressed) weight, and since the compressed data is used to decompress the weight, it is appropriate that the least significant bit of the weight's location corresponds to information specifying which data was used to decompress the particular weight used by the operation.

[0052] In some examples, the compressed data selector can be combined with the stub address of the first element to form the address of the first element; and the compressed data selector can be combined with the stub address of the result register to form the address of the result register, at least a portion of the result of the matrix multiplication being stored at that address of the result register. The stub address can be concatenated with the compressed data selector bit (placed in the least significant bit position) to form the addresses of the first element and the result register.

[0053] In some examples, the data processing apparatus includes a multiplexer circuit configured to select from the activation values ​​to match the non-zero uncompressed weights, thereby providing them as input to the matrix multiplication. The compressed data may take the form of a sparse mask and may be used to indicate where within the matrix the provided weights belong. For example, consider the following scenario: two weights are provided, and it is known that two "0" values ​​are removed in a series of four positions to form the compressed weights. In this case, two two-digit values ​​may be used to indicate which of the four positions each value belongs to. For example, "00" and "10" might mean that the first weight belongs to the first position and the second weight belongs to the third position, thus giving the sequence w0, 0, w1, 0. Similarly, "10" and "11" might mean that the first weight belongs to the third position and the second weight belongs to the fourth position, thus giving the sequence 0, 0, w0, w1. Therefore, the multiplexer circuit can consist of a set of multiplexers (one multiplexer for each possible position, four in these examples) and uses bits from the sparse mask as input to indicate which input should go into each position. This is because activation values ​​that have been multiplied and accumulated with zero values ​​can be discarded, and only activation values ​​multiplied with the weights of the compressed form in the extended form need to be processed. In other examples, bit fields can be used to indicate the position to which a non-zero value belongs; for example, field 1100 indicates that the non-zero value belongs to the first and second positions (of the four positions).

[0054] In some examples, the multiplexer circuit is configured to select from a subset of the activation values ​​to match the non-zero uncompressed weight. Depending on the compression form used, there will be cases where each weight cannot exist in every possible position. For example, consider again the previous example where two non-zero weights have been retained from the initial set of four values. In this case, the weight might appear to belong to any of the four positions. However, some combinations of positions are impossible. For example, if the first weight belongs to the fourth and last position, there is no place to place the second weight. Similarly, if the second weight belongs to the first position, there is no place to place the first weight. By recognizing this, it is possible to limit the number of inputs the multiplexer circuit can consider, and therefore the multiplexer circuit can select only from a subset of activation values ​​for a particular position.

[0055] In some examples, a data processing apparatus is provided, comprising: an input circuit configured to receive a matrix in a first format including a plurality of values; an output circuit configured to output the matrix in a second format including the plurality of values; and an adjustment circuit configured to perform a modification of the matrix from the first format to the second format, wherein the second format is computationally continuous for a data processing apparatus performing matrix multiplication using the matrix, the data processing apparatus having a first vector register and a second vector register configured to be partitioned in dynamic space and dynamic time.

[0056] To rapidly input and output data to and from the previously mentioned data processing apparatus (characterized by vector registers that are partitionable in both dynamic space and dynamic time), it may be necessary to provide the data in a given format. This enables matrix multiplication instructions to execute efficiently by providing data to them in a manner that allows the operation to proceed without having to scan the data and parse it into a form in which the execution can occur in a parallel manner. This can be achieved by an additional data processing apparatus that performs a preprocessing step to convert the data (e.g., the values ​​or weights of a matrix or model) from a first format to a more suitable second format. This second format makes computationally continuous for the data processing apparatus having the first vector register and the second vector register. This allows the operation to occur without the need for parsing during execution. In other words, the elements of the model in the second format can be taken and processed (operated) without reordering.

[0057] In some examples, the value comprises a first group and a second group, each of which is multiplied by a single activation value in the matrix multiplication; in the first format, the first group forms a first consecutive grouping, and the second group forms a second consecutive grouping; and the adjustment circuit is configured to interweave the value of each of the first and second groups by performing a "z"-shaped scan on the first and second groups to produce the second format. Thus, the value (e.g., weights) is modified such that all values ​​for an input (e.g., a model) are (possibly repeatedly) split, rather than all values ​​being together. For example, in the second model, the weights of the first group may be followed by the weights of the second group, and then by the weights of the first combination, in an alternating manner. When the first and second groups are represented as a matrix (with columns and rows), the performed "scan" thus forms a "z" shape. By presenting the weights in this way, the weights can be input into the processing circuit in such a way that the activation values ​​of a group can be applied (at once) to the elements of the first and second groups.

[0058] In some examples, the first and second groups are interleaved in blocks of size p in this second format; and p equals the number of activation values ​​multiplied simultaneously in the matrix multiplication, where p > 1. This interleaving does not necessarily involve alternation between the first and second groups—one value following another. In some examples, a block of elements from the first group is followed by a block of elements from the second group, and then several elements from the first group. The number of elements can be the same each time; for example, p elements from the first group followed by p elements from the second group, and so on. The value p can be chosen to correspond to the number of multiplications that can occur simultaneously in the matrix multiplication. For example, if p = 4, four values ​​from the first group can be followed by four values ​​from the second group, then four values ​​from the first group and four values ​​from the second group, and so on, until all values ​​have been repositioned in this manner.

[0059] In some examples, the adjustment circuit is configured to interleave the first group and the second group into a block of size p by performing double-layer interleaving, in which the "z"-shaped scan is performed p times on the first group and the second group. For example, the "z"-shaped scan may take values ​​from a first group of the first group and values ​​from a second group of the second group. This may then be followed by another set of values ​​from position 1+q of the first group and another set of values ​​from position 1+q of the second group, followed by another set of values ​​from position 1+2q of the first group and another set of values ​​from position 1+2q of the second group, followed by another set of values ​​from position 1+3q of the first group and another set of values ​​from position 1+3q of the second group. These together form the first "z"-shaped scan. The next "z"-shaped scan can then include a set of values ​​of similar size, derived from position r in the first set, followed by r in the second set, then r+q in the first set and r+q in the second set, then r+2q in the first set and r+2q in the second set, and then r+3q in the first set and r+3q in the second set. This can then be followed by a third "z"-shaped scan, and so on. Thus, each "z" takes a value (e.g., a weight) to be multiplied by the same subset (e.g., activation values), where the value "q" represents the number of activation values ​​present.

[0060] In some examples, in this second format, within q consecutive blocks of size p, the value is grouped according to its position within the first and second groups. The first and second groups of values ​​are interleaved to form q blocks of size p. Throughout the q blocks, the values ​​from each group maintain their ordering relative to other values ​​in that group, as they have in the first format. This allows the generation of the predicate mask to proceed in the same manner as other instructions. Specifically, since the relative ordering within each group is not disturbed, a loop counter can be used to compare the number of elements of the vector (or matrix) executed so far with the total number of elements in that vector (or matrix). This can then be used to generate a predicate mask to indicate the element to be executed in the next iteration. Within a single tick of the data processing device (which has a first vector register and the second vector registers), it is expected that all values ​​(e.g., weights) within q blocks of size p will be processed. That is, q*p values ​​can be processed within a single tick.

[0061] In some examples, each weight has an index j within group i; and in this second format, within every q blocks of size p, the value is first sorted according to its index j and then according to its group i. A value (e.g., a weight) can be defined as having a position (i,j), where i is the group number or channel number to which the value (e.g., weight) belongs, and j is the index number within that group number or channel. In this second format, within a given q blocks of size p, all values ​​with j=0 are first grouped together, and these values ​​are sorted in order of i (among them). Then, all values ​​with j=1 are grouped together, again sorted in order of i (among them), and so on.

[0062] In some examples, the adjustment circuit is configured to perform the modification in a single stage. That is, no intermediate format is generated between the first format and the second format. Instead, all reordering is done at once. In other implementations, intermediate formats may be generated. For example, a form of interleaving (e.g., a "z" scan) may be performed, and then, given that the sorting of these values ​​has already produced, further reordering may be performed within every q blocks of size p.

[0063] In some examples, the value includes compression weights; the output circuitry is configured to output compressed data; and the compressed data includes multiple items indicating how the compression weights can be decompressed into decompressed weights. The weights in the first format model may already be compressed weights, in which case the compressed data is provided along with the model. Alternatively, the weights in the first format model may be decompressed weights. In this case, the data processing apparatus can perform compression and generate the compressed data.

[0064] In some examples, the compressed data can be applied to multiple matrix multiplication instructions; and at least some of these matrix multiplication instructions indicate different compression weights for each other. Thus, the compressed data can be applied to multiple different matrix multiplication instructions.

[0065] In some examples, the compressed data can be applied to more than two sets of compressed weights. When the compressed data used for a weight is smaller than the weight itself, it is possible to store compressed data for multiple matrix multiplication instructions (each of which may be associated with a different weight). There are two ways to use this fact to store large amounts of compressed data. One is to store a larger number of weights for a small number of channels (weight reassemblies). The other is to store a smaller number of weights for a large number of channels. By storing the latter (i.e., by storing compressed data associated with more than two channels), it is possible to reduce register pressure. In other words, it is possible to reduce the number of registers required to store the compressed data while limiting the number of times compressed data is loaded into registers.

[0066] In some examples, the items are sorted in the compressed data according to the ticks in which the items are used within the multiple matrix multiplication instructions. For example, items used in the first tick of any matrix multiplication operation in the matrix multiplication operation are grouped together, then items used in the second tick of any matrix multiplication operation in the matrix multiplication operation are grouped together, and so on.

[0067] In some examples, the items are sorted within the compressed data such that items used in the same beat of the same single matrix instruction operation are adjacent. Therefore, continuous data can be extracted from the compressed data in a single operation for each beat of each operation.

[0068] The specific implementation scheme will now be described with reference to the accompanying drawings.

[0069] Figure 1 An example of a data processing apparatus 2 for processing support vector instructions is schematically illustrated. It should be understood that this is a simplified diagram for ease of explanation, and in practice, the apparatus may have variations not shown for simplicity. Figure 1Many elements are shown in the diagram. Device 2 includes processing circuitry 4 for processing data in response to instructions decoded by instruction decoder 6. Program instructions are retrieved from memory system 8 and decoded by instruction decoder to generate control signals that control processing circuitry 4 to process the instructions in a manner defined by the architecture. For example, decoder 6 may interpret the opcode of the decoded instruction and any additional control fields of the instruction to generate control signals that cause processing circuitry 4 to activate appropriate hardware units to perform operations, such as arithmetic operations, load / store operations, or logical operations. The device has a set of registers 10 for storing data values ​​to be processed by processing circuitry 4 and control information for configuring the operation of processing circuitry. In response to arithmetic or logical instructions, processing circuitry 4 reads operands from registers 10 and writes the results of these instructions back to registers 10. In response to load / store instructions, data values ​​are transferred between registers 10 and memory system 8 via processing circuitry. Memory system 8 may include one or more cache levels and main memory.

[0070] Register 10 includes a scalar register file 12, which includes multiple scalar registers for storing scalar values, each containing a single data element. Some instructions supported by the instruction decoder 6 and processing circuitry 4 are scalar instructions that process reading scalar operands from scalar register 12 to produce scalar results written back to the scalar register.

[0071] Register 10 also includes a vector register file 14 comprising multiple vector registers, each storing a vector value comprising multiple data elements. In response to a vector instruction, instruction decoder 6 controls processing circuitry 4 to perform multiple vector processing lanes on the corresponding elements of a vector operand read from one of the vector registers in vector register 14, thereby producing a scalar result to be written to scalar register 12 or another vector result to be written to vector register 14. Some vector instructions may produce vector results from one or more scalar operands, or may perform additional scalar operations on scalar operands in the scalar register file and perform vector processing lanes on vector operands read from vector register file 14. Therefore, some instructions may be hybrid scalar-vector instructions, where at least one of one or more source registers and the destination register of the instruction is vector register 14, and the other of the one or more source registers and the destination register is scalar register 12. Vector instructions may also include vector load / store instructions that transfer data values ​​between vector register 14 and a location in memory system 8. The load / store instruction may include: a consecutive vector load / store instruction, for which the location in memory corresponds to a consecutive address range; or a scatter / aggregate type vector load / store instruction, which specifies multiple discrete addresses and controls the processing circuit 4 to load data from each of these addresses into the corresponding element of the vector register, or to store data from the corresponding element of the vector register into a discrete address.

[0072] Processing circuitry 4 can support the processing of vectors with a range of different data element sizes. For example, the 128-bit vector register 14 can be partitioned into sixteen 8-bit data elements, eight 16-bit data elements, four 32-bit data elements, or two 64-bit data elements. The control register within register bank 10 can specify the currently used data element size, or alternatively, this can be a parameter for a given vector instruction to be executed. Therefore, this register can be considered as being partitioned in dynamic space.

[0073] Register 10 also includes a plurality of control registers for controlling the processing of processing circuitry 4. These registers may include, for example: a program counter register 16, which stores a program counter address indicating the address of the instruction corresponding to the current execution point being processed; a link register 18, which stores the return address to which processing is directed after a disposal function call; a stack pointer register 20, which indicates the location of the stack data structure within memory system 8; and a tick status register 22, which stores tick status information, which will be described in more detail below. It should be understood that these are just some of the types of control information that can be stored, and in practice, a given instruction set of the architecture may store many other control parameters as defined by the architecture. For example, a control register may specify the overall width of a vector register, or it may specify the current data element size for a given instance of vector processing.

[0074] The architecture supports overlapping vector instructions. For example, the load instruction VLDR and the vector multiply-accumulate (VMLA) instruction can be executed concurrently, even if there are data dependencies between these instructions. This is because each vector instruction executes only a subset of the values ​​at a time, and once the first instruction has operated on the first subset of values, the second instruction can then use those values; therefore, VMLA execution can begin before VLDR execution has finished. By allowing instruction overlap, expensive execution blocks (e.g., multipliers) can remain active for longer periods.

[0075] like Figure 2 As the example shows, the extent of instruction overlap can be possible. Execution of vector instructions is divided into portions called "beats," where each beat corresponds to the processing of a portion of a vector having a predetermined size. A beat is an atomic portion of a vector instruction that may be executed entirely or not at all, and cannot be partially executed. The size of the portion of the vector processed in a beat is defined by the architecture and can be any share of the vector. Figure 2In the example, a tick is defined as the atomic processing of a channel at one-quarter of the vector length, such that there are four ticks per vector instruction. Obviously, this is just an example, and other architectures may use different tick channel widths (e.g., 64-bit channels) or tick numbers (e.g., two or eight). The size of the portion of the vector corresponding to a tick can be the same as, larger than, or smaller than the size of the data element of the vector being processed. Therefore, even if the element size varies with implementation or runtime between different instructions, the tick is a fixed width for vector processing. If the portion of the vector processed in a tick comprises multiple data elements, the carry signal can be disabled at the boundaries between the corresponding elements to ensure that each element is processed independently. If the portion of the vector processed in a tick corresponds to only a portion of the elements, and the hardware is insufficient to compute several ticks in parallel, the carry output generated during one processing tick can be used as a carry input to the next processing tick, such that the results of the two ticks together form the data element.

[0076] like Figure 2 As shown, different microarchitectural implementations of processing circuit 4 can execute different numbers of ticks in one “tick” of the abstract architecture clock. Here, a “tick” corresponds to a unit of architectural state progression (e.g., in a simple architecture, each tick may correspond to an instance of updating all architectural states associated with the execution of an instruction, including updating the program counter to point to the next instruction). Those skilled in the art will understand that known microarchitectural techniques (such as pipelining) may mean that a single tick may require multiple clock cycles to execute at the hardware level, and that in fact, a single clock cycle at the hardware level can process multiple parts of multiple instructions. However, this microarchitectural technique is not visible to software because ticks are atomic at the architectural level. For simplicity, such microarchitectural techniques are omitted during the further description of this disclosure.

[0077] like Figure 2 As shown in the lower example, some implementations can schedule all four ticks of a vector instruction within the same tick by providing sufficient hardware resources to process all ticks in parallel within a single tick. This is suitable for higher-performance implementations. In this case, no overlap between instructions is required at the architectural level because the entire instruction can be completed in a single tick.

[0078] On the other hand, more area-efficient implementations can provide narrower processing units that can handle only two ticks per tick, and as... Figure 2 As shown in the intermediate example, instruction execution can overlap with the first and second cycles of the second vector instruction (which are performed in parallel with the third or fourth cycle of the first instruction), wherein those instructions are executed on different execution units within the processing circuitry (e.g., in...). Figure 2In this context, the first instruction is a load instruction executed using load / store unit 200, and the second instruction is a multiply-accumulate instruction executed using MAC 202.

[0079] A more energy / area efficient implementation could provide a narrower hardware unit that can process only a single tick at a time, where each tick can handle one tick, with instruction execution as follows: Figure 2 The top example shows overlapping and interleaving a beat.

[0080] It should be understood that Figure 2 The overlaps shown are just a few examples, and other specific implementations are possible. For example, some specific implementations of processing circuit 4 could support multiple instructions being issued in parallel in the same tick, resulting in a large instruction throughput. In this case, two or more vector instructions that start together in a loop might have some ticks that overlap with two or more vector instructions that start in the next loop.

[0081] In addition to varying the overlap amount depending on the specific implementation to scale to different performance points, the overlap amount between vector instructions can also vary runtime between different execution instances of vector instructions within the program. Therefore, processing circuit 4 can be provided as follows: Figure 1 The illustrated clock control circuit 30 is used to dynamically control time division, for example, the timing of executing a given instruction relative to a previous instruction. This provides microarchitecture with the freedom to select non-overlapping instructions in certain extreme cases that are more difficult to implement or based on the resources available for the instructions. For example, if there are consecutive instructions of a given type (e.g., multiply-accumulate) that require the same resources and all available MAC or ALU resources are already in use by another instruction, there may not be enough free resources to begin executing the next instruction, and therefore the issuance of the second instruction can wait until the first instruction has completed, rather than overlapping.

[0082] Figure 3 A data processing device 100 in the form of a preprocessor is shown. The preprocessor 100 takes models from a first input, which are received by input circuitry 120. These models include, for example, weight reassemblies each associated with the input to be tested by the neural network. An adjustment circuitry 130 then modifies the models to produce models in a second format, which are output by output circuitry 140. The adjustment circuitry 130 adjusts these models such that, in this second format, the weights are rearranged and can be inserted into and processed by a second data processing device 2 in such a way that re-parsing or "randomly" (rather than sequentially) access to these models is not required. In this way, the second format is computationally continuous for the second data processing device 2.

[0083] In the second data processing device 2, these models are received and stored in multiple vector registers R in the first group or library 14A of the vector registers. A1 R A2 R A3 In the middle. The second group or the register R of library 14B. B1 R B2 R B3 The corresponding set of activation values ​​is stored. Decoding circuit 6 receives matrix multiplication instructions and thus performs matrix multiplication operations. The neural network performs operations by multiplying the weights and activation values. Typically, at a single neuron, a set of weights (a vector of weights) is multiplied by a set of activation values ​​(a vector of activation values).

[0084] In the second format model, the weights and activation values ​​each have different bit widths. For example, each weight can be 4 bits, and each activation value can be 8 bits. By using a small bit width for the weights, the model can be made smaller. However, by keeping the activation values ​​large, it is still possible to produce good results from the neural network. Additionally, the data processing device 2 can use smaller weights to achieve improved throughput. Specifically, by carefully arranging the weights of the two channels of the model (as achieved by adjusting circuit 130), the two channels can be processed simultaneously or in parallel. This process is illustrated in the following figures.

[0085] Figure 4A This example illustrates the effect of the data processing device during preprocessing and execution of matrix multiplication instructions. Here, the matrix multiplication instruction takes the following form:

[0086] VMLAMAV.S8 RdaA,RdaB,Qn,Qm

[0087] Where “S8” refers to the fact that the activation value is 8 bits, “RdaA” is the location of the scalar register in which the first set of result bits is placed, “RdaB” is the location of the scalar register in which the second set of result bits is placed, “Qn” is the location of the vector registers in a series of processing channels from which the activation value is placed, and “Qm” is the location of the vector registers in these processing channels from which the weights are placed.

[0088] Preprocessing is performed by the preprocessing data processing unit 100. The input channels are shown as input B. Each channel consists of 16 weights, each four bits. There are two models within this input, and therefore a total of 32 weights. The other input A is a group of 16 activation values. Each activation value is 8 bits. Therefore, the nodes of the neural network multiply each activation value by its corresponding weight (e.g., weight 1 multiplied by activation value 1, weight 2 multiplied by activation value 2, and so on). The results of each multiplication of the model's weights are summed. Thus, for both channels, two results (16-bit or 32-bit) are produced in a pair of scalar registers at C (corresponding to RdaA and RdaB in the instruction definition above).

[0089] In this technique, weights belonging to two channels are interleaved to produce channels in a second format. This is achieved through a "z"-shaped scan across the weights. The "z" shape arises from the fact that the scan proceeds through consecutive weights from a first group in the first channel, then consecutive weights from a second group in the second channel, followed by consecutive weights from a third group in the first channel (which immediately follows the consecutive weights of the first group), and then consecutive weights from a fourth group (which immediately follows the consecutive weights of the second group), and so on. Thus, the "z"-shaped scan can be viewed as an ordered interleaving of blocks of values ​​within a channel. In this example, each group has four weights, resulting in a "striped" combination of channels 400 presented in 128 bits. The term "p" refers to the number of weights in each block. This also corresponds to the number of activation values ​​multiplied simultaneously. Meanwhile, the term "q" is used to indicate the number of channels for which entries are processed simultaneously.

[0090] In the first beat (beat 0), a set of multipliers 410 multiplies each weight of the first and second channels by the activation value. The results of the multiplications of the first channel are summed and stored in a scalar register. The results of the multiplications of the second channel are summed and stored in a second scalar register (in some embodiments, a single register may be used to store two multiply-accumulates). Because the weights of the first and second channels are interleaved, it is possible to use the same set of activation values ​​for both sets of multiplications without needing any data from beyond the 32-bit channel boundary associated with beat 0. This is important because the activation values ​​have a higher bit width (8 bits versus 4 bits) than the weights, and therefore a smaller number of activation values ​​can be stored in the vector register compared to the number of weights. In a series of four beats, each weight of the two channels is multiplied by this set of activation values, with half of the multiplications in each beat directed to the first channel and the other half directed to the second channel.

[0091] Because the same set of activation values ​​is used for each set of multiplications, a small number of load operations can be maintained. In fact, in some cases, the ratio of load operations to computation operations can be maintained at or close to 1:1.

[0092] It should be understood that the effective operation performed is therefore matrix multiplication. This is because each weight in the two channels is multiplied by a single set of activation values, and the results are summed. In other words, and... Figure 4A In the example, a (1×16) matrix is ​​multiplied by a (16×2) matrix (or vector). The second input in these inputs is two-dimensional, and therefore the operation under discussion is matrix multiplication. However, Figure 4A The illustrated apparatus performs matrix multiplication through the interleaving of (2×16) inputs. This enables efficient multiplication by avoiding path boundary crossings within a beat-based context.

[0093] When using vector processing, a "predicate mask" can be used to indicate that the vector is not a specific entry to be processed. For example, it has already been... Figure 4A The code assumes that the matrix multiplication parameters fit exactly within a tick. Alternatively, if the multiplication to be performed is a 1×17 matrix multiplied by a 17×1 matrix, this might not fit within all ticks. Therefore, the operation will be performed iteratively, multiplying the first set of elements in the first iteration and the second set in the second iteration. This is fine as long as the total number of elements fits and the register is fully filled within a multiple of the register size. However, if this is not the case, it is appropriate to provide a mechanism to indicate that a particular element should not be processed in a tick. Without this mechanism, "garbage" data that remains will be computed and will affect the final result.

[0094] Figure 4A Four 32-bit beats are shown. This allows for the multiplication of a 16×2 matrix with 4-bit weights. However, if these weights form a 17×2 matrix, this would have to be done in two iterations, with the second iteration multiplying only the 1×2 matrix of weights. In this case, a predicate mask is used to indicate that only a portion of one of these beats (the portion holding the 2×1 matrix elements) should be processed in the second iteration.

[0095] Predicate masks can be generated using a simple loop counter, since it is known a priori how many elements are processed per operation, and the total number of elements to be processed is known. Therefore, the run count of the remaining elements to be processed can be preserved. This can then be used to generate the predicate mask in each iteration. In these examples, each multiplication with the activation value is performed twice (once per channel), and therefore each entry in the predicate mask refers to a pair of values. Thus, the predicate mask in the previous example could be a single entry followed by 15 zeros (one entry for each element multiplied twice, i.e., each activation value).

[0096] However, regarding Figure 4A The illustrated 2×4 matrix "z"-shaped reshaping rearranges the elements, and therefore the weights no longer appear in order. Therefore, additional steps are taken to allow the generation of predicate masks using a loop counter.

[0097] about Figure 4B The first method is illustrated. In this technique, each channel of the model is filled with empty entries (0), which have no effect on the computation being performed. This filling is performed such that each channel is expanded to a multiple of the total beat width (in...). Figure 4A (In the case of 32×4). The predicate mask generated for these instructions will now be exactly 1 for these instructions because there is no "dangling" of elements. That is, in integer iterations, the entire set of elements or weights fits exactly in the beat.

[0098] The drawback of this method is that it can bloat the channel-based model with unnecessary additional data. Furthermore, it leads to the unnecessary execution of null value calculations. Even if these null value calculations may not affect the final result and may even be easy to perform, they still require some processing.

[0099] Figure 4C An example is shown where, for instance, in-cycle reshaping is performed via preprocessor 100 to avoid... Figure 4B Alternatives to the model bloat and unnecessary computations that occur in [the context of the previous sentence]. Figure 4C In this context, the intra-cycle reshaping is shown as occurring on the data structure that has already been produced as a result of a "z"-shaped reshaping (q=2, p=4). However, it is certainly possible to perform both reshapings in a single step. Figure 4C The primary purpose is to illustrate the properties of intra-beat reshaping. Specifically, intra-beat reshaping rearranges the elements within each beat, as determined by "z"-shaped reshaping. In these examples, eight weights (elements) are processed together in a beat, and therefore intra-beat reshaping occurs within each block of the eight weights. Intra-beat reshaping groups the elements within a beat together based on their order within each channel. In other words, the weights W processed together in a given beat... i,j (Where i is the channel index and j is the weight index within that channel) First, it is reordered according to j and then according to i. Therefore, as... Figure 4C As shown, the element W processed in the first beat 0,0 W 0,1 W 0,2 W 0,3 W 1,0 W 1,1 W 1,2 W 1,3 Reordered as W0,0 W 1,0 W 0,1 W 1,1 W 0,2 W 1,2 W 0,3 W 1,3 .

[0100] Predicate mask generation can now continue in the manner previously described using a loop counter. If there are three "suspended" weights per channel (e.g., W...),... 0,0 W 0,1 W 0,2 W 1,0 W 1,1 W 1,2 If the predicate mask is set to three ones followed by twelve zeros, then the predicate mask can be set to three ones followed by twelve zeros. As before, each entry in the predicate mask corresponds to multiple weights equal to the number of channels, because each corresponding weight for each channel is multiplied by the same activation value. Note that in the case of in-beat reshaping (i.e., only looking at the structure generated after a "z"-shaped rescan), there are weights that will be multiplied but are not the desired ones. For example, weight W 0,3 It will be multiplied (it is not in the suspend position), and the weight W 1,2 It will not be multiplied (it is in suspension).

[0101] Figure 5 The multiplication performed by multiplication circuit 410 and the relationship between the inputs are illustrated in more detail. In this example, eight weights, each with four bits, are accessed from the first vector register Qm in one clock cycle, for a total of 32 bits. Similarly, in the same clock cycle, four activation values, each with eight bits, are accessed from the second vector register Qn, for a total of 32 bits. These eight weights consist of four weights from the first channel and four weights from the second channel. Therefore, these four activation values ​​are multiplied by each of the four weights in the first group and each of the four weights in the second group. The results of the multiplications are added together. That is, the first group of four multiplications is added together by the first adder 500, and the second group of four multiplications is added together by the second adder 510. The results are stored in the first scalar register RdaA and the second scalar register RdaB, respectively.

[0102] In each subsequent beat, the activation values ​​and weights of different groups are multiplied together. Weights belonging to the first channel are added together using the same adder 500, and weights belonging to the second channel are added together using the same adder 510. In each case, the result of the addition is added to the result stored in scalar registers RdaA and RdaB. Therefore, at the end of four beats, scalar registers RdaA and RdaB store the result of the matrix multiplication.

[0103] Because the weights are smaller than the activation values ​​(4 bits and 8 bits in this example), more weights can be stored in the same space. By organizing the weights of multiple channels in the freed-up space, and by carefully organizing these weights, additional weights can be processed simultaneously.

[0104] In this example, a pair of scalar registers, RdaA and RdaB, are used for output data. In practice, by assuming that one accumulator / adder 500 always outputs data to an even-numbered register and another accumulator / adder 510 always outputs to an odd-numbered register, they will always map to different memory banks within the register file (when the register file is partitioned into even / odd banks on the microarchitecture), thus reducing the number of connections or ports required to implement the second scalar register.

[0105] Because this technique is applied to a clock-based architecture (where registers can be partitioned in both time and space as needed), it is possible to use the same hardware to process larger or smaller elements as needed.

[0106] Figure 6 A similar arrangement is shown where each weight is only 1 bit. Therefore, in this example, there are a total of 128 bits associated with eight different channels (16 bits per channel). In input B, these bits are arranged as a (64×2) matrix.

[0107] In this example, the preprocessing stage achieves double-layer interleaving by repeatedly performing "z"-shaped scans to produce a striped output of 600. Figure 5 Similar to the previous example, each "z"-shaped scan uses weights from a group of data from each channel. However, in Figure 6 In the example, the "z"-shaped scan repeatedly skips due to the size of each weight and the channel vector. That is, after taking four weights from the first channel (in the first row of the matrix) and four weights from the second channel (in the second row of the matrix) to produce an output of 600, the scan jumps to the first four weights of the third channel, which is further below the first row. Therefore, the scan is discontinuous because several entries are skipped within a specific iteration of the scan. Then, for the next four entries, the second iteration returns to near the top of the first column (in fact, the entries in this group can start from the end of the first group of entries taken from the first column (i.e., from entry 5 in the first column)).

[0108] During the execution by the data processing device 2, in the processing Figure 4A Compared to the 4-bit matrix stored in B, there are many more individual (and narrower) B inputs. Due to hardware area efficiency considerations, the multiplier circuit 610 is expected to... Figure 4ASimilar to or identical to the multiplier circuit 410, 610 also consists of eight multipliers, which are intended to process data from two channels and store the results in two output scalars. This means that each instance of instruction execution must select eight 1-bit inputs from the 32 available 1-bit values ​​from B. The selection register imm 620 is used to store which consecutive eight bits (four bits from the two channels) are to be processed by the multiplication circuit 610 for a specific clock cycle. Figure 5 As in the example, these bits are loaded into the multiplication circuit 610 and multiplied by the activation value, with the result being provided to the scalar registers RdaA and RdaB.

[0109] As each weight is loaded into the multiplication circuit 610, it is expanded (e.g., expanded to four bits). That is, in the case of unsigned or signed positive numbers, the most significant bit is filled with 0, and in the case of signed negative numbers, the most significant bit is filled with 1. In the case where the second element is one bit and the number {-1, 1} is represented by the value {1, 0}, 1 will be expanded to 0b1111, and 0 will be expanded to 0b0001.

[0110] It should be understood that in this example, only one pair of scalar registers is still provided. This is because the operation effectively handles the vector-matrix product of 16 8-bit values ​​(1×16) from A and 32 1-bit values ​​(16×2) from B, which has two result terms (1×2). Again, this 16×2 from B is only a portion of the 64×2 available terms stored within it. In practice, typical usage would see four vector registers loaded with consecutive vectors, so a virtual 1x64 vector (distributed across these four vector registers) can be used to multiply all of the contents of the 64×2 stored in B. The operation would then execute the instructions four times using different A registers (in consecutive order) and different matching immediate terms (0×0, 0×1, 0×2, 0×3) to appropriately multiply and accumulate each 1×16 segment of the larger virtual 1×64 of B with its corresponding 16x2 segment. This 4-instruction sequence will perform a vector matrix product of 64 8-bit values ​​(1×64) and 128 1-bit values ​​in B (64×2), resulting in two accumulated terms (1×2). For example, regarding... Figure 14A and Figure 14B As shown, other parameters of the instruction (such as the address of register A) can be used to determine the immediate term.

[0111] It should be noted that reshaping within a beat also applies. Figure 6 In the case of [the previous sentence], each weight is only 2 bits in size.

[0112] Figure 7 A variant instruction is shown where the weights have different bit widths due to compression. Here, the instruction takes the following form:

[0113] VMLADBBAV.S8 RdaA,RdaB,Qn,Qm,Q0,# <sel>

[0114] These parameters correspond to the instructions previously illustrated. "Q0" stores the corresponding compressed data in the form of a sparse mask or multiplexer index. Meanwhile, "sel" (which is an example of a protected compressed data selector) is an immediate value indicating which sparse mask is taken from Q0. Specifically, as will be the case regarding... Figures 13A to 13D As shown, since the compressed data is significantly smaller than the weighted data, Q0 can store compressed data (e.g., a sparse mask) for multiple operations. For this reason, the sel parameter is used to indicate which part of the data needs to be compressed for the current operation. The compressed data can be stored in registers 730 and 740 (e.g., one register per channel to be decompressed per tick).

[0115] A matrix can be considered sparse if a subset (e.g., half) of its values ​​are zero (rather than non-zero). However, for the purposes of matrix multiplication, zero values ​​have no effect. That is, any multiplication by zero always results in zero, and adding that result to a series of other multiplications has no consequence (x + 0 = x). Therefore, to some extent, zero values ​​in a matrix can be ignored as long as the relative positions of the other values ​​are properly preserved. Thus, one way to compress a matrix is ​​to erase zero values, pack the remaining values ​​together, and provide a sparsity mask to reveal the true location of the remaining values.

[0116] Therefore, in Figure 7 In the example, input A again consists of 16 8-bit activation values. Meanwhile, input B consists of 32 weights, each 8 bits, with 16 weights belonging to each of the two channels. However, these weights have been compressed into 16 8-bit weights and 8 4-bit sparse masks, which indicate how the compressed form is converted to the uncompressed form. In this example, a compression ratio of 1.6x is achieved. That is, half the entries in the input channels are zero (2x), plus the overhead of decompressing the sparse masks. Therefore, the average number of bits used to represent the activation values ​​is reduced because the number of bits used to represent zero is zero (assuming the sparse mask itself is not included).

[0117] As in Figure 4A In the example, a "z" scan is performed to interweave the weights of the two channels into a single striped form 700 (in other words, the model is placed in a second format). Like... Figure 4A The situation is similar, with each group having 16 bits. That is, 16 bits are taken from the first channel, then 16 bits from the second channel, and then the next 16 bits from the first channel, and so on. The execution is again distributed across four beats. In each beat, four weights (two weights from each channel in the network) are multiplied by four activation values. In effect, due to the compression ratio, this is equivalent to four weights multiplied by four activation values, where two of these weights are zero (and therefore the multiplication doesn't need to occur).

[0118] A series of multiplexers 710 are used to reposition activation values ​​to compensate for the removed zero weights. This repositioning is achieved using sparse mask information and will be related to... Figure 8 and Figure 9 A more detailed example follows.

[0119] Once the location is determined, multiplication is performed using multiplication circuit 720, and the result is provided to a pair of scalar registers RdaA and RdaB as previously discussed.

[0120] Because the same set of activation values ​​is used for each set of multiplications, the number of load operations can be kept small. In fact, in some cases, the ratio of load operations to computation operations can be maintained at or close to 1:1, where one or more additional loads are required for sparse masks that indicate how compression has been performed.

[0121] Regarding the instructions described above, predicate generation can be performed in the conventional manner, provided that padding (to match) has been performed before the weights are compressed in the preprocessor 110. Figure 4B (Similar method) or has already performed intra-beat reshaping (to match) Figure 4C (Similar methods).

[0122] Figure 8 This illustrates how matrix (or vector) compression can be performed. In this case, the values ​​"x" and "y" are treated as non-zero values. Therefore, for Figure 8 Any of the six different (1×4) matrices illustrated can be reduced to a single (1×2) matrix by removing zero values.

[0123] Therefore, compressed data can take the form of a (sparse) mask that indicates the location of non-zero values. For example, compressed data 0010 might mean that the first non-zero value is at position 00 (i.e., 0) and the second non-zero value is at position 10 (i.e., 2). Assuming a right-to-left numbering scheme, this would therefore correspond to... Figure 8 The second form shown.

[0124] There are several other forms that can be used to compress data. For example, data can be compressed into <value>><position> pairs. For data in a particular (known) format, knowledge of the format can be used to perform further compression. For example, if it is known that the first position will always be zero, the compressed data can be reduced to two bits to indicate the position of the second zero. Other techniques will be known to a person skilled in the art.

[0125] Figure 9 The operation of the multiplexer circuit 710 is shown in more detail. In this example, Qm contains weights (in this case, these weights have been compressed to remove zero values), and Qn contains the activation values. Sparse mask data from register Q0 is then included to indicate where the removed zeros should be located, such that the remaining non-zero weights are multiplied by the correct activation values. Additionally, recall that the weights are associated with two different channels. The first two weights belong to the first channel, and the latter two weights belong to the second channel. Multiplexer 920 is provided to select the sparse mask associated with the current beat (or subbeat). That is, in this example, only 32 bits of register Q0 (8 bits per beat) are needed to store the sparse information.

[0126] In this example, each multiplexer in the multiplexer circuit 710 receives a 2-bit switching signal to indicate which activation values ​​from Qn (available within the current tick) should be adopted. Unadopted activation values ​​are those that will be multiplied by zero weight (displayed by a sparse mask). Therefore, the eight-bit sparse mask is divided into four two-bit batches, each batch acting as a switching signal to one of these multiplexers to determine which activation value it should receive. Since the sparse mask may be different for each of the two channels, entirely different values ​​may be provided to each multiplexer.

[0127] As before, the result of the multiplication is added together by adders 900 and 910 and stored in scalar registers RdaA and RdaB.

[0128] Figure 10 An example is shown where a set of weights (y,0,x,0) belonging to the first channel is multiplied by a set of activation values ​​(a,b,c,d). Theoretically, the multiplication that occurs is (ya) + (0b) + (xc) + (0d). Therefore, this is equivalent to ya + xc. In another example, the second model has a set of weights (0,y,x,0) belonging to the first model, and these weights are multiplied by the same set of activation values ​​(a,b,c,d). Here, the multiplication that occurs is (0a) + (yb) + (xc) + (0d). This is equivalent to yb + xc. Therefore, for any given channel, any four of these activation values ​​can be used, but only two of these activation values ​​are used per channel. A sparse mask is used to indicate which activation values ​​are used for each channel.

[0129] It should be understood that, although Figure 9 The example illustrates a 4-to-1 multiplexer, but such complexity is unnecessary. This is because not every permutation of activation values ​​is possible in every instance. For example, Figure 11 The possible combinations are shown. Note that there are only three positions that weight "x" can occupy, and there are only three positions that weight "y" can occupy. This is because if weight "x" is placed in... Figure 11 If the fourth (leftmost) position is selected, there will be no space for the weight "x" (the order of the weights is preserved). Similarly, if the weight "y" is placed in the rightmost position, there will be no space for the weight "x" to maintain the order of the weights as presented in the compressed data.

[0130] Generally speaking, when there are n positions for uncompressed weights and m non-zero weights, there are n+1-m positions that each weight can enter, and therefore each position requires (n+1-m)-to-1 multiplexers.

[0131] therefore, Figure 12 The arrangement in which a 3-to-1 multiplexer is used in the multiplexer circuit 1100 is shown.

[0132] It should be understood that the storage required to compress the data may be significantly less than the amount of data required for the weights. For example, when using sparse mask data, four fields, each 1 bit, can be used to represent the non-zero positions of a set of four unpacked weights. If those weights are each 8 bits, then the total four-bit position encoding plus 16 bits (2 × 8 bits, assuming 50% sparsity) of the weight values ​​are decompressed into 32 bits of data, resulting in a compression ratio of 32:20 (or 1.6:1).

[0133] In practice, within the memory system, each mask in a mask for a given channel can be provided at a time. That is, if there are four channels, each with N weights, the mask associated with the N weights of the first channel can be stored before the N weights of the second channel, and then the N weights of the third channel, followed by the N weights of the fourth channel. In practice, loading compressed data of this format for matrix multiplication will require multiple registers to store data belonging to multiple channels, or it will require repeatedly loading mask data for different channels into a single register.

[0134] For example, Figure 13A This illustrates the order of loading and computation operations that occur during matrix multiplication. Here, the term Axx refers to the vector register storing the activation values, and the term Byy refers to the vector register storing the weights. Recall that by reducing the size of the weights (e.g., from 8 bits to 4 bits), each vector register storing the weights can store the weights equivalent to two channels. Therefore, Czz refers to a pair of scalar registers (e.g., RdaA and RdaB), each storing the result of multiplying the activation value of an activation value register by the weight belonging to one channel of a vector register (which stores the weights).

[0135] Figure 13A This illustrates the order in which weights and activation values ​​are loaded into the relevant registers. Additionally, Figure 13A The order in which multiplication occurs is shown.

[0136] use Figure 13A The sorting shown first performs the calculation on a set of weights for a pair of channels (e.g., as shown in the image). Figure 7 (As shown), then move to the next set of weights on the adjacent pair of channels, processing a total of 4 channels. That is, first perform each operation in the computation on the weights of the four channels stored in B00 and B01, and then further proceed to perform the computation on the next set of weights of the four channels stored in B10 and B11. Similarly, consume A00 and A10, and then further advance across the activation value vector to A01 and A11. This continues to B20, B21 and A02, A12 (not shown), etc., until the entire vector of n elements has been processed.

[0137] The above description Figure 7 The behavior of one of these operations is illustrated, for example, multiplying A00 in register Q0 by B00 in register Q4.

[0138] It should be understood that if the sparse data for all weights used in the weights of two channels were stored in a single register, then this would require two registers to store all the mask data for four channels, or it would require a single register into which the mask data is repeatedly loaded and unloaded between the two pairs of channels. For example, after each operation, mask data associated with the other two channels would need to be loaded.

[0139] Note that in these examples, a select register is not required. Instead, the immediate "sel" value is passed with each instruction to indicate which part of the data that instruction needs to compress. (See also: Regarding...) Figure 14A and Figure 14B The exploration revealed that the explicit specification of the immediate value could be predictable because there is a one-to-one mapping between one of the source registers (which holds the compressed matrix data) and the immediate value.

[0140] Figure 13B This example illustrates how the preprocessor 100 reorders the natural memory layout of the mask to alleviate register pressure. Here, it is assumed that each channel contains 64 weights w and there are four channels. Item W i,j This refers to the j-th weight of channel i. In this case, the preprocessor 100 reorders the masks such that it first stores the masks for the first 32 weights of each of the four channels, and then stores the masks for the last 32 weights of each of the four channels. In other words, the masks are reordered so that as many weights from each channel as possible (equal number) are stored in the vector register 14 of the processor 2.

[0141] Therefore, one of the vector registers in vector register 14 is used to store the sparse mask data during execution. Figure 13A After the eight operations shown, the mask data needs to be reloaded. That is, until register B20 needs the mask data.

[0142] It should be noted that, as an alternative, a single vector register can store all 64 weights from both channels. However, as... Figure 13A As illustrated, the second operation processing channels 3 and 4 will require loading mask data for channels 3 and 4, then the third operation will require loading mask data for channels 1 and 2, then the fourth operation will again require loading mask data for channels 3 and 4, and so on. Therefore, one or more additional vector registers will be needed, or redundant loading operations will be performed. Figure 13B The illustrated reordering prevents this from happening.

[0143] Figure 13C It shows in Figure 13B Further reordering can be performed after the reordering performed in the first step. Here, the masks in the register are reordered based on the beat in which they will be used. That is, the masks used in beat 0 are grouped together, then the masks used in beat 1, then the masks used in beat 2, and so on. Grouping can take into account multiple operations. For example, the mask presented at the first beat boundary is... Figure 13A The mask required for the weights used in the first beat of each of the first eight operations. Similarly, the mask rendered at the boundary of the second beat is... Figure 13A The mask required for the weights used in the second beat of each of the first eight operations.

[0144] Figure 13D It shows in Figure 13B or Figure 13C After the reordering is performed in the middle, another reordering can be performed. Figure 13D The reordering performed in the middle is similar to Figure 4C The reordering is performed within the clock. Specifically, the masks within each clock boundary are paired together, allowing for a single 8-bit extraction (of the two masks) of all weights disposed of within a single clock of a single operation. Therefore, in Figure 13D In this context, the mask is sorted by weighted index followed by channel number (within each beat). For example... Figure 7 As shown, the first tick of the first operation will need to be used for weight W. 0,0 To W 0,4 (Right now, Figure 7 The first two elements of the first weighted column in the matrix and W 1,0 To W 1,4 (Right now, Figure 7 The mask for the first two elements of the second weighted column in the first operation. Therefore, the remaining elements from the first beat are reserved for other operations. In the second beat of the first operation, the weight W will be used. 0,4 To W 0,7 (Right now, Figure 7 The third and fourth elements of the first weighted column in the matrix) and W 1,4 To W 1,7 (Right now, Figure 7 The mask is the third and fourth elements of the second weight column in the register. The previously shown sel parameter can be used to indicate which masks in the register are used for a specific operation. Since the sel parameter indicates the required mask, and since the required mask depends on the weights used, the same sel parameter will be used for each weight register. In fact, Figure 13A The register number shown for register B can be directly converted to a sel number. Therefore, the required mask index will be:

[0145] <beat number> x 8+ <sel>x 2 and

[0146] (<beat number> x 8+) <sel>x 2)+1

[0147] Note that the above figure illustrates an example of compression, where a sparse mask in bit field form is used to indicate the location of non-zero values ​​throughout the matrix. Other forms of sparse masks or compression can also be used. For example, non-zero values ​​and their indices within blocks can be stored. Thus, the values ​​"000110" "11" could mean that the value 6 belongs to position 3.

[0148] It should also be understood that, although Figures 13A to 13D Several different steps for rearranging compressed / sparse masked data are shown, but all of these steps may be performed in a single rearrangement operation, or only a subset of these steps may be performed.

[0149] Figure 14A This illustrates a technique that can be used to reduce the number of bits required to encode instructions such as matrix multiplication instructions. This technique is also applicable to other types of instructions. In this example, the least significant bit of the address of one source register (e.g., a register used to store weights) is used to determine the addresses of other source registers, the destination register, and the sel value. Figure 14B In the example, the least significant bit is 09 (hexadecimal). These bits are then combined with the stub address, which is the bit provided for the destination register (0x111111), to form the full address of the first destination register (0x11111109). Similarly, the bits of the second destination register (0x222222) form the second stub address, and this second stub address is combined with the least significant bit 09 to form the full address of the second destination register (0x22222209). Similarly, the bits provided for other source registers (0x333333) form the third stub address, which is combined with the least significant bit 09 to form the full address of other source registers (e.g., the source register holding the activation value) (0x33333309). Finally, bit 09 directly corresponds to the sel value.

[0150] In this example, 8 bits are represented by the encoding of each of the two source registers, the destination register, and the sel value, for a total of 24 bits stored (using only 8 bits to represent 32 bits would store 32 - 8 = 24 bits). Other encoding schemes can save different numbers of bits. This technique assumes that the offsets of the base source register are the same as those of the other source and destination registers. It should be noted that in different implementations, this technique can be used for any combination of source registers, destination registers, and the sel value. In different implementations, the bits used can be derived from any of these different registers. For example, bits from one of the destination registers can be used to determine the address of the other destination register, the address of the source register, and the sel value. In other implementations, the base value of certain types of registers can be hard-coded, and a single offset can be provided for some or each of the destination or source registers and also used as the sel value.

[0151] Note that in the example above, the load width may be greater than the processor width. That is, the register may be larger than the number of bits that can be processed across all clock cycles. In these cases, it is possible to load the values ​​associated with two adjacent matrix multiplication instructions into a vector register. These matrix multiplication instructions can then be directed toward either the upper half or the lower half of the bits (whichever is more important).

[0152] Another optimization is the implicit use of specific registers. For example, in Figure 7 In the example, register R0 is designated as part of the instruction. Of course, in practice, the register used to store the sparse mask can be fixed, thus reducing the number of bits required to encode the instruction.

[0153] Figure 15 A flowchart 1600 illustrating an example of this technique is shown, particularly relevant to data processing apparatus 2. Here, at step 1610, a matrix multiplication instruction is received. This refers to a set of first elements (e.g., activation values) in a first vector register and a set of second elements (e.g., weights) in a second vector register. The first and second vector registers are such that they can be dynamically partitioned spatially and temporally (as needed). Then, at step 1620, a matrix multiplication operation is generated, causing the execution unit to perform the multiplication of the first element and the second element. The first and second elements have different bit widths from each other.

[0154] Figure 16 A flowchart 1700 illustrating the present technology, as exemplified by preprocessor 100, is shown. At step 1710, a model in a first format including multiple weights is received. At step 1720, the model is modified from the first format to a second format. At step 1730, the model in the second format including the weights is output. For a data processing apparatus 2 performing matrix multiplication using this model, the second format is computationally continuous, and the data processing apparatus has a first vector register 14A and a second vector register 14B, both configured to be partitioned in dynamic space and dynamic time.

[0155] Figure 17A and Figure 17B Examples of emulator implementations that can be used are illustrated. While the previously described embodiments implement the invention according to devices and methods for operating specific processing hardware supporting the technology of interest, it is also possible to provide an instruction execution environment implemented using a computer program according to the embodiments described herein. Such computer programs are generally referred to as emulators because they provide a software-based implementation of a hardware architecture. Types of emulator computer programs include simulators, virtual machines, models, and binary converters (including dynamic binary converters). Typically, an emulator implementation may run on a host processor 1830, which optionally runs a host operating system 1820 supporting emulator programs 1810A and 1810B. In some arrangements, there may be multiple layers of emulation between the hardware and the provided instruction execution environment and / or multiple different instruction execution environments provided on the same host processor. Historically, powerful processors have been required to provide emulator implementations that execute at a reasonable speed, but such approaches may be reasonable in certain situations, such as when it is necessary to execute native code of another processor for compatibility or reuse reasons. For example, an emulator implementation may provide an instruction execution environment with additional functionality not supported by the host processor hardware, or an instruction execution environment typically associated with a different hardware architecture. An overview of the simulation is given in "Some Efficient Architecture Simulation Techniques" (Robert Bedichek, Winter 1990 USENIX Conference, pp. 53-63).

[0156] Where embodiments have been previously described with reference to a particular hardware architecture or feature, equivalent functionality may be provided in a simulated embodiment by suitable software constructs or features. For example, a particular circuit may be implemented as computer program logic in a simulated embodiment. Similarly, memory hardware (such as registers or caches) may be implemented as software data structures in a simulated embodiment. Where one or more hardware components mentioned in the previously described embodiments are arranged on host hardware (e.g., host processor 1830), some simulated embodiments may utilize this host hardware where appropriate.

[0157] Emulator programs 1810A and 1810B can be stored on a computer-readable storage medium (which may be a non-transitory medium) and provide a program interface (instruction execution environment) for object code 1800 (which may include application programs, operating systems, and management programs), the same as the interface of the hardware architecture modeled by emulator 1810. Therefore, emulator program 1810 can be used to execute program instructions of object code 1800 from within the instruction execution environment, enabling host computer 1830, which does not actually possess the hardware characteristics of the devices discussed above, to emulate these characteristics.

[0158] The simulator program 1810A corresponds to the preprocessor device 100 discussed earlier and includes input logic 1805, the behavior of which corresponds to the behavior of input circuit 120. Output logic 1815 is provided, the behavior of which corresponds to output circuit 140, and adjustment logic 1825 is provided, the behavior of which corresponds to the behavior of adjustment circuit 140.

[0159] The simulator program 1810B corresponds to the previously discussed data processing device 2 and includes decoding logic 1835, the behavior of which corresponds to the behavior of the decoding circuit 6. A first data structure 1845 is provided, corresponding to the first library 14A of the vector register, and a second data structure 1855 is provided, corresponding to the second library 14B of the vector register.

[0160] The concepts described herein may be embodied in computer-readable code used to manufacture devices embodying the described concepts. For example, this computer-readable code may be used at one or more stages of the semiconductor design and manufacturing process, including electronic design automation (EDA) stages, to manufacture integrated circuits containing devices embodying these concepts. The aforementioned computer-readable code may additionally or alternatively implement the definition, modeling, simulation, verification, and / or testing of devices embodying the concepts described herein.

[0161] For example, computer-readable code for manufacturing a device embodying the concepts described herein may be embodied in code that defines the hardware description language (HDL) representation of these concepts. For instance, this code may define a register-transfer level abstraction of one or more logic circuits for defining a device embodying these concepts. This code may define the HDL representation of one or more logic circuits implementing the device using Verilog, SystemVerilog, Chisel, or VHDL (Very High Speed ​​Integrated Circuit Hardware Description Language), as well as intermediate representations such as FIRRTL. Computer-readable code may provide a definition embodying the concepts using system-level modeling languages ​​such as SystemC and SystemVerilog, or other behavioral representations of the concepts that can be interpreted by a computer to implement conceptual simulation, functional and / or formal verification and testing.

[0162] Additionally or alternatively, computer-readable code may define a low-level description of an integrated circuit component embodying the concepts described herein, such as one or more netlists or integrated circuit layout definitions, including representations such as GDSII. One or more netlists or other computer-readable representations of an integrated circuit component may be generated by applying one or more logic synthesis processes to an RTL representation to produce a definition for manufacturing a device embodying the present invention. Alternatively or additionally, one or more logic synthesis processes may generate a bitstream from computer-readable code loaded into a field-programmable gate array (FPGA) to configure the FPGA to embody the described concepts. The FPGA may be deployed for the purpose of verifying and testing these concepts prior to manufacturing in an integrated circuit, or the FPGA may be deployed directly in a product.

[0163] Computer-readable code may include a mixture of code representations for manufacturing apparatus, such as one or more of RTL representations, netlist representations, or other computer-readable definitions used in semiconductor design and manufacturing processes to manufacture apparatus embodying the present invention. Alternatively or additionally, the concept may be defined in a combination of computer-readable definitions to be used in semiconductor design and manufacturing processes to manufacture apparatus and computer-readable code defining instructions to be executed by the defined apparatus once manufactured.

[0164] Such computer-readable code can be contained in any known transient computer-readable medium (such as wired or wireless code transmission over a network) or non-transitory computer-readable medium (such as semiconductors, disks, or optical discs). Integrated circuits made using computer-readable code may include components such as a central processing unit, a graphics processing unit, a neural processing unit, a digital signal processor, or other components that individually or collectively implement the concept, or more.

[0165] In this application, the phrase "configured as..." is used to refer to the fact that the elements of the device have a configuration capable of performing the defined operations. In this context, "configuration" refers to the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operations, or a processor or other processing device may be programmed to perform the function. "Configured as" does not mean that the elements of the device need to be changed in any way to provide the defined operations.

[0166] While exemplary embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to those precise embodiments, and that various changes, additions, and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims. For example, features of the dependent claims can be combined with features of the independent claims in various ways without departing from the scope of the invention.< / sel> < / sel> < / sel>

Claims

1. A data processing apparatus, the data processing apparatus comprising: A first vector register and a second vector register, both configured to be partitioned in dynamic space and dynamic time; and A decoding circuit configured to receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including indications to a set of first elements in a first vector register and a set of second elements in a second vector register, and to generate a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein the matrix multiplication operation is configured to cause one or more execution units to perform a matrix multiplication of the set of first elements and the set of second elements; and The average bit width of the first element is different from the average bit width of the second element.

2. The data processing apparatus according to claim 1, wherein The first element comprises a single row of n activation values, and the second element comprises a pair of columns, each of which comprises n weights; and The matrix multiplication instruction includes an instruction to a result register configured to store a first set of bits of the result of the matrix multiplication and a second set of bits of the result of the matrix multiplication.

3. The data processing apparatus according to claim 2, wherein... The matrix multiplication instruction includes instructions for an additional result register; The result register is configured to store the first set of bits of the result of the matrix multiplication; and The additional result register is configured to store the second set of bits of the result of the matrix multiplication.

4. The data processing apparatus according to any of the preceding claims, wherein The number of rows in the first group of elements multiplied in the matrix multiplication is less than the number of columns in the second group of elements.

5. The data processing apparatus according to any of the preceding claims, wherein... The matrix multiplication has one row of the first group of elements and two columns of the second group of elements.

6. The data processing apparatus according to any of the preceding claims, wherein The matrix multiplication instructions include uncompressed matrix multiplication instructions; The first element comprises n activation values ​​for a single group; The second element comprises n weights for m groups, where m > 1; and The bit width of the second element is 1 / m of the bit width of the first element.

7. The data processing apparatus according to any of the preceding claims, wherein The bit width of the second element is four bits or less.

8. The data processing apparatus according to any of the preceding claims, wherein The bit width of the second element is one bit.

9. The data processing apparatus according to any of the preceding claims, wherein The second element is signed.

10. The data processing apparatus according to any of the preceding claims, wherein The weights are expanded before the matrix multiplication.

11. The data processing apparatus according to any one of claims 8 to 10, wherein The one or more matrix multiplication instructions include an indicator value, or the data processing device includes a selection register configured to store the indicator value; and The indicator value is configured to indicate a subset of the weights used in the matrix multiplication during a specific beat of the data processing device.

12. The data processing apparatus according to any one of claims 8 to 10, wherein Bits of at least one of the indications of the first set of first elements in the first vector register and the second set of second elements in the second vector register are used to indicate a subset of the weights used in the matrix multiplication during a specific beat of the data processing apparatus.

13. The data processing apparatus according to any of the preceding claims, wherein The matrix multiplication instruction includes a compressed matrix multiplication instruction, which includes an indication of compressed data; The first element consists of n activation values ​​in a single row; The second element comprises n compression weights for m groups, where m > 1; The compressed data indicates how to decompress the n compressed weights to form mn uncompressed weights.

14. The data processing apparatus according to claim 13, wherein The compressed data comprises multiple parts, each of which can be applied to one of a plurality of matrix multiplication instructions comprising the compressed matrix multiplication instruction; and The compressed matrix multiplication instruction includes a compressed data selector configured to select one portion of the compressed data to be applied to form the n uncompressed weights.

15. The data processing apparatus according to claim 14, wherein The compressed data can be applied to multiple matrix multiplication instructions; At least some of the matrix multiplication instructions indicate second elements that are distinct from each other; and The compressed data includes multiple items.

16. The data processing apparatus according to claim 15, wherein The compressed data can be applied to n weights of more than m groups.

17. The data processing apparatus according to any one of claims 15 to 16, wherein The items are sorted in the compressed data according to the timing of using the items within the plurality of matrix multiplication operations.

18. The data processing apparatus according to any one of claims 15 to 17, wherein The items are sorted in the compressed data such that items used in the same beat of the same single matrix multiplication operation are adjacent.

19. The data processing apparatus according to any one of claims 14 to 18, wherein The compressed data selector is the least significant bit of another parameter of the compressed matrix multiplication instruction.

20. The data processing apparatus according to any one of claims 14 to 19, wherein The compressed data selector is the least significant bit of the address of the second element.

21. The data processing apparatus according to claim 20, wherein The compressed data selector can be combined with the stub address of the first element to form the address of the first element; and The compressed data selector can be combined with the stub address of the result register to form the address of the result register, at least a portion of the result of the matrix multiplication being stored at the address of the result register.

22. The data processing apparatus according to any one of claims 14 to 21, wherein the data processing apparatus comprises: A multiplexer circuit is configured to select from the activation values ​​to match the non-zero uncompressed weights, thereby providing them as input to the matrix multiplication.

23. The data processing apparatus according to claim 22, wherein The multiplexer circuit is configured to select from a subset of the activation values ​​to match the non-zero uncompressed weights.

24. A data processing method, the data processing method comprising: Receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including indications to a set of first elements in a first vector register and a set of second elements in a second vector register, wherein each of the first vector register and the second vector register is configured to be partitioned in dynamic space and in dynamic time; In response to receiving the matrix multiplication instruction, a matrix multiplication operation is generated, wherein the matrix multiplication operation causes one or more execution units to perform a matrix multiplication of the first set of elements and the second set of elements; and The average bit width of the first element is different from the average bit width of the second element.

25. A computer program for controlling a host data processing device to provide an instruction execution environment, the computer program comprising: A first data structure and a second data structure, both configured to be partitioned in dynamic space and dynamic time; and Decoding logic, configured to receive one or more matrix multiplication instructions, the one or more matrix multiplication instructions including indications to a set of first elements in a first data structure and a set of second elements in a second data structure, and to perform a matrix multiplication operation in response to receiving the matrix multiplication instructions, wherein... The matrix multiplication operation is configured to cause the execution logic to perform a matrix multiplication of the first set of elements and the second set of elements; and The average bit width of the first element is different from the average bit width of the second element.

26. A data processing apparatus, the data processing apparatus comprising: An input circuit configured to receive a matrix in a first format comprising multiple values; An output circuit configured to output the matrix in a second format including the plurality of values; and An adjustment circuit, configured to perform a modification of the matrix from the first format to the second format, wherein... For a data processing apparatus that performs matrix multiplication using the matrix, the second format is computationally continuous, and the data processing apparatus has a first vector register and a second vector register, both of which are configured to be partitioned in dynamic space and dynamic time.

27. The data processing apparatus according to claim 26, wherein The values ​​include a first group and a second group, each of which will be multiplied by a single activation value in the matrix multiplication. In the first format, the first group forms a first consecutive grouping, and the second group forms a second consecutive grouping; and The adjustment circuit is configured to interleave the values ​​of each of the first group and the second group by performing a "z" scan on the first group and the second group to produce the second format.

28. The data processing apparatus according to claim 26, wherein The first and second groups are interleaved in a block of size p in the second format; and p is equal to the number of activation values ​​multiplied simultaneously in the matrix multiplication, where p>1。 29. The data processing apparatus according to claim 26, wherein The adjustment circuit is configured to interleave the first group and the second group into a block of size p by performing double-layer interleaving, in which the "z"-shaped scan is performed p times on the first group and the second group.

30. The data processing apparatus according to any one of claims 26 to 29, wherein In the second format, within q consecutive blocks of size p, the values ​​are grouped according to their positions in the first group and the second group.

31. The data processing apparatus according to any one of claims 26 to 30, wherein Each weight has an index j within group i; and In the second format, within each q block of size p, the values ​​are first sorted according to the index j of the values ​​and then according to the group i of the values.

32. The data processing apparatus according to any one of claims 26 to 31, wherein The adjustment circuit is configured to perform the modification in a single stage.

33. The data processing apparatus according to any one of claims 26 to 32, wherein The value includes the compression weight; The output circuit is configured to output compressed data; and The compressed data includes multiple items that indicate how the weights of the compression can be decompressed into decompressed weights.

34. The data processing apparatus according to claim 33, wherein The compressed data can be applied to multiple matrix multiplication instructions; and at least some of the matrix multiplication instructions indicate different compression weights for each other.

35. The data processing apparatus according to claim 34, wherein The compressed data can be applied to more than two sets of compression weights.

36. The data processing apparatus according to any one of claims 34 to 35, wherein The items are sorted in the compressed data according to the tick of the items used within the plurality of matrix multiplication instructions.

37. The data processing apparatus according to any one of claims 34 to 36, wherein The items are sorted in the compressed data such that items used in the same beat of the same single matrix instruction operation are adjacent.

38. A data processing method, the data processing method comprising: Receives a matrix in a first format that includes multiple weights; The output includes the matrix in a second format that incorporates the multiple weights; as well as The matrix is ​​modified from the first format to the second format, wherein the second format is computationally continuous for the data processing apparatus performing matrix multiplication, and the data processing apparatus has a first vector register and a second vector register, both of which are configured to be partitioned in dynamic space and dynamic time.

39. A computer program for controlling a host data processing device to provide an instruction execution environment, the computer program comprising: Input logic, configured to receive a matrix in a first format comprising multiple values; Output logic, configured to output the matrix in a second format including the plurality of values; and The adjustment logic is configured to perform a modification of the model matrix from the first format to the second format, wherein... For a data processing apparatus that performs matrix multiplication using the matrix, the second format is computationally continuous, and the data processing apparatus has a first vector register and a second vector register, both of which are configured to be partitioned in dynamic space and dynamic time.

Images