Computing system and method for controlling computing system

Abstract

To efficiently use a memory bandwidth.SOLUTION: A computing system 1 is suitable for mounting AI processing. A plurality of accelerators 110 performs matrix multiplication. LLC200 caches data of an external memory 300 that stores a computation result by each of the plurality of accelerators. A memory controller 120 determines whether or not an access to the LLC200 is congested and controls the LLC200 when it is determined that the access is congested. The control is to perform conversion for reducing the accuracy of a value of each component of a matrix on the matrix read from the external memory 300 in response to an access from an accelerator 110 that performs the matrix multiplication of the matrix and another matrix stored in the external memory 300, and to transfer the matrix.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a computing system and a computing method. [Background technology]

[0002] There is a demand for faster computational processing (hereinafter referred to as "AI processing") that realizes AI (artificial intelligence), and several technologies for this purpose have been proposed (see, for example, Patent Documents 1 to 5). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-523912 [Patent Document 2] Japanese Patent Application Publication No. 2023-77818 [Patent Document 3] US Patent Application Publication No. 2019 / 0354846 [Patent Document 4] US Patent Application Publication No. 2020 / 0258263 [Patent Document 5] US Patent Application Publication No. 2022 / 0044114 Summary of the Invention [Problem to be solved by the invention]

[0004] In many AI processes, matrix multiplication accounts for the majority of the processing. For example, in AI processes implemented using deep neural network models, the process of performing a matrix multiplication in the next stage is repeated using the matrix resulting from the previous stage of matrix multiplication.

[0005] Computational systems that achieve high-speed AI processing using computing devices (accelerators) suitable for matrix multiplication are widely known. While the speed of matrix multiplication operations themselves has increased due to the performance improvements of accelerators, such as improvements in integration, the performance improvements in memory bandwidth for data transfer between the memory that stores the calculation results have not kept pace. This is one of the factors hindering the speedup of AI processing.

[0006] In one aspect, the present invention aims to make efficient use of memory bandwidth. [Means for solving the problem]

[0007] In one embodiment, a computing system includes a plurality of accelerators, a cache memory, and a controller. The plurality of accelerators perform matrix multiplication operations. The cache memory caches data from an external memory that stores the results of operations performed by each of the plurality of accelerators. The controller determines whether access to the cache memory is congested, and controls the cache memory if it determines that access to the cache memory is congested. This control involves transferring a matrix read from the external memory in response to an access from an accelerator that performs a matrix multiplication operation between a matrix stored in the external memory and another matrix after performing a conversion that reduces the precision of the values ​​of each component of the matrix. [Effects of the Invention]

[0008] In one aspect, the present invention allows for efficient use of memory bandwidth. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the flow of calculations in AI processing. [Figure 2] FIG. 1 illustrates an example of a configuration of a computing system. [Figure 3] FIG. 2 is a diagram illustrating an example of a detailed configuration of a computing unit. [Figure 4] FIG. 2 is a diagram illustrating an example of the internal configuration of a PE. [Figure 5] FIG. 10 is a diagram illustrating an example of an exponent range table. [Figure 6] 10 is a flowchart showing the processing content of a first example of processing performed by an accelerator. [Figure 7] 10 is a flowchart showing the processing content of a second example of processing performed by an accelerator. [Figure 8] 10 is a flowchart illustrating an example of processing performed by a memory controller. [Figure 9] 10 is a flowchart illustrating an example of a precision determination process. [Figure 10] 10 is a flowchart illustrating an example of a read process. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0011] First, we will explain the calculation flow in AI processing.

[0012] Figure 1 is a schematic representation of an example of the flow of calculations in AI processing. This example shows a case where AI processing is realized using a deep neural network model. Note that Figure 1 shows an example where no activation function calculations are performed.

[0013] In AI processing using a deep neural network model, the matrix multiplication of a matrix whose components are features and a matrix whose components are weights is performed in parallel as processing for each layer. The matrix resulting from the calculation in the previous layer (layer L-1 in Figure 1) (or a matrix containing the result of the calculation and the result of the activation function) becomes the input for the matrix multiplication with a matrix whose components are weights in the next layer (layer L in Figure 1).

[0014] When implementing the above-described model in a computing system equipped with an accelerator chip having multiple accelerators suitable for matrix multiplication, the multiple matrix multiplication operations in each layer are distributed among the multiple accelerators. If the scale of the implemented model is large, an external memory connected to the accelerator chip is used as a storage area for temporarily storing the large amount of data handled by each layer.

[0015] While the performance of accelerators has improved due to increased integration, the speed of matrix multiplication operations has not kept pace with the improvements in memory bandwidth for data transfer between external memory used for temporary storage of data in each layer.

[0016] For example, consider a computing system equipped with an accelerator chip that has 16 accelerators, each of which has a computing unit configured as a systolic array with 64x64 PEs. Systolic arrays are widely known as a computing unit configuration that has the advantage of efficiently executing matrix multiplication operations. Note that "PE" is an abbreviation for Processing Element. Each PE has one multiplier and one adder, and performs a multiply-and-accumulate operation on each element of the two matrices that are the target of the matrix multiplication operation, calculating the value of one element in the matrix resulting from the matrix multiplication operation.

[0017] If the clock frequency of this computing system is 500MHz, the theoretical computing performance of this system is 2 (multiplications and additions) x 64 x 64 x 16 x 500MHz, which is approximately 65 TFLOPS. On the other hand, HBM3-type DRAM, which has recently begun to be used as high-speed external memory, has a memory bandwidth of 819GB / s. "HBM" stands for High Bandwidth Memory, and "DRAM" stands for Dynamic Random Access Memory.

[0018] As such, there is a large performance gap between the memory bandwidth of external memory and the computational performance of matrix multiplication. Furthermore, increasing memory bandwidth is technically difficult, and dramatic improvements are currently not expected. To address this performance gap, cache memory is sometimes used to cache data stored in external memory.

[0019] The cache memory located farthest from the processing unit, i.e., the cache memory located closest to the external memory, is specifically called LLC. "LLC" is an abbreviation for Last Level Cache. By inserting an LLC between the accelerator chip and the external memory, it is expected that the performance difference between the accelerator and the external memory can be bridged. However, if multiple accelerators on an accelerator chip share the LLC, a situation will arise in which the accelerators compete for the memory bandwidth of the LLC.

[0020] In AI processing using the above-mentioned model, although it is necessary to execute a large number of matrix multiplication operations, high accuracy is rarely required for these operations. The embodiment described below focuses on such characteristics of the requirements for matrix multiplication operations in AI processing, and enables matrix multiplication operations that are executed repeatedly in a hierarchical manner to be performed in a short time.

[0021] In this embodiment, when the result of a matrix multiplication operation in a previous layer is obtained, the matrix resulting from the operation is stored in an external memory, and the range of values ​​taken by the exponent part of the floating-point values ​​of each element of the matrix is ​​calculated and stored in a table in the memory controller. When processing the next layer, the memory controller controls the LLC according to the degree of access congestion to the LLC to cause the LLC to perform a conversion that reduces the precision of the values ​​of each element of the matrix resulting from the operation in the previous layer. The memory controller determines the precision of the converted values ​​based on the range of values ​​shown in the table. This determination is made by selecting the representation format (data format) of the floating-point values ​​of each element of the matrix to be multiplied in the next layer. The memory controller transfers the converted matrix received from the LLC to the accelerator responsible for the matrix multiplication operation in the processing of the next layer.

[0022] In this embodiment, by doing as described above, when a representation format with low representability is selected, the amount of data representing the matrix operation result is reduced. As a result, the amount of data transferred from the LLC to the accelerator is reduced, which makes it possible to effectively use the memory bandwidth of the LLC, and ultimately the memory bandwidth of the external memory.

[0023] Next, the configuration of the computing system according to this embodiment will be described. Figure 2 shows an example of the configuration of the computing system 1. This computing system 1 performs high-speed AI processing as described with reference to Figure 1.

[0024] The computing system 1 includes an accelerator chip 100 and an LLC 200, and is connected to an external memory 300.

[0025] The accelerator chip 100 is connected to an external memory 300 via an LLC 200. The accelerator chip 100 includes a plurality of accelerators 110 and a memory controller 120.

[0026] Each of the plurality of accelerators 110 includes a computing unit 111 , a register 112 , and a control unit 113 .

[0027] The computing unit 111 performs a matrix multiplication operation. In this embodiment, the above-mentioned systolic array is used as the computing unit 111.

[0028] The register 112 stores and temporarily holds data (values ​​of each element of the matrix) input to and output from the arithmetic unit 111.

[0029] The control unit 113 controls the arithmetic unit 111 and the register 112 .

[0030] The external memory 300 stores and saves the results of calculations performed by each of the multiple accelerators 110 included in the accelerator chip 100, for example, the matrix of the calculation result in the previous layer in the matrix multiplication calculation described with reference to Fig. 1. The LLC 200 is a cache memory that caches data from the external memory 300.

[0031] The memory controller 120 controls the operation of the LLC 200 and the external memory 300 .

[0032] In this embodiment, one of the controls performed by the memory controller 120 is to transfer a matrix read from the external memory 300 after performing a transformation that reduces the precision of the values ​​of each element of the matrix. This matrix is ​​read from the external memory 300 in response to an access from the accelerator 110 that performs a matrix multiplication operation between the matrix and another matrix. The memory controller 120 determines whether access to the LLC 200 is congested, and performs this control if it determines that access is congested.

[0033] In this embodiment, in order to make the above-described determination, the memory controller 120 is provided with a FIFO memory (not shown) that temporarily stores accesses to the external memory 300 from each of the multiple accelerators 110 as a queue. "FIFO" is an abbreviation for First In First Out. The memory controller 120 compares the number of accesses stored in the FIFO memory with a predetermined threshold. If the result of this comparison indicates that the number of accesses exceeds the threshold, the memory controller 120 determines that accesses to the LLC 200 are congested.

[0034] The method by which the memory controller 120 determines whether access to the LLC 200 is congested is not limited to the above-described method, and various other methods can be adopted. For example, when the memory controller 120 receives an access to the external memory 300, it may determine whether the memory controller 120 itself is performing control processing on the LLC 200 in response to another access to the external memory 300. In this case, if the memory controller 120 is performing that control processing, it determines that access to the LLC 200 is congested.

[0035] In the configuration example shown in Fig. 2, a CPU chip 400 having a CPU 410 is connected to the LLC 200. The CPU 410 controls the computing system 1. Note that "CPU" is an abbreviation for Central Processing Unit. For example, the CPU 410 provides each of the accelerators 110 with a matrix whose components are weights among the matrices to be computed in the matrix multiplication operation described with reference to Fig. 1.

[0036] Next, an example of a detailed configuration of the arithmetic unit 111 included in the accelerator 110 in FIG. 2 will be described with reference to FIG.

[0037] 3, the accelerator 110 is configured by arranging a plurality of PEs 130 in rows and columns, and in one example, is configured by 64×64 PEs 130 arranged in rows and columns. Of the plurality of PEs 130, the elements i1, i2, i3, ... of each row of matrix A are sequentially input to the inputs of the PEs 130 arranged in a single row, and the elements j1, j2, j3, ... of each column of matrix B are sequentially input to the inputs of the PEs 130 arranged in a single column.

[0038] 4 shows an example of the internal configuration of the PE 130. The PE 130 includes a multiplier 131, an adder 132, and an accumulator 133.

[0039] Multiplier 131 multiplies component a input to PE 130 in row i of matrix A by component b input to PE 130 in column j of matrix B, and outputs the multiplication result to adder 132.

[0040] Adder 132 adds the multiplication result by multiplier 131 and the value stored in accumulator 133, and outputs the addition result to accumulator 133. Accumulator 133 stores the addition result by adder 132 and outputs the stored value c. In other words, adder 132 and accumulator 133 sequentially add the multiplication results sequentially output from multiplier 131.

[0041] The PE 130 has the above-mentioned configuration and executes the calculation expressed by the following formula 1. That is, the PE 130 executes a multiply-and-accumulate operation between the k-th component A[i][k] in row i of matrix A and the k-th component B[k][j] in column j of matrix B, where k is a variable. Then, the PE 130 outputs, as the result of this operation, the value C[i][j] of the component corresponding to row i and column j in matrix C, which is the matrix product of matrix A and matrix B.

[0042]

number

[0043] As described above, in this embodiment, a systolic array is adopted as the configuration of the computing unit 111 included in the accelerator 110. However, the computing unit 111 may be configured in another way that has excellent performance in executing matrix multiplication operations.

[0044] In this embodiment, the accelerator 110 supports operations in various representation formats of floating-point numbers. Specifically, the PE 130 included in each accelerator 110 of the accelerator chip 100 is capable of executing operations using floating-point numbers represented in the widely known representation formats FP32, FP16, BF16, and FP8, as well as the recently used FP19 representation format. Furthermore, the values ​​of each element of the matrix resulting from the matrix multiplication operation output from the accelerator 110 are represented in the FP32 format.

[0045] The FP32 format has a bit width of 1 bit for the sign, 8 bits for the exponent, and 23 bits for the mantissa. The FP16 format has a bit width of 1 bit for the sign, 5 bits for the exponent, and 10 bits for the mantissa. The BF16 format has a bit width of 1 bit for the sign, 8 bits for the exponent, and 7 bits for the mantissa. The FP8 format is available in two variations: E5M2 and E4M3. The E5M2 has a bit width of 1 bit for the sign, 5 bits for the exponent, and 2 bits for the mantissa. The E4M3 has a bit width of 1 bit for the sign, 4 bits for the exponent, and 3 bits for the mantissa. The FP19 format has a bit width of 1 bit for the sign, 8 bits for the exponent, and 10 bits for the mantissa. Therefore, of these floating-point number representation formats supported by the accelerator 110, the FP32 format is the representation format capable of representing values ​​with the highest precision.

[0046] In this embodiment, the accelerator 110 externally acquires information on the representation format of floating-point numbers representing the values ​​of each element of an input matrix, i.e., a matrix to be subjected to a matrix multiplication operation. The multiplier 131 of each of the PEs 130 included in the accelerator 110 is configured to multiply each element of the matrix represented in the representation format indicated by the acquired information and sequentially output the multiplication result values ​​to the adder 132 in FP32 format. Meanwhile, the adder 132 and the accumulator 133 are configured to sequentially add the multiplication result values ​​sequentially output from the multiplier 131 with precision equivalent to the FP32 format and output the result value of the sequential addition in FP32 format. In this way, the sequential addition by the adder 132 and the accumulator 133 is configured to always be performed with high precision equivalent to the precision of the FP32 format, regardless of the information acquired from the outside, thereby suppressing degradation of the calculation precision caused by the product-sum calculation by the PE 130.

[0047] The memory controller 120 obtains information on the representation formats of floating-point numbers that the accelerator 110 supports for execution of calculations directly from the CPU 410 or via the accelerator 110. The memory controller 120 selects the representation formats indicated in the obtained information as options and selects, from the options, the precision of the conversion to be performed by the LLC 200, i.e., the precision of the values ​​of each matrix element after the conversion in the conversion that reduces the precision of the values. Next, the method of this selection will be described.

[0048] In the following description, among the multiple accelerators 110, one that writes the pre-conversion matrix described above to the external memory 300, i.e., one that outputs the pre-conversion matrix as a calculation result, will be referred to as the "previous-stage accelerator 110." Also, among the multiple accelerators 110, one that performs a matrix multiplication operation using a matrix whose component values ​​have been reduced in precision by the above-described conversion by the LLC 200 will be referred to as the "next-stage accelerator 110."

[0049] When writing the matrix before conversion, i.e., the matrix resulting from the matrix multiplication operation, to the external memory 300, the front-stage accelerator 110 calculates the range of values ​​(maximum and minimum values) of the exponent part of the value of each element of the matrix expressed in the floating-point number representation format. The front-stage accelerator 110 notifies the memory controller 120 of the calculated range of values.

[0050] When the memory controller 120 receives notification of the value range from the accelerator 110, the memory controller 120 adds the value range to an exponent range table stored in a storage unit (not shown) that the memory controller 120 itself has.

[0051] An example of the exponent range table is shown in Fig. 5. The exponent range table indicates the range of values ​​(maximum and minimum) that the exponent takes for each component of the resulting matrix, in association with the address (e.g., the starting address) of the storage area in the external memory 300 into which the preceding accelerator 110 writes the resulting matrix of the matrix multiplication operation.

[0052] Thereafter, the next-stage accelerator 110 queries the memory controller 120 to obtain information on the precision of the values ​​of each matrix element that is the target of the matrix multiplication operation that it performs. Upon receiving this query, the memory controller 120 reads, from the exponent range table, the range of values ​​that the exponent of each matrix element resulting from the matrix multiplication operation performed by the previous-stage accelerator 110 can take. From among the representation formats included in the aforementioned options, the memory controller 120 selects an representation format that has an exponent width that can express the values ​​in the read range and that uses the fewest number of bits to express the values. Having the memory controller 120 select such a representation format reduces the occurrence of overflow or underflow in the converted value.

[0053] In this way, the precision after conversion of the value of each element of the matrix resulting from the operation by the preceding accelerator 110 is determined. After that, the memory controller 120 controls the LLC 200 to convert the representation format of the value of each element of the matrix read from the external memory 300 by the access into the selected representation format and then send it to the next-stage accelerator 110.

[0054] For example, it is assumed that the memory controller 120 has acquired information that the accelerator 110 supports the following floating-point number representation formats: FP32, FP16, BF16, FP8 (E5M2 and E4M3), and FP19. In this case, it is assumed that the range of values ​​taken by the exponent part of the value of each matrix element resulting from the matrix multiplication operation, which the preceding accelerator 110 has notified the memory controller 120 about, is the range shown in the first row of the exponent part range table in FIG.

[0055] The first row of the exponent range table in Figure 5 shows that the maximum value in the range of values ​​that the exponent of each matrix element can take is "+7" and the minimum value is "-8." Therefore, this range of values ​​cannot be expressed with 4 bits, which limits the representable range from -7 to +8, and at least 5 bits, which limits the range from -14 to +15, are required as the bit width of the exponent.

[0056] In the above example, among the representation formats of floating-point numbers supported by the accelerator 110, those with a 5-bit exponent bit width include the FP16 format and the E5M2 FP8 format. Here, the E5M2 FP8 format is a representation format that represents floating-point numbers in one octet (8 bits), and the FP16 format is a representation format that represents floating-point numbers in two octets (16 bits). Therefore, the number of bits used to represent values ​​is fewer in both formats than in the FP32 format, which represents floating-point numbers in four octets (32 bits), and of these, the E5M2 FP8 format is fewer than the FP16 format. Therefore, in this example, the memory controller 120 selects the E5M2 FP8 format and instructs the LLC 200 to convert the values ​​of each matrix element into the E5M2 FP8 format, which reduces the precision of the values ​​of each matrix element.

[0057] In converting the representation format of each matrix element, it is possible to not allow overflow in the converted value, but allow underflow. To do this, the memory controller 120 should select a representation format that has an exponent with a bit width that can represent the maximum value of the read range of values, and that uses the fewest number of bits to represent the value.

[0058] The memory controller 120 controls the LLC 200 in the above manner to convert a matrix, each element of which is expressed in FP32 format, read from the external memory 300, to reduce its precision, and then transfer the converted matrix to the next-stage accelerator 110. The next-stage accelerator 110 receives the converted matrix and performs a matrix multiplication operation between the converted matrix and another matrix, as described above. This matrix multiplication operation involves multiplication of each matrix element with the converted precision, and sequential addition of the multiplication results with a precision equivalent to the FP32 format, i.e., sequential addition with the precision before the conversion. The next-stage accelerator 110 writes the matrix resulting from this matrix multiplication operation to the external memory 300. At this time, values ​​expressed in FP32 format are written to the external memory 300 as the values ​​of each element of the matrix resulting from the operation.

[0059] The flow of the above-mentioned processes performed by the accelerator 110 and the memory controller 120 will be described below with reference to the drawings.

[0060] First, an example of processing performed by the accelerator 110 as the first layer processing of the processing for each layer in AI processing will be described with reference to the flowchart of FIG.

[0061] When the processing in FIG. 6 starts, first, in S101, the processing for executing the matrix multiplication operation as the processing of the first layer is performed as described above.

[0062] In the process of S101, a matrix multiplication operation is performed between a matrix (a matrix of features in the example of FIG. 1) whose elements are input values ​​from an input unit (not shown) of the calculation system 1 and a matrix (a matrix of weights in the example of FIG. 1) that the accelerator 110 has acquired in advance before the start of the process of FIG. 6. In this matrix multiplication operation, multiplication is performed on matrices whose elements are expressed in an expression format that can express values ​​with the highest precision among the expression formats of floating-point numbers that the accelerator 110 supports for execution of operations. Therefore, in this embodiment, the elements of the matrix that is the target of the matrix multiplication operation at this time are expressed in the FP32 format.

[0063] In S102, a write request including the matrix resulting from the matrix multiplication operation performed in the process of S101 is sent to the memory controller 120. This write request requests that the matrix resulting from the matrix multiplication operation be written to the external memory 300.

[0064] In S103, a process is performed to obtain the range of values ​​(maximum and minimum values) that the exponent part of each component of the matrix obtained from the matrix multiplication operation in S101.

[0065] In S104, a notification informing the memory controller 120 of the range of values ​​acquired in the process of S103 is sent to the memory controller 120, and then the process of FIG. 6 ends.

[0066] Next, an example of processing performed by the accelerator 110 as processing of each layer from the second layer onward among the processing for each layer in AI processing will be described with reference to the flowchart of FIG.

[0067] 7 starts, first, in S111, a query is sent to the memory controller 120 to obtain information on the precision of the values ​​of each element of the matrix that is the target of the matrix multiplication operation executed by the process described below. Note that the target matrix is ​​a matrix stored in the external memory 300, i.e., a matrix that is the result of the matrix multiplication operation performed by the accelerator 110 at the previous stage.

[0068] In S112, a process is performed to acquire a response regarding information on the precision of the values ​​of each element of the matrix, which is sent from the memory controller 120 in response to the inquiry sent in the process of S111.

[0069] In S113, a process is performed in which a read request for the matrix to be subjected to the matrix multiplication operation, which will be executed by the process described later, from the external memory 300 is sent to the memory controller.

[0070] In S114, a process is performed to acquire the matrix sent from the memory controller 120 in response to the read request sent in the process of S113. Note that the values ​​of each element of the matrix acquired in this process are expressed in the representation format indicated by the response acquired in the process of S112.

[0071] In S115, a process is performed to perform a matrix multiplication operation between the matrix acquired in the process of S114 and a matrix (for example, the weight matrix in FIG. 1) acquired by the accelerator 110 before the start of the process of FIG.

[0072] In S116, a write request including the matrix resulting from the matrix multiplication operation performed in S115 is sent to the memory controller 120. This write request requests that the matrix resulting from the matrix multiplication operation be written to the external memory 300.

[0073] In S117, a process is performed to obtain the range of values ​​(maximum and minimum values) that the exponent part of each component of the matrix is ​​to take from the matrix resulting from the matrix multiplication operation in the process of S115.

[0074] In S118, a process is performed in which a notification is sent to the memory controller 120 to inform the memory controller 120 of the range of values ​​acquired in the process of S117, and then the process of FIG. 7 ends.

[0075] Next, an example of processing performed by the memory controller 120 will be described with reference to the flowchart of FIG.

[0076] 8 starts, first, in S201, a process is performed to determine whether or not a query for information on the precision of the values ​​of each element of the matrix to be operated on has been received. This query is sent from the accelerator 110 in the process of S111 in FIG. 7 described above.

[0077] If it is determined in the determination process of S201 that an inquiry has been received (if the determination result is YES), the process proceeds to S202 where an accuracy determination process is performed, and then the process proceeds to S203. The accuracy determination process will be described in detail later.

[0078] On the other hand, if it is determined in the determination process of S201 that no inquiry has been received (if the determination result is NO), the process of S202 is skipped and the process proceeds to S203.

[0079] In S203, it is determined whether a read request for the matrix has been received. This read request is sent from the accelerator 110 in the process of S113 in FIG.

[0080] If it is determined in the determination process of S203 that a read request has been received (if the determination result is YES), the process proceeds to S204 where the read process is performed, and then the process proceeds to S205. The details of this read process will also be described later.

[0081] On the other hand, if it is determined in the determination process of S203 that a read request has not been received (if the determination result is NO), the process of S204 is skipped and the process proceeds to S205.

[0082] In S205, it is determined whether a matrix write request has been received. This write request is sent from the accelerator 110 in the process of S102 in Fig. 6 or the process of S116 in Fig. 7.

[0083] If it is determined in the determination process of S205 that a write request has been received (if the determination result is YES), the process proceeds to S206. Then, in S206, the matrix included in the write request is transferred to the LLC 200, and the LLC 200 is instructed to write the matrix to the external memory 300. Thereafter, the process proceeds to S207. In response to this write instruction, the LLC 200 writes the matrix to the external memory 300.

[0084] On the other hand, if it is determined in the determination process of S205 that a write request has not been received (if the determination result is NO), the process of S206 is skipped and the process proceeds to S207.

[0085] In S207, it is determined whether or not a notification of the range of values ​​that the exponent part of each element of the matrix that is the target of the write request has been received. This notification is sent from the accelerator 110 by the processing of S104 in Fig. 6 or the processing of S118 in Fig. 7.

[0086] If it is determined in the determination process of S207 that a notification has been received (if the determination result is YES), the process proceeds to S208, where the notified value range is added to the exponent part range table stored in the storage unit of the memory controller 120. After that, the process returns to S201, and the above-mentioned process is repeated.

[0087] On the other hand, if it is determined in the determination process of S207 that a notification has not been received (the determination result is NO), the process of S208 is skipped, and thereafter the process returns to S201 and the above-mentioned processes are repeated.

[0088] The above processing is performed by the memory controller 120. Note that when the processing of S202, S204, and S206 is completed, the processing may immediately return to S201 and the subsequent processing may be repeated.

[0089] Next, a description will be given of the details of the accuracy determination process, which is the process of S202 in Fig. 8. Fig. 9 is a flowchart showing the processing contents of an example of the accuracy determination process.

[0090] When the precision determination process is started, first, in S211, the accelerator 110 acquires the representation format of floating-point numbers that it supports for execution of calculations.

[0091] Next, in S212, the process is performed to determine whether or not access to the LLC 200 is congested, as described above. If it is determined that access to the LLC 200 is congested (the determination result is YES), the process proceeds to S214.

[0092] On the other hand, if it is determined in the determination process of S212 that access to the LLC 200 is not congested (the determination result is NO), the process proceeds to S213. Then, in S213, a process is performed to select the representation format with the highest precision among the representation formats supported by the accelerator 110 for executing operations obtained in the process of S211, and then the process proceeds to S218. Therefore, in this embodiment in which the accelerator 110 supports the execution of operations using floating-point numbers expressed in the representation formats FP32, FP16, BF16, FP8, and FP19, the FP32 format is selected in the process of S213.

[0093] In S214, the range of values ​​taken by the exponent part of the value of each component of the matrix to be calculated, i.e., the matrix to be read in the read process (processing of S204 in Figure 8) performed following this precision determination process, is obtained from the exponent part range table.

[0094] In S215, a process is performed to determine whether or not underflow is permitted in the matrix multiplication operation by the accelerator 110. In this embodiment, whether or not underflow is permitted is acquired directly from the CPU 410 or is notified via the accelerator 110.

[0095] In the judgment process of S215, if it is determined that underflow is allowed (if the judgment result is YES), the process proceeds to S216, and if it is determined that underflow is not allowed (if the judgment result is NO), the process proceeds to S217.

[0096] In S216, a process is performed to select, from the representation formats obtained in the process of S211, a representation format that has, as an exponent part, a bit width that can represent the maximum value in the range obtained in the process of S214 and that uses the fewest number of bits to represent the value.Then, the process proceeds to S218.

[0097] In S217, a process is performed to select, from the representation formats obtained in the process of S211, a representation format that has an exponent part with a bit width that can represent both the maximum and minimum values ​​of the range obtained in the process of S214 and that uses the fewest number of bits to represent the value.Then, the process proceeds to S218.

[0098] The selection of the expression format in the processes of S217 and S218 is performed by the method described above with reference to FIG.

[0099] In S218, the representation format selected in the process of S213, S216, or S217 is sent to the accelerator 110 that sent the query as a response to the query about the precision of the values ​​of each element of the matrix to be operated on. After that, this precision determination process ends, and the process returns to S203 in FIG. 8.

[0100] The above processing is the accuracy determination processing.

[0101] Next, a detailed description will be given of the read processing, which is the processing of S204 in Fig. 8. Fig. 10 is a flowchart showing the processing contents of an example of the read processing.

[0102] When the processing of FIG. 8 starts, first, in S221, processing is performed to notify the LLC 200 of the representation format selected in the above-described precision determination processing.

[0103] Next, in S222, a process is performed to instruct LLC 200 to read from external memory 300 the matrix that is the target of the read request that was determined to have been received in the determination process of S203 in Figure 8, and to convert the precision of the values ​​of each component of the read matrix.

[0104] Upon receiving the instruction from the processing of S222, the LLC 200 reads the matrix that is the target of the read request from the external memory 300. Then, the LLC 200 converts the representation format of each element of the read matrix into the representation format notified by the processing of S221, thereby performing a conversion that reduces the precision of the values ​​of each element of the read matrix, and sends the converted matrix to the memory controller 120.

[0105] In addition, if the FP32 format is selected in the accuracy determination process, that is, if the representation format with the highest accuracy is selected by the processing of S213 in Figure 9, LLC200 will treat the matrix read from external memory 300 as the converted matrix as is.

[0106] In S223, the matrix after the transformation in S222 is received from the LLC 200. Then, in the following S224, the received matrix is ​​transferred to the accelerator 110 that issued the read request. After that, the read process ends, and the process returns to S205 in FIG. 8.

[0107] The above processing is the read processing.

[0108] By having the accelerator 110 and the memory controller 120 perform the processes described above, respectively, the calculation system 1 can execute hierarchically repeated matrix multiplication calculations in a short processing time.

[0109] Although the disclosed embodiments and their advantages have been described in detail above, it will be appreciated that those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims.

[0110] 9, the expression format is selected from among the expression formats of floating-point numbers supported by the accelerator 110, in which the exponent part has a bit width that can express the range of values ​​that the exponent part of the value of each element of a matrix takes and in which the number of bits used to express the value is the smallest. Alternatively, the memory controller 120 may select and return an expression format designated by the user of the arithmetic system 1, as instructed by the CPU 410, from among the expression formats of floating-point numbers supported by the accelerator 110. [Explanation of symbols]

[0111] 1. Computing System 100 accelerator chips 110 Accelerator 111 Arithmetic unit 112 registers 113 Control Unit 120 Memory Controller 130 PE 131 Multiplier 132 Adder 133 Accumulator 200 LLC 300 external memory 400 CPU chips 410 CPU

Claims

1. a plurality of accelerators for performing matrix multiplication operations; a cache memory that caches data from an external memory that stores the results of operations performed by each of the plurality of accelerators; a controller that determines whether access to the cache memory is congested, and when it is determined that access to the cache memory is congested, controls the cache memory to transfer the matrix read from the external memory in response to an access from one accelerator of the plurality of accelerators that performs the matrix multiplication operation between a matrix stored in the external memory and another matrix, after performing a conversion that reduces the accuracy of values ​​of each component of the matrix; A computing system comprising:

2. an accelerator among the plurality of accelerators that outputs the matrix as the calculation result calculates a range of values ​​that an exponent part of each element of the matrix expressed in a floating-point number representation format can take, and notifies the controller of the range; The controller determines the precision after the conversion based on the range.

2. The computing system according to claim 1.

3. 3. The arithmetic system according to claim 2, wherein the conversion is a conversion in which the value of each element of the matrix is ​​expressed in the representation format having a bit width capable of expressing the maximum value of the range as an exponent, and in which the number of bits used to express the value is reduced.

4. 3. The arithmetic system according to claim 2, wherein the conversion is a conversion in which the value of each element of the matrix is ​​expressed in the expression format having a bit width capable of expressing both the maximum and minimum values ​​of the range as an exponent, and in which the number of bits used to express the value is reduced.

5. 4. The computing system according to claim 2, wherein the controller selects the representation format for the values ​​of each element of the matrix after the conversion from among a selection of representation formats.

6. 2. The arithmetic system according to claim 1, wherein the one accelerator performs multiplication and sequential addition of results of the multiplication to perform a multiply-and-add operation between elements of the matrix and elements of the other matrix as the matrix multiplication operation, with the precision before the conversion.

7. 1. A control method for a computing system including a plurality of accelerators that perform matrix multiplication operations, and a cache memory that caches data in an external memory that stores results of operations performed by each of the plurality of accelerators, the method comprising: determining whether access to the cache memory is congested; controlling the cache memory when it is determined that the cache memory is congested, and transferring the matrix read from the external memory in response to an access from one accelerator of the plurality of accelerators, the one accelerator performing the matrix multiplication operation between the matrix stored in the external memory and another matrix, after performing a conversion that reduces the accuracy of values ​​of each component of the matrix; 1. A control method for a computing system comprising:

Images