Cache structure and electronic devices

Abstract

A cache structure and electronic device, wherein the cache structure includes an N-level cache, the i-th level cache in the N-level cache includes at least one level i cache node, each level i cache node includes a level i read-only cache area with read-only privileges and a level i read-write cache area with read-write privileges, each of the multiple level i cache nodes shares one level i+1 cache node, the n-th level cache is electrically connected to external memory, each level 1 cache node is electrically connected to a corresponding computing unit, the read-only cache area in the N-level cache is used to cache input matrices as computation input parameters in matrix calculations and to transmit input matrices between external memory and multiple computing units, and the read-write cache area in the N-level cache is used to cache output matrices as computation output parameters in matrix calculations and to transmit output matrices between external memory and multiple computing units.

Description

Technical Field

[0001] This application claims the priority of Chinese Patent Application No. 202310740144.X filed on June 20, 2023, and the entire content of the Chinese patent application is incorporated herein by reference as part of this application.

[0002] Embodiments of the present disclosure relate to a cache structure and an electronic device.

Background Art

[0003] The operating speed of a processor is much higher than that of memory. When the processor directly accesses the read / write data of memory, it needs to wait for a certain period of time. Therefore, in the process of accessing memory, a cache memory (Cache) is usually used to improve system efficiency and the access speed of the processor to memory. Usually, the processor preferentially searches for data from the Cache. For example, if the data requested by an application or software exists in the Cache, it is called a cache hit, and conversely, it is called a cache miss.

Summary of the Invention

Means for Solving the Problems

[0004] At least one embodiment of the present disclosure provides a cache structure for matrix calculations, the cache structure comprising an N-level cache, the i-th level cache in the N-level cache comprising at least one level i cache node, each level i cache node comprising a level i read-only cache area with read-only privileges and a level i read-write cache area with read-write privileges, each of multiple level i cache nodes sharing one level i+1 cache node, all level N-1 cache nodes sharing one level N cache node included in the N-th level cache in the N-level cache, the level N cache node comprising a level N read-only cache area with read-only privileges and a level N read-write cache area with read-write privileges The N-level cache includes a write cache area, the N-level cache is electrically connected to an external memory, each level 1 cache node is electrically connected to a corresponding computing unit, the read-only cache area in the N-level cache with read-only privileges is used to cache input matrices as computation input parameters in the matrix calculation and to transmit the input matrices between the external memory and the plurality of computing units, the read-write cache area in the N-level cache with read-write privileges is used to cache output matrices as computation output parameters in the matrix calculation and to transmit the output matrices between the external memory and the plurality of computing units, where N is a positive integer greater than 1 and i is any positive integer between 1 and N-1.

[0005] For example, in a cache structure according to at least one embodiment of the present disclosure, each computing unit includes an arithmetic module and a register array, the register array of each computing unit is used to store some parameters in the input matrix and some parameters in the output matrix, and the arithmetic module of each computing unit includes a plurality of multipliers and / or sum-accumulates for performing multiplication and / or sum-accumulate calculations in parallel.

[0006] For example, in a cache structure according to at least one embodiment of the present disclosure, the matrix calculation includes matrix multiplication and / or matrix sum-of-products calculations.

[0007] For example, in a cache structure according to at least one embodiment of the present disclosure, an input matrix as a computation input parameter is read into a register array of the corresponding computation unit via a read-only cache path consisting of a read-only cache area having read-only privileges in the N-level cache; an output matrix as a computation output parameter is read into a register array of the corresponding computation unit via a read-write cache path consisting of a read-write cache area having read-write privileges in the N-level cache; an updated output matrix is ​​obtained after computation by the arithmetic module of the corresponding computation unit; and the updated output matrix is ​​written back from the corresponding computation unit to the external memory or a read-write cache area of ​​any level of cache via the read-write cache path.

[0008] For example, in a cache structure according to at least one embodiment of the present disclosure, in response to N being 3, the read-only cache path includes a read-only cache area in a level 1 cache node corresponding to the corresponding compute unit, a read-only cache area in a level 2 cache node electrically connected to the level 1 cache node, and a read-only cache area in a level 3 cache node, and the read-write cache path includes a read-write cache area in a level 1 cache node corresponding to the corresponding compute unit, a read-write cache area in a level 2 cache node electrically connected to the level 1 cache node, and a read-write cache area in a level 3 cache node.

[0009] For example, in a cache structure according to at least one embodiment of the present disclosure, the external memory is dynamic random memory.

[0010] At least one embodiment of the present disclosure further provides an electronic device comprising a cache structure described in any embodiment of the present disclosure.

[0011] For example, in an electronic device according to at least one embodiment of the present disclosure, the electronic device includes N levels of task distribution units and N hierarchical thread groups, each level of task distribution units includes at least one task distribution unit, each hierarchical thread group includes at least one thread group, the quantity of the j-th level task distribution unit located at the j-th level in the N-level task distribution unit is the same as and corresponds one-to-one with the quantity of the level j cache nodes included in the corresponding j-th level cache in the N-level cache, the quantity of the level j thread group located at the j-th level in the N hierarchical thread groups is the same as and corresponds one-to-one with the quantity of the level j cache nodes included in the corresponding j-th level cache in the N-level cache, and the threads included in each level j thread group cache data by sharing the corresponding level j cache node, where j is a positive integer less than or equal to N.

[0012] For example, in an electronic device according to at least one embodiment of the present disclosure, the quantity of the Nth-level task distribution unit located at the Nth level in the N-level task distribution unit is 1, and the total tasks for matrix calculation are divided into P1 first tasks, each assigned to a group of P1 level N threads and arranged to be executed in a time-sharing manner, where P1 is a positive integer.

[0013] For example, in an electronic device according to at least one embodiment of the present disclosure, the electronic device is arranged to perform a first task corresponding to a group of level N threads within a single operation cycle.

[0014] For example, in an electronic device according to at least one embodiment of the present disclosure, the electronic device reads the parameters of an input matrix required to perform a plurality of first tasks from the P1 first tasks into the level N cache node in a batch, and arranges a plurality of compute units to perform the plurality of first tasks within a plurality of operation cycles.

[0015] For example, in an electronic device according to at least one embodiment of the present disclosure, the N-level task distribution unit is further configured to select at least one first task, divide each first task into P2 second tasks, thereby obtaining P2 level N-1 thread groups divided from the level N thread group corresponding to each first task, and to distribute the P2 level N-1 thread groups to P2 N-1 level task distribution units, where P2 is the number of level N-1 cache nodes.

[0016] For example, in an electronic device according to at least one embodiment of the present disclosure, each k-th level task distribution unit is arranged to divide a group of level k threads distributed from a received k+1-th level task distribution unit into P3 groups of level k-1 threads and distribute them to the corresponding P3 k-1-th level task distribution units, where P3 is the number of level k-1 cache nodes that share one level k cache node, the corresponding P3 k-1-th level task distribution units are P3 k-1-th level task distribution units that correspond one-to-one with P3 level k-1 cache nodes, the P3 level k-1 cache nodes share a level k cache node corresponding to the k-th level task distribution unit, and k is a positive integer greater than 1 and less than N.

[0017] For example, in an electronic device according to at least one embodiment of the present disclosure, each first level task distribution unit is configured to distribute a group of received level 1 threads to a corresponding level 1 cache node and to execute the matrix calculation task using a compute unit corresponding to the corresponding level 1 cache node.

[0018] For example, in an electronic device according to at least one embodiment of the present disclosure, the number of threads included in the level N thread group is determined by the number of level 1 cache nodes and the number of tasks that each computing unit can execute in parallel, the total number of threads included in each level k thread group is equal to the sum of the threads included in P3 level k-1 thread groups, and the total number of threads included in each level 1 thread group is equal to the number of tasks that the computing unit can execute in parallel.

[0019] For example, in an electronic device according to at least one embodiment of the present disclosure, each task distribution unit is arranged to divide a group of threads based on the data correlation between the parameters of the input matrix.

[0020] For example, in an electronic device according to at least one embodiment of the present disclosure, the parameters of the input matrix cached in a read-only cache area having read-only privileges in a level j cache node can be multiplexed by a plurality of level j-1 cache nodes that share the level j cache node and correspond to the plurality of level j-1 threads, for a plurality of level j-1 threads partitioned based on data correlations between the parameters of the input matrix.

[0021] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings of the embodiments are briefly described below. However, as will be apparent, the drawings described below are only relevant to some embodiments of this disclosure and do not limit the disclosure. [Brief explanation of the drawing]

[0022] [Figure 1] Figure 1 is a schematic diagram of a multicore chip system. [Figure 2] Figure 2 is a schematic diagram of a cache structure according to at least one embodiment of the present disclosure. [Figure 3] Figure 3 is a schematic diagram of an electronic device according to at least one embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic block diagram of an N-level task distribution unit according to at least one embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic configuration diagram of a hierarchical thread group according to at least one embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of the division relationship of a thread group according to an embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0023] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, hereinafter, while referring to the drawings of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described. As is clear, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments that can be obtained by those skilled in the art without creative labor all belong to the protection scope of the present disclosure.

[0024] Unless otherwise defined, technical terms or scientific terms used in the present disclosure should have the ordinary meaning understood by those skilled in the art. The "first", "second", and similar terms used in the present disclosure do not represent any order, quantity, or importance, but are used to distinguish different components. Similar terms such as "comprising" or "containing" mean that the elements or objects appearing before the term cover the elements or objects listed after the term and their equivalents, but do not exclude other elements or objects. Similar terms such as "connected" or "coupled" are not limited to physical or mechanical connections, and can include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] To make the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0026] Computers typically include main memory and cache. Because the processor's (single-core CPU, multi-core CPU processing core, or GPU, etc.) access speed to main memory is slower than that of the cache, the cache can be used to compensate for this slowness and improve memory access speed. The cache is usually integrated into the processor.

[0027] Figure 1 is a schematic diagram of a multicore chip system. As shown in Figure 1, the system is a typical 4-core system-on-chip, including four cores, three levels of cache memory corresponding to each of the four cores (L1 cache memory, L2 cache memory, and L3 cache memory), a network-on-chip, dynamic random memory, and other IP cores (Intellectual Property Cores). I-L1$ is the private instruction L1 cache memory for each core, D-L1$ is the private data L1 cache memory for each core, two cores share one L2 cache memory, and the four cores share an L3 cache memory. The L3 cache memory and other IP cores (e.g., direct memory access / video / display, etc.) access dynamic random memory via the network-on-chip.

[0028] L1 cache memory is closest to the core, has the smallest capacity, and is the fastest.

[0029] L2 cache memory has a larger capacity, for example, 256K, but is slower than L1 cache memory. L2 cache memory can be understood as a buffer for L1 cache memory. Because L1 cache memory is very expensive to manufacture, its capacity is limited. The function of L2 cache memory is to store data that the processor needs when processing and that L1 cache memory cannot store.

[0030] L3 cache memory has the largest capacity of the three levels of cache, for example, 12MB, but also the slowest access speed. L3 cache memory and dynamic random memory can be considered buffers for L2 cache memory. The capacity increases progressively from L1 to L3 cache memory, but the manufacturing cost per unit decreases.

[0031] When a processor operates, it first searches for the necessary data in the L1 cache memory. If it fails to find the data, it searches the L2 cache memory, and if it still fails to find the data, it searches the L3 cache memory. If the processor cannot find the necessary data in any of the three levels of cache, it retrieves the data from dynamic random memory. The longer the search path, the longer the search takes.

[0032] Similarly, the cache in a GPU (Graphics Processing Unit) also has a cache structure similar to that of a multi-core system-on-a-chip, as shown in Figure 1, including, for example, a hierarchical three-level cache structure, but a detailed explanation is omitted here.

[0033] For example, matrix calculations may include matrix multiplication and / or matrix product-sum calculations. For matrix multiplication, C(M,N) = A(M,K) × B(K,N), and the output matrix C (M rows, N columns) is obtained by multiplying the input matrix A (M rows, K columns) and the input matrix B (K rows, N columns). For example, for matrix product-sum calculations, C'(M,N) = A(M,K) × B(K,N) + C(M,N), and the updated output matrix C' is obtained by adding the result of multiplying the input matrix A (M rows, K columns) and the input matrix B (K rows, N columns) with the current value C of the output matrix C (M rows, N columns).

[0034] For matrix multiplication or matrix product-sum calculations, input matrices A and B are read-only matrices; that is, when performing a matrix calculation, the parameters of input matrices A and B are read from external memory, such as dynamic random memory, through three levels of cache to the computing unit in the Core or GPU architecture, and the calculation can be performed. The parameters of input matrices A and B in external memory cannot be overwritten. Output matrices C (and C') are read-write matrices; that is, when performing a matrix calculation, the parameters of output matrix C are read from external memory through three levels of cache to the computing unit in the Core or GPU architecture and the calculation is performed. The result C' of the calculation needs to be returned to external memory to update or overwrite the parameters of the output matrix C that were originally stored there.

[0035] Currently, in processors, L1, L2, and L3 cache memories are all read-write caches. For the matrix calculations described above, the matrix size can be large, for example, if M and N are large, the cache capacity requirements become high, leading to a large cache area and an increase in chip size. Therefore, if all matrix calculations involve data transport via read-write caches, latency will be high, design complexity will be high, and power consumption will be high.

[0036] At least one embodiment of the present disclosure provides a cache structure and an electronic device. The cache structure is used for matrix calculations, the cache structure includes an N-level cache, the i-th level cache in the N-level cache includes at least one level i cache node, each level i cache node includes a level i read-only cache area with read-only privileges and a level i read-write cache area with read-write privileges, each of multiple level i cache nodes shares one level i+1 cache node, all level N-1 cache nodes share one level N cache node included in the N-th level cache in the N-level cache, the N-th level cache is electrically connected to external memory, each level 1 cache node is electrically connected to the corresponding computing unit, the read-only cache area with read-only privileges in the N-level cache is used to cache input matrices as computation input parameters in matrix calculations and to transmit input matrices between external memory and multiple computing units, the read-write cache area with read-write privileges in the N-level cache is used to cache output matrices as computation output parameters in matrix calculations and to transmit output matrices between external memory and multiple computing units, where N is a positive integer greater than 1 and i is any positive integer between 1 and N-1.

[0037] This cache structure performs structural optimization for matrix calculations and uses a cache structure in which read-only cache areas and read-write cache areas are separated. The read-only cache area, which has read-only privileges, transmits and caches input matrices, and the read-write cache area, which has read-write privileges, transmits and caches output matrices. In each level of the cache, the proportion of read-only data is high compared to read-write data, and the design complexity of the read-only cache area is low, resulting in a small occupied area and low power consumption overhead. Therefore, this cache structure can effectively reduce the hardware overhead and complexity of the cache design and reduce the occupied area.

[0038] The cache structures of the embodiments of this disclosure may be used in a single-core CPU or a multi-core CPU, or in a GPU or GPGPU, and this disclosure is not specifically limited thereto.

[0039] The embodiments of this disclosure will be described in detail below with reference to the drawings, but this disclosure is not limited to these specific embodiments.

[0040] Figure 2 is a schematic diagram of a cache structure according to at least one embodiment of the present disclosure.

[0041] As shown in Figure 2, the cache structure includes N levels of caches. Figure 2 shows the specific structures of the Level 1, Level 2, and Level N caches. The structures of the other levels of caches are similar and are therefore not shown again.

[0042] For example, in an N-level cache, the i-th level cache includes at least one level i cache node, each level i cache node includes a level i read-only cache area with read-only privileges and a level i read-write cache area with read-write privileges, where i is any positive integer between 1 and N-1.

[0043] For example, as shown in Figure 2, when i=1, the level 1 cache includes multiple level 1 cache nodes, each level 1 cache node containing one level 1 read-only cache area and one level 1 read-write cache area. When i=2, the level 2 cache includes multiple level 2 cache nodes, each level 2 cache node containing one level 2 read-only cache area and one level 2 read-write cache area.

[0044] For example, as shown in Figure 2, the Nth level cache in an N-level cache includes one level N cache node, and the level N cache node includes one level N read-only cache area with read-only privileges and one level N read-write cache area with read-write privileges.

[0045] For example, each of several level i cache nodes shares one level i+1 cache node, and all level N-1 cache nodes share a level N cache node. Here, sharing means that the multiple (e.g., M) level i cache nodes are electrically connected to the level i+1 cache node, and each of the M level i cache nodes can exchange data with the level i+1 cache node, for example, by obtaining necessary data from the level i+1 cache node.

[0046] For example, as shown in Figure 2, multiple Level 1 cache nodes share one Level 2 cache node, multiple Level 2 cache nodes share one Level 3 cache node, and all Level N-1 cache nodes share a Level N cache node.

[0047] For example, the configuration of which cache nodes share one higher-level cache node at each level can be set as needed, and each level may be different, and this disclosure does not specifically limit this. For example, in response to N=4, it is possible to configure it so that every four level 1 cache nodes share one level 2 cache node, every eight level 2 cache nodes share one level 3 cache node, and all level 3 cache nodes share a level 4 cache node.

[0048] For example, as shown in Figure 2, the Nth level cache is electrically connected to an external memory, which may be, for example, a dynamic random memory.

[0049] For example, as shown in Figure 2, each level 1 cache node is electrically connected to the corresponding computing unit.

[0050] For example, the computing unit may be a Core, or in a GPU architecture, the computing unit may include an arithmetic module and a register array.

[0051] For example, each register array is used to store some parameters in the input matrix and some parameters in the output matrix, and the stored parameters are determined based on the operations that the arithmetic module of the computing unit needs to perform. For example, for one large matrix multiply-accumulate task, assume that each arithmetic module has four threads (e.g., four multipliers) and can perform the calculation C'(2×2)=C(2×2)+A(2×1)×B(1×2) in parallel in each operation cycle, so that each computing unit caches the parameters A(2×1) (2 rows, 1 column) and B(1×2) (1 row, 2 columns) that it needs to perform the calculation in its register array, and simultaneously caches C'(2×2) (2 rows, 2 columns) and / or C(2×2) in its register array.

[0052] Each computing unit's arithmetic module includes multiple multipliers and / or sum-accumulates for performing multiplication and / or sum-accumulate calculations in parallel. For example, one arithmetic module may include four threads, such as four multipliers or four sum-accumulates, and can perform four computing tasks, such as four sets of multiplication or sum-accumulate calculations, in parallel.

[0053] For example, a read-only cache area with read-only access in an N-level cache is used to cache input matrices as computation input parameters in matrix calculations and to transmit input matrices between external memory and multiple computing units. For example, a read-write cache area with read-write access in an N-level cache is used to cache output matrices as computation output parameters in matrix calculations and to transmit output matrices between external memory and multiple computing units.

[0054] For example, matrix calculations include matrix multiplication and / or matrix sum-of-products calculations. In response to matrix multiplication being expressed as C = A × B, and matrix sum-of-products being expressed as C' = A × B + C, A and B represent input matrices as calculation input parameters, and C and C' represent output matrices as calculation output parameters, where C represents the current value of the output matrix, i.e., the result of the previous sum-of-products calculation, and C' represents the updated value of the output matrix after the current sum-of-products calculation.

[0055] For example, input matrices A and B, used as computation input parameters, are read into the corresponding computation unit's register array via a read-only cache path consisting of a read-only cache area with read-only privileges in an N-level cache.

[0056] For example, using Figure 2 as an example, for any computing unit, the parameters of the input matrices (e.g., input matrices A and B) required for it to perform matrix calculations are first searched for in the Level 1 read-only cache area (e.g., Level 1 read-only cache area 1) of the Level 1 cache node electrically connected to the computing unit. If a miss is found, the parameters are then searched for in the read-only cache area (e.g., Level 2 read-only cache area 1) of the Level 2 cache node electrically connected to the Level 1 cache node. If a miss is still found, the parameters are then searched for in the read-only cache area (e.g., Level 3 read-only cache area 1) of the Level 3 cache node electrically connected to the Level 2 cache node. If these parameters are still not found up to Level N cache nodes, they are read from external memory and loaded into the computing unit's register array via a read-only cache path consisting of Level N cache nodes, ..., Level 3 read-only cache area 1, Level 2 read-only cache area 1, and Level 1 read-only cache area 1.

[0057] For example, if a parameter required by a read-only cache area of ​​a Level 2 cache node is hit, the parameter is read from the read-only cache area of ​​the Level 2 cache node and loaded into the register array of the compute unit via Level 2 read-only cache area 1 and Level 1 read-only cache area 1 in the read-only cache path.

[0058] For example, the output matrix C, as a computation output parameter, is read into the register array of the corresponding computation unit via a read / write cache path consisting of read / write cache areas with read / write privileges in an N-level cache. The computation module of the corresponding computation unit then calculates the updated output matrix C', and the updated output matrix C' is written back from the corresponding computation unit to external memory or a read / write cache area in a certain level of cache via the read / write cache path.

[0059] For example, using Figure 2 as an example, in response to a matrix multiply-accumulate calculation, the parameters of the output matrix required for the calculation (e.g., the current value C of the output matrix) for any computing unit are first searched for in the level 1 read / write cache area (e.g., level 1 read / write cache area 1) of a level 1 cache node electrically connected to the computing unit. If a miss is found, the parameters are then searched for in the read / write cache area (e.g., level 2 read / write cache area 1) of a level 2 cache node electrically connected to the level 1 cache node. If a miss is still found, the parameters are then searched for in the read / write cache area (e.g., level 3 read / write cache area 1) of a level 3 cache node electrically connected to the level 2 cache node. If these parameters are missed up to level N cache nodes, they are read from external memory and loaded into the register array of the computing unit via a read / write cache path consisting of level N cache nodes, ..., level 3 read / write cache area 1, level 2 read / write cache area 1, and level 1 read / write cache area 1.

[0060] Subsequently, the calculation module of the computing unit completes the multiply-accumulate calculation to obtain the updated output matrix C', which is then written back from the computing unit to external memory or to the read-write cache area of ​​any level of cache via a read-write cache path consisting of level 1 read-write cache area 1, level 2 read-write cache area 1, level 3 read-write cache area 1, ..., level N cache nodes.

[0061] For example, consider a 3-level cache structure, i.e., in response to N being 3, the level 1 cache includes multiple level 1 cache nodes, each level 1 cache node includes one level 1 read-only cache area and one level 1 read-write cache area, each level 1 cache node corresponds to and is electrically connected to one computing unit; the level 2 cache includes multiple level 2 cache nodes, each level 2 cache node includes one level 2 read-only cache area and one level 2 read-write cache area, each M (e.g., 2, 4, 8, etc.) level 1 cache node shares one level 2 cache node, i.e., the M level 1 cache nodes are electrically connected to the level 2 cache node; the level 3 cache includes one level 3 cache node, each level 3 cache node includes one level 3 read-only cache area and one level 3 read-write cache area, all level 2 cache nodes share the level 3 cache node, i.e., all level 2 cache nodes are electrically connected to the level 3 cache node, and the level 3 cache node is electrically connected to external memory.

[0062] For example, a read-only cache path includes a read-only cache area in a Level 1 cache node corresponding to a corresponding compute unit, a read-only cache area in a Level 2 cache node electrically connected to a Level 1 cache node, and a read-only cache area in a Level 3 cache node; and a read-write cache path includes a read-write cache area in a Level 1 cache node corresponding to a corresponding compute unit, a read-write cache area in a Level 2 cache node electrically connected to a Level 1 cache node, and a read-write cache area in a Level 3 cache node.

[0063] For example, input matrices A and B are read via a read-only cache path from external memory or a read-only cache area at a certain level of cache into the register array of the corresponding computing unit, output matrix C is read via a read-write cache path from external memory or a read-write cache area at a certain level of cache into the register array of the corresponding computing unit, and after the arithmetic module completes the calculation C' = A × B + C, the updated output matrix C' is written back via a read-write cache path to a read-write cache area at a certain level of cache or to external memory.

[0064] For example, in one embodiment, input matrices A and B, and output matrices C and C' are all 256×256 size matrices with INT8 precision (data size is 1 byte), and we consider the implementation of matrix sum-of-products calculation C'=C+A×B. Assume that each calculation unit has four threads (or four summators) and that C'(2×2)=C(2×2)+A(2×1)×B(1×2) can be processed in parallel in each operation cycle, where A(2×1) represents the 2x1 parameter in input matrix A, B(1×2) represents the 1x2 parameter in input matrix B, C(2×2) represents the current 2x2 value of output matrix C corresponding to A(2×1)×B(1×2) obtained in the previous sum-of-products calculation, and C'(2×2) represents the updated result of the current sum-of-products calculation.

[0065] For example, the cache structure provides a total of 64 Level 1 cache nodes, with every four Level 1 cache nodes sharing one Level 2 cache node, and 16 Level 2 cache nodes sharing one Level 3 cache node.

[0066] For example, if the data transport delay from a Level 1 cache node to a compute unit is 8 clock cycles, the transport delay from a Level 2 cache node (via a Level 1 cache node) to a compute unit is 32 clock cycles, and the transport delay from a Level 3 cache node (via Level 1 and Level 2 cache nodes) to a compute unit is 64 clock cycles, then the amount of data stored in each Level 1 cache node is: Eight sets of A(2×1) and B(1×2), i.e., 8×(2×1+1×2)×1byte = 32 bytes, are stored in the Level 1 read-only cache area. One set of C(2×2), i.e., 1×2×2×1byte = 4 bytes, is stored in the level 1 read / write cache area.

[0067] Therefore, the ratio of data volume between the Level 1 read-only cache area and the Level 1 read-write cache area is 8:1.

[0068] The Level 2 read-only cache area stores 32 sets of A(4×1) and B(1×4), i.e., 32×(4×1+1×4)×1byte = 256 bytes, and the Level 2 read-write cache area stores 1 set of C(4×4), i.e., 1×4×4×1byte = 16 bytes. Therefore, the ratio of data volume between the Level 2 read-only cache area and the Level 2 read-write cache area is 16:1.

[0069] The Level 3 read-only cache area stores 64 sets of A(16×1) and B(1×16), i.e., 64×(16×1+1×16)×1byte = 2048 bytes, and the Level 3 read-write cache area stores 1 set of C(16×16), i.e., 16×16×1byte = 256 bytes. Therefore, the ratio of data volume between the Level 3 read-only cache area and the Level 3 read-write cache area is 8:1.

[0070] As can be seen from the proportional relationship, the proportion of read-only data is very high compared to read-write data, and this is related to the matrix size involved in the calculation, the data transport delay, and the computing power (computational hardware resources) of the computing unit. The size of input matrices A and B and the data transport delay determine the amount of read-only data, and the computing power of the computing unit determines the amount of read-write data. Since the hardware resources of the computing unit are limited, the amount of read-write data is limited. However, the size of input matrices A and B can increase depending on the application needs, and the data transport delay increases with increasing chip size, so the cache capacity for storing input matrices A and B is relatively large. Therefore, based on the read-only characteristics of input matrices A and B, designing the cache for storing input matrices A and B as a read-only cache can effectively reduce the complexity of the cache design and decrease hardware overhead and the area occupied by the cache structure.

[0071] In a cache structure according to at least one embodiment of this disclosure, a cache structure is designed in which read / write and read-only are separated, making it easier to perform matrix calculations. In matrix calculations, the input matrix has read-only properties, and the output matrix has read / write properties. The data of the input matrix is ​​cached in the read-only cache area of ​​each level of cache, and the data of the output matrix is ​​cached in the read / write cache area of ​​each level of cache. Because the proportion of read-only data is very high compared to read / write data, this configuration can save the occupied area of ​​the cache structure, and because the design complexity of the read-only cache area is low and power consumption is low, the overall design complexity and power consumption of the cache structure can be reduced. In particular, when the size of the input matrix is ​​large, the input matrix and output matrix are stored in different cache areas and operate independently of each other without interfering with one another. Therefore, when reading the parameters of input matrices A and B, the parameters of output matrix C are not flushed, reducing delay due to data transport, reducing hardware overhead, and improving computational efficiency.

[0072] At least one embodiment of the present disclosure further provides an electronic device. Figure 3 is a schematic diagram of an electronic device according to at least one embodiment of the present disclosure.

[0073] As shown in Figure 3, the electronic device 100 includes a cache structure 101 described in any embodiment of the present disclosure. The structure, function, and technical effects of the cache structure 101 are as described above and are omitted here for further detail.

[0074] As shown in Figure 3, the electronic device 100 further includes an N-level cache, an N-level task distribution unit 102 with a one-to-one correspondence to the N-level cache, and a group of N hierarchical threads 103.

[0075] For example, each level of task delivery units includes at least one task delivery unit, and each level of threads includes at least one thread group.

[0076] For example, a group of threads refers to a combination of a set of threads. All threads in each group execute the same instructions, but each thread reads and writes its own private data.

[0077] For example, the task distribution unit may be implemented in hardware form, for example by a dedicated hardware structure, or in software form, for example in the form of program code, and this disclosure is not specifically limited thereto.

[0078] For example, in an N-level task distribution unit, the number of j-level task distribution units located at the j-th level is the same as the number of level j cache nodes included in the corresponding j-level cache in an N-level cache, and there is a one-to-one correspondence, where j is a positive integer less than or equal to N.

[0079] For example, if the j-th level cache contains J level j cache nodes, where J is a positive integer, then the number of level j task distribution units located at the j-th level is also J, and there is a one-to-one correspondence with the J level j cache nodes, with each level j task distribution unit managing its corresponding level j cache node.

[0080] For example, the Nth level cache includes one level N cache node, and the number of Nth level task distribution units located at the Nth level in an N-level task distribution unit is also 1, and the Nth level task distribution unit manages the level N cache node.

[0081] For example, in a set of N hierarchical threads, the number of level j threads at the j-th level corresponds one-to-one with the number of level j cache nodes in the corresponding j-th level cache in an N-level cache, and the threads in each level j thread group share the corresponding level j cache node to cache data.

[0082] For example, the j-th level cache includes J level j cache nodes, the number of level j threads is also J, and there is a one-to-one correspondence between the J level j cache nodes and the J threads. For example, a thread in level j thread group 1 of J level j threads can share the corresponding level j cache node (e.g., level j cache node 1) in the J level j cache nodes to cache or transmit data.

[0083] For example, the N-level thread groups described here are threads executed within a single operation cycle, such as a single operation time. Based on different computation tasks, computation tasks can be divided into multiple levels of N thread groups and executed in time-sharing across different operation cycles.

[0084] For example, a level N thread group contains 256 threads, and these 256 threads can be further divided into multiple level N-1 thread groups, and each level N-1 thread group can be further divided into multiple level N-2 thread groups. Ultimately, the number of threads in each level 1 thread group is determined by the number of tasks that a computing unit can execute in parallel, and so on. The number of threads in a level N thread group is determined by the number of tasks that each computing unit can execute in parallel and the total number of computing units (or level 1 cache nodes). For example, the number of threads in a level N thread group is the product of the number of tasks that each computing unit can execute in parallel and the total number of computing units (or level 1 cache nodes).

[0085] For example, in an N-level task distribution unit, the Nth-level task distribution unit is arranged to divide the total number of tasks for matrix calculations into P1 first tasks, each assigned to a group of P1 level N threads, and executed in a time-sharing manner, where P1 is a positive integer. For example, each first task is executed by one corresponding group of level N threads.

[0086] For example, the value of P1 can be determined based on the total number of threads and the total amount of data for all tasks included in a Level N thread group. For example, let's take the example of input matrices A, B, C, and C' being 256x256 matrices with INT8 precision (data size of 1 byte), and performing a matrix product sum calculation C' = C + A × B. Assuming that the total number of threads in the Level N thread group is 64 × 4 = 256, and that there are a total of 64 Level 1 cache nodes, and each computing unit can execute 4 tasks in parallel, then P1 = 256 × 256 / 256 = 256.

[0087] For example, in some embodiments, within a single operation cycle, the electronic equipment is arranged to perform a first task corresponding to a group of level N threads.

[0088] For example, in this embodiment, all computing units can be uniformly scheduled to execute the same first task within a single operation cycle. For instance, P1 first tasks can be executed sequentially using time-sharing.

[0089] For example, in another embodiment, the electronic device reads the parameters of the input matrix required to execute multiple first tasks out of P1 first tasks into a level N cache node in a batch, and schedules multiple computing units to execute multiple first tasks within multiple operation cycles.

[0090] For example, in this embodiment, each computing unit can schedule and control independently, and perform out-of-order execution on the multiple first tasks by switching threads. For example, depending on the size of the memory space, if the memory space stores task information for D (where D is a positive integer greater than 1) first tasks, and the task information includes the position, coordinates, etc., of the input matrix parameters required for the computation of each first task, then the parameters of the input matrices required for the D first tasks can be read all at once into a level N cache node, and each computing unit can decide which first task to execute now based on whether the input data required for computation is ready. For example, in this embodiment, instead of executing multiple level N thread groups simultaneously in one operation cycle, each computing unit can be controlled independently and execute the same or different first tasks within one operation cycle, and after multiple operation cycles, the D first tasks can be executed as simultaneously as possible.

[0091] For example, the Nth-level task distribution unit further selects at least one first task and divides each first task into P2 second tasks, thereby obtaining P2 levels N-1 thread groups divided from the level N thread group corresponding to each first task, and is arranged to distribute the P2 levels N-1 thread groups to P2 N-1-level task distribution units, where P2 is the number of level N-1 cache nodes.

[0092] For example, an N-level task distribution unit can divide the total number of tasks for matrix calculations into P1 first tasks, and can select and divide one or more first tasks. For each first task, it can be divided into P2 second tasks, that is, the level N thread group corresponding to the first task can be divided into P2 level N-1 thread groups, and these P2 level N-1 thread groups can be distributed to P2 N-1 level task distribution units.

[0093] For example, distributing P2 groups of level N-1 threads to P2 N-1 level task distribution units may include assigning and storing task information for the divided second tasks, such as the position and coordinates in the input matrix of parameters required for the calculation of each second task, to the corresponding second level task distribution unit.

[0094] For example, the selection of the number of divided first tasks can be determined based on the size of the memory space for the task information of the second tasks that the second-level task distribution unit can store. For instance, if it can only store the task information of P2 second tasks, one first task can be divided; if it can store the task information of multiple sets of P2 second tasks, multiple first tasks can be divided.

[0095] For example, each k-th level task distribution unit is configured to divide the group of level k threads distributed from the (k+1)th level task distribution unit into P3 groups of level k-1 threads and distribute them to the corresponding P3 (k-1)th level task distribution units. For example, P3 is the number of level k-1 cache nodes that share one level k cache node, and the corresponding P3 (k-1)th level task distribution units are P3 (k-1)th level task distribution units that correspond to P3 level k-1 cache nodes that share the level k cache node corresponding to the k-th level task distribution unit. Here, k is a positive integer greater than 1 and less than N.

[0096] For example, each first-level task distribution unit is configured to distribute the received group of level 1 threads to the corresponding level 1 cache node and to execute matrix calculation tasks using the computing unit corresponding to the level 1 cache node, for example, by using the arithmetic module of the computing unit corresponding to the level 1 cache node to perform specific multiplication or sum-of-accumulate calculations.

[0097] For example, the number of threads in a level N thread group is determined by the number of level 1 cache nodes and the number of tasks that each computing unit can execute in parallel. The total number of threads in each level k thread group is equal to the sum of the threads in P3 level k-1 thread groups, and the total number of threads in each level 1 thread group is equal to the number of tasks that each computing unit can execute in parallel.

[0098] Figure 4 is a schematic block diagram of an N-level task delivery unit according to at least one embodiment of the present disclosure.

[0099] Figure 4 shows the structure of an N-level task delivery unit corresponding to the case where N=3, i.e., a 3-level cache structure. Of course, this disclosure is not limited to this, and the structural relationships for other values ​​of N are similar and are not shown again here.

[0100] For example, as shown in Figure 4, the third-level task distribution unit corresponds to a level 3 cache, has a quantity of 1, and corresponds to a level 3 cache node. The level 3 task distribution unit is configured to divide the total tasks for matrix calculations into P1 first tasks, assign each to a group of P1 level 3 threads, and execute them in a time-sharing manner. For example, the P1 first tasks may be executed sequentially in different operation cycles as described above, or out-of-order execution may be performed on the P1 first tasks by switching threads, and this disclosure is not specifically limited thereto.

[0101] For example, a level 3 task distribution unit is further configured to select at least one first task, for example, one or more or P1 first tasks, and divide each first task into P2 second tasks, thereby obtaining P2 level 2 thread groups divided from the level 3 thread group corresponding to each first task, and to distribute these P2 level 2 thread groups to P2 second level task distribution units.

[0102] For example, the number of Level 2 task distribution units is the same as the number of Level 2 cache nodes and corresponds to them one-to-one, while the number of Level 1 task distribution units is the same as the number of Level 1 cache nodes and corresponds to them one-to-one.

[0103] For example, each second-level task distribution unit is configured to divide the group of level 2 threads distributed from the third-level task distribution unit into three groups of level 1 threads and distribute them to the corresponding three first-level task distribution units.

[0104] For example, regarding the second-level task distribution unit 1, it corresponds to a level 2 cache node 1, which is shared by P3 level 1 cache nodes. In this case, the second-level task distribution unit 1 is configured to distribute the divided P3 level 1 threads to P3 first-level task distribution units corresponding to the P3 level 1 cache nodes.

[0105] For example, each first-level task distribution unit is configured to distribute the received group of level 1 threads to the corresponding level 1 cache node to execute a specific matrix calculation task.

[0106] For example, in one embodiment, input matrix A, input matrix B, output matrices C and C' are still 256x256 matrices with INT8 precision (data size is 1 byte), and the matrix product sum calculation C' = C + A × B is implemented.

[0107] For example, the cache structure provides a total of 64 Level 1 cache nodes, with every four Level 1 cache nodes sharing one Level 2 cache node, and 16 Level 2 cache nodes sharing one Level 3 cache node.

[0108] Therefore, a Level 1 thread group contains 4 threads, a Level 2 thread group contains 4 × 4 = 16 threads, and a Level 3 thread group contains 4 × 4 × 16 = 256 threads.

[0109] The output matrix C has a size of 256 × 256 (256 rows and 256 columns), and is divided into a total of 256 × 256 / 256 = 256 level 3 thread groups. Therefore, each level 3 thread group can perform the calculation C'(16 × 16) = C(16 × 16) + A(16 × 256) × B(256 × 16), each level 2 thread group divided from one level 3 thread group can perform the calculation C'(4 × 4) = C(4 × 4) + A(4 × 256) × A(256 × 4), and each level 1 thread group divided from one level 2 thread group can perform the calculation C'(2 × 2) = C(2 × 2) + A(2 × 256) × B(256 × 2).

[0110] For example, the third-level task distribution unit divides the total number of matrix calculation tasks into 256 first tasks, and each first task corresponds to a level 3 thread group containing 256 threads. Furthermore, the third-level task distribution unit divides each first task into 16 second tasks, that is, it divides the 256 threads in the level 3 thread group into 16 level 2 thread groups, each containing 16 threads, and arranges to assign the 16 level 2 thread groups corresponding to the 16 second tasks to 16 second-level task distribution units.

[0111] For example, each Level 2 task distribution unit divides the single group of Level 2 threads containing 16 threads, which it received from the Level 3 task distribution unit, into four groups of Level 1 threads and distributes them to the corresponding four Level 1 task distribution units. Each Level 1 task distribution unit then assigns four threads from the received Level 1 thread group to an arithmetic module to perform matrix calculations.

[0112] For example, if the parallel processing capability of each computing unit is to complete C'(2×2)=C(2×2)+A(2×1)×B(1×2) in each operation cycle, then 256 operation cycles are required to complete the calculation C'(2×2)=C(2×2)+A(2×256)×B(256×2). Assuming all computing units perform calculations simultaneously, a total of 256 clock cycles are required for one group of level 3 threads to perform the calculation C'(16×16)=C(16×16)+A(16×256)×B(256×16).

[0113] Figure 5 is a schematic diagram of a hierarchical thread group according to at least one embodiment of the present disclosure.

[0114] As shown in Figure 5, the third-level task unit is arranged so that one total task for matrix calculation is divided into multiple first-level tasks, with each first-level task corresponding to one level 3 thread group. Based on the above, one level 3 thread group is divided into multiple level 2 thread groups, for example, each level 2 thread group corresponds to one second task divided from the first task. Based on the above, each level 2 thread group is also divided into multiple level 1 thread groups, for example, each level 1 thread group corresponds to one third task divided from the second task.

[0115] For hierarchical task delivery, this method effectively utilizes the correlation between parameters, thereby ensuring task localization and the execution of tasks with similar parameter locations in a single thread group as much as possible. It also ensures that tasks related to the next level of cache nodes belonging to the same cache node are closely correlated, maximizing data multiplexing, improving the multiplexing rate of cache data, reducing the number of data reads from external memory, decreasing cache capacity, and reducing the overhead of cache hardware resources.

[0116] For example, each task distribution unit is configured to divide thread groups based on the data correlation between parameters in the input matrix.

[0117] For example, for multiple groups of level j-1 threads partitioned based on the data correlation between parameters of the input matrix, data cached in a read-only cache area with read-only privileges in a level j cache node can be multiplexed by multiple level j-1 caches that share a level j cache node and correspond to multiple groups of level j-1 threads.

[0118] Figure 6 is a schematic diagram of the thread group division relationship according to one embodiment of the present disclosure.

[0119] As shown in Figure 6, the gray rectangle represents the data stored in the level 3 read-only cache area for input matrix A, which is read from external memory when a certain first task is executed, and the white rectangle represents the data stored in the level 3 read-only cache area for input matrix B, which is read from external memory when the same first task is executed.

[0120] For example, let's assume the cache structure provides a total of 64 level 1 cache nodes, with every four level 1 cache nodes sharing one level 2 cache node, and the 16 level 2 cache nodes sharing one level 3 cache node.

[0121] For example, in Figure 6, a dashed box indicates one level 2 cache node, and a total of 16 level 2 cache nodes are shown in Figure 6. Each level 2 cache node includes a level 2 read / write cache area for caching data of output matrix C, and a level 2 read-only cache area. The level 2 read-only cache area is used to cache the data of input matrix A required by the cache node (for example, represented as a gray rectangular box located to the left of each level 2 read / write cache area in Figure 6) and the data of input matrix B required by the cache node (for example, represented as a white rectangular box located above each level 2 read / write cache area shown in Figure 6).

[0122] For example, in the example in Figure 6, the data of input matrices A and B in a level 3 read-only cache area can be multiplexed by the 16 level 2 read-only cache areas belonging to it. For example, if one group of level 3 threads is divided into 16 groups of level 2 threads, and these 16 groups of level 2 threads are arranged in a 4x4 configuration, then the data of input matrix A in each level 2 read-only cache area (gray rectangles adjacent to each level 2 read-write cache area) is multiplexed by the four horizontal level 2 read-only cache areas, and the data of each input matrix B (white rectangles adjacent to each level 2 read-write cache area) is multiplexed by the four vertical level 2 read-only cache areas.

[0123] Similarly, the data of input matrices A and B in a level 2 read-only cache area can be multiplexed by the four level 1 read-only cache areas belonging to it, a detailed explanation of which is omitted here.

[0124] Of course, Figure 6 shows one possible partitioning policy; for example, 16 level 2 read-only cache areas could be arranged in a 1x16 configuration. In this case, the 16 level 2 read-only cache areas could multiplex the data of input matrix A in common, but they could not multiplex the data of input matrix B.

[0125] In the above embodiment, the task distribution hierarchical policy ensures data multiplexing as much as possible, enables the division of thread groups based on data correlation, reduces the data size requiring caching, decreases cache capacity, and reduces the overhead of cache hardware resources.

[0126] For example, the optimization for GPUs or GPGPUs by the cache structure and task delivery unit is more pronounced, and the electronic device can be implemented as a GPGPU or GPU, for instance. Of course, this disclosure is not limited thereto, and the electronic device can be implemented as any electronic device requiring a cache structure such as a single-core CPU, multi-core CPU, GPU, or GPGPU. For example, the central processing unit (CPU) may be an X86 or ARM architecture, and this disclosure does not specifically limit the architecture of the electronic device.

[0127] Of course, the N-level task delivery units and N-tiered thread groups in the electronic device according to at least one embodiment of the present disclosure are not limited to the cache structure according to at least one embodiment of the present disclosure, but can be applied to other tiered cache structures, and the present disclosure does not specifically limit them.

[0128] Furthermore, while this disclosure references various units of the systems relating to embodiments thereof, any number of different units may be used and run on the client and / or server. The units are illustrative only, and different units may be used in different aspects of the system and method.

[0129] As those skilled in the art will understand, a computer program can instruct the relevant hardware to complete all or some of the steps of the above method, and the program is stored in a computer-readable storage medium, such as read-only memory, magnetic disk, or optical disk. Optionally, all or some of the steps of the above embodiment may be implemented using one or more integrated circuits. Accordingly, each module / unit of the above embodiment may be implemented in hardware form or in software function module form. This disclosure is not limited to any particular combination of hardware and software.

[0130] Unless otherwise specified, all terms used herein have the same meaning as those commonly understood by those skilled in the art. Furthermore, it should be understood that terms as defined in ordinary dictionaries, unless explicitly defined herein, should not be described in an idealized or highly formalized sense, but rather should be interpreted as having a meaning consistent with their meaning in the context of the relevant art.

[0131] The above is a description of the Disclosure and should not be considered a limitation thereon. While several exemplary embodiments of the Disclosure have been described, those skilled in the art will readily understand that many of these exemplary embodiments can be modified without departing from the novel education and merits of the Disclosure. Accordingly, all such modifications are intended to fall within the scope of the Disclosure, limited to the claims. As should be understood, the above is a description of the Disclosure and should not be considered limited to any specific embodiment disclosed, and the intent to modify the disclosed embodiments and other embodiments falls within the scope of the appended claims. The Disclosure is limited by the claims and their equivalents.

Claims

1. A cache structure for matrix calculations, wherein the cache structure includes an N-level cache. In the aforementioned N-level cache, the i-th level cache includes at least one level i cache node, and each level i cache node includes a level i read-only cache area with read-only privileges and a level i read-write cache area with read-write privileges. Each of the multiple level i cache nodes shares one level i+1 cache node, all level N-1 cache nodes share one level N cache node included in the Nth level cache of the N level cache, and the level N cache node includes a level N read-only cache area with read-only privileges and a level N read-write cache area with read-write privileges. The N-level cache is electrically connected to external memory, and each level 1 cache node is electrically connected to the corresponding computing unit. The read-only cache area having read-only access in the N-level cache is used to cache the input matrix as a computation input parameter in the matrix calculation, and to transmit the input matrix between the external memory and the multiple computation units. The read / write cache area having read / write access in the N-level cache is used to cache the output matrix as a computation output parameter in the matrix calculation, and to transmit the output matrix between the external memory and the multiple computation units. A cache structure where N is a positive integer greater than 1 and i is any positive integer between 1 and N-1.

2. Each computing unit includes an arithmetic module and a register array. The register array of each calculation unit is used to store some of the parameters in the input matrix and some of the parameters in the output matrix. The cache structure according to claim 1, wherein the arithmetic module of each computing unit includes a plurality of multipliers and / or sum-accumulates for performing multiplication and / or sum-accumulate calculations in parallel.

3. The cache structure according to claim 1 or 2, wherein the matrix calculation includes matrix multiplication and / or matrix sum-of-products calculations.

4. The input matrix, as a computation input parameter, is read into the register array of the corresponding computation unit via a read-only cache path consisting of a read-only cache area with read-only privileges in the N-level cache. The cache structure according to any one of claims 1 to 3, wherein the output matrix as a computation output parameter is read into a register array of the corresponding computation unit via a read / write cache path consisting of a read / write cache area having read / write privileges in the N-level cache, and after computation by the arithmetic module of the corresponding computation unit, an updated output matrix is ​​obtained, and the updated output matrix is ​​written back from the corresponding computation unit to the external memory or a read / write cache area of ​​any level of cache via the read / write cache path.

5. In response to N being 3, The read-only cache path includes a read-only cache area in a level 1 cache node corresponding to the corresponding compute unit, a read-only cache area in a level 2 cache node electrically connected to the level 1 cache node, and a read-only cache area in a level 3 cache node. The cache structure according to claim 4, wherein the read / write cache path includes a read / write cache area in a level 1 cache node corresponding to the corresponding computing unit, a read / write cache area in a level 2 cache node electrically connected to the level 1 cache node, and a read / write cache area in a level 3 cache node.

6. The cache structure according to any one of claims 1 to 5, wherein the external memory is a dynamic random memory.

7. Electronic device comprising a cache structure according to any one of claims 1 to 6.

8. The electronic device includes an N-level task distribution unit that corresponds one-to-one with the N-level cache, and a group of N hierarchical threads. Each level of task delivery units includes at least one task delivery unit, and each level of threads includes at least one thread group. In the aforementioned N-level task distribution unit, the number of j-level task distribution units located at the j-th level is the same as the number of level j cache nodes included in the corresponding j-level cache in the aforementioned N-level cache, and there is a one-to-one correspondence between them. In the aforementioned N-level thread group, the number of level j threads located at the j-th level is the same as the number of level j cache nodes included in the corresponding j-th level cache in the aforementioned N-level cache, and there is a one-to-one correspondence between them. The threads included in each level j thread group share the corresponding level j cache node to cache data. The electronic device according to claim 7, wherein j is a positive integer less than or equal to N.

9. The electronic device according to claim 8, wherein the quantity of the N-level task distribution unit located at the N-th level in the N-level task distribution unit is 1, and the total tasks for matrix calculation are divided into P1 first tasks, each assigned to a group of P1 level N threads and arranged to be executed in a time-sharing manner, where P1 is a positive integer.

10. The electronic device according to claim 9, wherein the electronic device is arranged to perform a first task corresponding to a group of level N threads within a single operation cycle.

11. The electronic device reads the parameters of the input matrix necessary to execute multiple first tasks from the P1 first task into the level N cache node all at once, and The electronic device according to claim 9, which is arranged to schedule a plurality of computing units to execute a plurality of first tasks within a plurality of operation cycles.

12. The N-level task distribution unit is further configured to select at least one first task, divide each first task into P2 second tasks, thereby obtaining P2 level N-1 thread groups divided from the level N thread group corresponding to each first task, and to distribute the P2 level N-1 thread groups to P2 N-1 level task distribution units. The electronic device according to any one of claims 9 to 11, wherein P2 is the number of level N-1 cache nodes.

13. Each k-th level task distribution unit is configured to divide the group of level k threads distributed from the k+1-th level task distribution unit into P3 groups of level k-1 threads and distribute them to the corresponding P3 k-1-th level task distribution units. P3 is the number of level k-1 cache nodes that share one level k cache node, and the corresponding P3 level k-1 task distribution units are P3 level k-1 task distribution units that correspond one-to-one with P3 level k-1 cache nodes, and the P3 level k-1 cache nodes share the level k cache node corresponding to the k level task distribution unit. The electronic device according to any one of claims 9 to 12, wherein k is a positive integer greater than 1 and less than N.

14. The electronic device according to any one of claims 8 to 13, wherein each first-level task distribution unit is arranged to distribute a group of received level 1 threads to a corresponding level 1 cache node and to execute the matrix calculation task using a computing unit corresponding to the corresponding level 1 cache node.

15. The number of threads included in the aforementioned Level N thread group is determined by the number of Level 1 cache nodes and the number of tasks that each computing unit can execute in parallel. The total number of threads in each level k thread group is equal to the sum of the threads in the P3 level k-1 thread groups. The electronic device according to claim 13 or 14, wherein the total number of threads included in each level 1 thread group is equal to the number of tasks that the computing unit can execute in parallel.

16. The electronic device according to any one of claims 8 to 15, wherein each task distribution unit is arranged to divide the thread group based on the data correlation between the parameters of the input matrix.

17. The electronic device according to claim 16, wherein, for a plurality of level j-1 thread groups divided based on the data correlation between the parameters of the input matrix, the parameters of the input matrix cached in a read-only cache area having read-only privileges in a level j cache node can be multiplexed by the plurality of level j-1 cache nodes, and the plurality of level j-1 cache nodes share the level j cache node and correspond to the plurality of level j-1 thread groups.

Images