Computing device, data processing method for computing device, computer readable storage medium and computer program product
By introducing group-shared memory on the AI chip, the tensor core and vector core can access each other, solving the flexibility and performance improvement of storage model design on the AI chip, and achieving more efficient data processing and operator fusion.
Patent Information
- Application Number
- CN202411109838.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-13
AI Technical Summary
Designing efficient storage models on AI chips to improve the performance of AI models is a challenge, especially due to limited area and insufficient flexibility in data interactions in existing designs, resulting in limited performance improvement.
The data interaction is performed using group shared memory between the tensor kernel and the vector kernel, allowing the tensor kernel and the vector kernel to be producers and consumers, and data interaction is performed through on-chip memory, avoiding direct access to off-chip memory and data. Pre-arrangement requirements.
It improves the flexibility and performance of data processing, reduces the limitations on software programming, enhances the fusion capability of operators and the efficiency of on-chip computing, and is suitable for a variety of operator requirements.
Smart Images

Figure CN118643005B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the field of processors, and more particularly, to a data processing method for a computing device, a computing device, a computer-readable storage medium, and a computer program product. Background Art
[0002] In the field of artificial intelligence (AI), AI model training and reasoning involve a large amount of parallel computing. Currently, AI chips are usually used to accelerate AI models. The basic idea is to use a large number of processors to jointly solve the same problem. In this process, the computing power of the chip, the read and write capabilities of the memory, and their interaction methods play a key role in performance.
[0003] In order to improve performance, the design guidelines of AI chips mainly include: using tensor cores that occupy a larger chip area to perform matrix operations, and using vector cores to perform other general calculations besides matrix operations; using on-chip memory to perform data interaction between vector cores and tensor cores as much as possible; and increasing the transmission granularity from off-chip memory to on-chip memory as much as possible to make full use of bandwidth.
[0004] Therefore, how to design an efficient storage model within the limited area on the AI chip to improve the performance of the AI model is a problem that needs to be solved currently. Summary of the invention
[0005] According to one aspect of the present disclosure, a computing device is provided. The computing device includes: a tensor core, the tensor core is used to perform matrix operations; an on-chip memory, the on-chip memory includes a group shared memory shared by the multiple vector cores; multiple vector cores, the multiple vector cores are used to read the input data of the matrix operation of the tensor core from the off-chip memory of the computing device based on the data requirements of the tensor core, and each vector core writes the input data to the group shared memory based on the data arrangement requirements of the matrix operation; wherein the tensor core reads the arranged input data from the group shared memory, performs matrix operations based on the arranged input data, and writes the result data of the matrix operation to the group shared memory.
[0006] In some implementations, the multiple vector cores are used to read input data for matrix operations of the tensor cores from an off-chip memory of the computing device based on data requirements of the tensor cores, including: the multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory and load the input data into vector registers of the multiple vector cores via the on-chip memory.
[0007] In some implementations, the multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory, and load the input data to the vector registers of the multiple vector cores via the on-chip memory, including: each vector core reads part of the input data from the off-chip memory, and reads the part of the data to the vector register of the vector core via the L2 cache and L1 cache of the on-chip memory, respectively, wherein the size of the part of the data is 1 / N of the matrix size, and N is the number of the multiple vector cores; the multiple vector cores write their respective part of the data to the group shared memory; and each vector core reads the part of the data it needs from the group shared memory to the vector register of the vector core through coordinate mapping.
[0008] In some implementations, the multiple vector cores are used to read the input data of the matrix operation of the tensor core from the off-chip memory of the computing device based on the data requirements of the tensor core, including: each vector core sends a thread warp read request to the off-chip memory to request to read the portion of the input data corresponding to the vector core; and each vector core reads the portion of the input data corresponding to the vector core from the off-chip memory and loads it to the vector register of the vector core via the on-chip memory.
[0009] In some implementations, each vector core further reads result data of the matrix operation performed by the tensor core from the group of shared memories into a vector register of the vector core, and performs general calculations on the result data.
[0010] According to another aspect of the present disclosure, a data processing method for a computing device is provided. The computing device includes: a tensor core, the tensor core is used to perform matrix operations; an on-chip memory, the on-chip memory includes a group shared memory shared by the multiple vector cores; and multiple vector cores. The method includes: the multiple vector cores read the input data of the matrix operation of the tensor core from the off-chip memory of the computing device based on the data requirements of the tensor core; each vector core writes the input data to the group shared memory based on the data arrangement requirements of the matrix operation; and the tensor core reads the arranged input data from the group shared memory, performs matrix operations based on the arranged input data, and writes the result data of the matrix operation to the group shared memory.
[0011] In some implementations, the multiple vector cores read input data for matrix operations of the tensor cores from an off-chip memory of the computing device based on data requirements of the tensor cores, including: the multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory and load the input data into vector registers of the multiple vector cores via the on-chip memory.
[0012] In some implementations, the multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory, and load the input data to the vector registers of the multiple vector cores via the on-chip memory, including: each vector core reads part of the input data from the off-chip memory, and reads the part of the data to the vector register of the vector core via the L2 cache and L1 cache of the on-chip memory, respectively, wherein the size of the part of the data is 1 / N of the matrix size, and N is the number of the multiple vector cores; the multiple vector cores write their respective part of the data to the group shared memory; and each vector core reads the part of the data it needs from the group shared memory to the vector register of the vector core through coordinate mapping.
[0013] In some implementations, the multiple vector cores read the input data of the matrix operation of the tensor cores from the off-chip memory of the computing device based on the data requirements of the tensor cores, including: each vector core sends a thread warp read request to the off-chip memory to request to read the portion of the input data corresponding to the vector core; and each vector core reads the portion of the input data corresponding to the vector core from the off-chip memory and loads it to the vector register of the vector core via the on-chip memory.
[0014] In some implementations, the method further includes: each vector core also reads result data of the matrix operation performed by the tensor core from the group shared memory into a vector register of the vector core, and performs general calculations on the result data.
[0015] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program code is stored. When the computer program code is executed, the method described above is executed.
[0016] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program performs the method described above when executed by a machine. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The present disclosure will be better understood and other objects, details, features and advantages of the present disclosure will become more apparent through the description of specific embodiments of the present disclosure given with reference to the following drawings.
[0018] Figure 1 A schematic diagram of an exemplary AI model is shown.
[0019] Figure 2 A schematic diagram of the structure of an exemplary computing device is shown.
[0020] Figure 3 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown.
[0021] Figure 4 An exemplary flow chart of a data processing method for a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0023] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "one embodiment" and "some embodiments" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.
[0024] Figure 1 A schematic diagram of an exemplary AI model 100 is shown.
[0025] like Figure 1As shown in , the AI model 100 includes an input layer 110, one or more hidden layers 120, and an output layer 130. Among them, the training process of the AI model 100 involves forward calculation and reverse calculation, and the reasoning process of the AI model 100 involves forward calculation. The goal of the forward calculation is to extract the feature map of the input data and perform higher-dimensional encoding and decoding on it. The core calculation includes matrix operations and activation function operations; the goal of the reverse calculation is to calculate the gradient of the weights to converge the weight matrix. The core calculation includes matrix operations and derivations. In addition, the training process of the AI model 100 also includes loss calculations. The core calculations include mathematical operations such as reduce operations and logical operations, as well as optimizer calculations. The goal is to update the weights, and the core calculations include point-to-point vector operations.
[0026] It can be seen that matrix operations occupy the largest part in the training and reasoning process of the AI model 100. Here, matrix operations generally refer to matrix multiplication operations of two matrices, which can also be called multiplication-addition operations in some cases. In AI chips, matrix operations generally also require the input data to be transformed and processed to meet the requirements of matrix operations. For example, before a hidden layer 120 performs matrix operations on the weight matrix and the input matrix (that is, the feature matrix output by the previous layer), it is also necessary to perform affine transformation and perspective transformation on the feature matrix, as well as linear transformation and nonlinear transformation on the transformed data.
[0027] Figure 2 FIG. 2 shows a schematic diagram of the structure of an exemplary computing device 200. Figure 2 As shown in FIG. , the computing device 200 may include a tensor core 210 and a plurality of vector cores 220 ( Figure 2 4 vector cores 220-1, 220-2, 220-3 and 220-4 are shown in the figure as examples. The tensor core 210 and the vector core 220 are both units for performing calculations, collectively referred to as calculation units or calculation cores. The tensor core 210 is used to perform matrix operations on two input matrices, and the vector core 220 is used to perform other operations other than the matrix operations, such as affine transformation, perspective transformation, linear transformation, nonlinear transformation, etc., which are collectively referred to as general calculations in this article. In this article, Figure 2 The plurality of vector cores 220 shown may constitute a computing group for executing a thread group in parallel, wherein each vector core 220 executes a warp in the thread group, as described in detail below. In some embodiments, the computing device 200 may include a plurality of such computing groups ( Figure 2Only one computing group is exemplarily shown in FIG. 1 ). Herein, the computing device 200 may be, for example, an AI chip including a GPU (Graphic Processing Unit, Graphics Processing Unit), a general purpose GPU (General Purpose GPU, GPGPU), etc.
[0028] In addition, the computing device 200 also includes a unit for storing data, which is called a storage unit. Specifically, the computing device 200 includes an on-chip memory 230 and an off-chip memory 240, wherein the on-chip memory 230 is closer to the computing unit, has a larger transmission bandwidth, and has a faster transmission speed, but generally has a smaller storage capacity, and the off-chip memory 240 generally has a larger storage capacity, but is farther from the computing unit, has a smaller transmission bandwidth, and has a slower transmission speed.
[0029] In some embodiments, the on-chip memory 230 further includes an L1 cache 232 for multiple vector cores 220, a group shared memory (Group Shared Memory, GSM) 234 for data exchange between multiple vector cores 220, and an L2 cache 236 for all tensor cores 210 and vector cores 220. Among them, the L1 cache 232 can be used to cache the intermediate data required and generated by the operation of the vector core 220, and the group shared memory 234 can be used for data sharing and exchange between multiple vector cores 220. The L2 cache 236 can be shared by all tensor cores 210 and vector cores 220 and connected to each tensor core 210 and vector core 220 through a high-speed interconnect (not shown in the figure). Generally, the L1 cache 232 and the group shared memory 234 are closer to each computing unit than the L2 cache 236, have smaller storage space, higher bandwidth, and lower latency. The on-chip memory 230 can usually be implemented using a static random access memory (Static Random-Access Memory, SRAM).
[0030] The off-chip memory 240 may be located on a different chip from each computing unit and the on-chip memory 230, and may be implemented, for example, using a dynamic random access memory (DRAM) or a high bandwidth memory (HBM). The off-chip memory 240 may be the largest memory of the computing device 200, and is used to store at least one of the following data: data required by the computing device 200 to perform calculations, and data generated by the computing device 200 to perform calculations.
[0031] Furthermore, in some embodiments, multiple threads of the warp executed by each vector core 220 may also schedule associated vector registers 222 to provide temporary storage for data required or generated by the corresponding threads of the warp of the vector core 220 .
[0032] When the computing device 200 is used to perform AI model training and reasoning, it is necessary to load the training data or application data from the off-chip memory 240 to each computing unit and perform corresponding operations. Among them, as mentioned above, the tensor core 210 is used to perform matrix operations, and the vector core 220 is used to perform general calculations other than matrix operations. Since the vector core 220 and the tensor core 210 are targeted at different computing scenarios, the task division method within the kernel is also different, and it is necessary to focus on the problem of data reordering and interaction between each vector core 220 and between the vector core 220 and the tensor core 210.
[0033] Currently, the tensor core 210 can read the data required for matrix operations from the off-chip memory 240 via the L2 cache 236 at a larger granularity, such as 512-byte, to perform matrix operations (e.g. Figure 2 As shown in the figure, the arrows 242 and 244 indicate the reading paths of the two input matrices of the matrix operation respectively). In this case, the data needs to be pre-arranged on the off-chip memory 240 according to the hardware rules of the tensor core 210. Here, the hardware rules of the tensor core 210 depend on the implementation method of the tensor core 210 to perform the matrix operation. For example, if the tensor core 210 uses the img2col method to perform the convolution operation conv (which includes multiple consecutive matrix operations), the two four-dimensional input matrices, namely the feature matrix and the weight matrix, need to be aligned and arranged on the off-chip memory 240 in the order of NCHW and OIHW with a granularity of 512Byte. That is, the input matrix needs to be processed in advance by the kernel program on the host side so that it is arranged on the off-chip memory 240 according to the above rules. Here, NCHW represents the number dimension N, the number of channels dimension C, the height dimension H and the width dimension W of the feature map matrix, respectively, and OIHW represents the output dimension O, the input dimension I, the height dimension H and the width dimension W of the weight matrix, respectively.
[0034] In this case, the tensor core 210 further includes two input buffers 212 and 214 to cache data read from the off-chip memory 240 , respectively.
[0035] The matrix operation results of the tensor core 210 are also written to the vector registers 222 (such as Figure 2 ), or written out to the off-chip memory 240 via the L2 cache 236 (as shown by the arrow 246 in FIG. Figure 2 as indicated by arrow 248).
[0036] On the other hand, the vector core 220 reads data from or writes data to the L2 cache 236 at a smaller granularity than the granularity at which the tensor core 210 reads data from the off-chip memory 240, such as 4-byte, 128 bytes, etc. Here, the larger the granularity at which the vector core 220 reads and writes data to the L2 cache 236, the better the performance will be, but it needs to be arranged more strictly according to the hardware rules of the tensor core 210.
[0037] In addition, the computing granularity in the current computing device 220 can be divided into threads, warps, thread groups (TGs) and wide thread groups (WTGs) in order from small to large, where each WTG may include multiple TGs, each TG may include multiple warps, and each warp may include multiple threads. Data exchange between different computing granularities can be performed through various parts of the on-chip memory 230 or the off-chip memory 240. For example, multiple threads in a warp of a vector core 220 can directly use the shuffle instruction to exchange data through their respective vector registers 222, multiple warps in a TG of a computing group composed of multiple vector cores 220 can exchange data through the group shared memory 234, and multiple TGs can exchange data through the off-chip memory 240.
[0038] In addition, the data exchange capability between the vector core 220 and the tensor core 210 is relatively limited. The tensor core 210 can only write unidirectionally to the vector register 222 of the vector core 220, and multiple vector cores 222 can only exchange data through the group shared memory 234 (in the case of a small amount of data) or the off-chip memory 240 (in the case of a large amount of data).
[0039] It can be seen that the current design of the computing device 200 for the AI model mainly uses the tensor core 210 as the producer of data and the vector core 220 as the consumer of data. The memory access process of the tensor core 210, the vector core 220, the on-chip memory 230 and the off-chip memory 240 is as follows: the tensor core 210 reads data from the off-chip memory 240, writes the generated data to the vector register 222 of the vector core 220 after performing matrix operations, and the vector core 220 performs general operations on the data (and possible data read from the off-chip memory 240) and then writes it to the off-chip memory 240 through the on-chip memory 230 (for example, via the L1 cache 232 and the L2 cache 236 in sequence).
[0040] However, this design only focuses on local issues of the AI model, such as the fusion calculation of matrix operations and activation function operations, so that the data of the tensor core 210 can only be loaded from the off-chip memory 240, which is not flexible enough. In addition, the data must be arranged in advance in the off-chip memory 240 according to the hardware rules of the tensor core 210, so that the task division method must also be processed according to the specified hardware rules, otherwise the input data needs to be rearranged and written into the off-chip memory 240. This specified arrangement method is not friendly to software programming and is difficult to cope with a variety of operator requirements. In addition, in this design, the tensor core 210 and the vector core 220 interact in a one-way manner, that is, the tensor core 210 acts as a producer and the vector core 220 acts as a consumer of the data generated by the tensor core 210. At this time, the data that can be observed in the vector register 222 of each vector core 220 is the data after the tensor core 210 divides the vector core 220. The interaction capability between the vector cores 220 cannot be fully utilized, and the tensor core 210 cannot directly read the data processed by the vector core 220 from the on-chip memory 230, which will also cause a huge loss in operator performance.
[0041] To address at least one of the above problems, in the solution of the present invention, the tensor core 210 and the vector core 220 can access each other through the group shared memory 234. The vector core 220 can also act as a producer to pre-process the data and write it to the group shared memory 234 according to the hardware rules of the tensor core 210, and the tensor core 210 can act as a consumer to read data from the group shared memory 234 and perform matrix operations. Furthermore, the tensor core 210 can also act as a producer to output the result of the matrix operation to the group shared memory 234, and the vector core 220 can act as a consumer to post-process and rearrange the result of the matrix operation and output it to the off-chip memory 240 or to the group shared memory 234.
[0042] Figure 3 A schematic diagram of the structure of a computing device 300 according to an embodiment of the present invention is shown. Figure 3 In, with Figure 2 Like numbers refer to like contents, and the computing device 300 and Figure 2 The difference of the computing device 200 shown is mainly that the group shared memory 334 is used for data exchange between the tensor core 310 and the vector core 320. In addition, in some embodiments, the tensor core 310 may not include the input buffers 212 and 214.
[0043] and Figure 2 Similar to the computing device 200 shown in FIG. , the computing device 300 may include a tensor core 310 and a plurality of vector cores 320 ( Figure 34 vector cores 320-1, 320-2, 320-3 and 320-4 are shown in the figure as examples. The tensor core 310 and the vector core 320 are both units for performing calculations, collectively referred to as calculation units or calculation cores. The tensor core 310 is used to perform matrix operations on two input matrices, and the vector core 320 is used to perform other operations other than the matrix operations, such as affine transformation, perspective transformation, linear transformation, nonlinear transformation, etc., which are collectively referred to as general calculations in this article. In this article, Figure 3 The multiple vector cores 320 shown may constitute a computing group (Computing Group) for executing a thread group (ThreadGroup) in parallel, wherein each vector core 320 executes a warp in the thread group. In some embodiments, the computing device 300 may include multiple such computing groups ( Figure 3 Only one computing group is exemplarily shown in FIG. 1 ). Herein, the computing device 300 may be, for example, an AI chip including a GPU (Graphic Processing Unit), a general purpose GPU (GPGPU), or the like.
[0044] In addition, the computing device 300 also includes a unit for storing data, which is called a storage unit. Specifically, the computing device 300 includes an on-chip memory 330 and an off-chip memory 340, wherein the on-chip memory 330 is closer to the computing unit, has a larger transmission bandwidth, and has a faster transmission speed, but generally has a smaller storage capacity, and the off-chip memory 340 generally has a larger storage capacity, but is farther from the computing unit, has a smaller transmission bandwidth, and has a slower transmission speed.
[0045] In some embodiments, the on-chip memory 330 further includes an L1 cache 332 for multiple vector cores 320, a group shared memory 334 for data exchange between multiple vector cores 320 and between vector cores 320 and tensor cores 310, and an L2 cache 336 for all tensor cores 310 and vector cores 320. Among them, the L1 cache 332 can be used to cache the intermediate data required and generated by the vector core 220 operation, and the group shared memory 334 can be used for data sharing and exchange between multiple vector cores 220 and between vector cores 320 and tensor cores 310. The L2 cache 336 can be shared by all tensor cores 310 and vector cores 320 and connected to each tensor core 310 and vector core 320 through a high-speed interconnect (not shown in the figure). Generally, the L1 cache 332 and the group shared memory 334 are closer to each computing unit than the L2 cache 336 and have higher bandwidth and lower latency. The on-chip memory 330 can usually be implemented using SRAM.
[0046] The off-chip memory 340 may be located on a different chip from each computing unit and the on-chip memory 330, and may be implemented using DRAM or HBM, for example. The off-chip memory 340 may be the largest memory of the computing device 300, and is used to store at least one of the following data: data required by the computing device 300 to perform calculations, and data generated by the computing device 300 to perform calculations.
[0047] In addition, multiple threads of the warp executed by each vector core 320 may also schedule associated vector registers 322 to provide temporary storage for data required or generated by the corresponding threads of the warp of the vector core 320 .
[0048] and Figure 2 Unlike the computing device 200 shown in , the group shared memory 334 of the computing device 300 may be designed to have a larger storage capacity than the group shared memory 234 of the computing device 200, so as to implement data interaction between the tensor core 310 and the vector core 320. Since the two input buffers 212 and 214 may not be provided in the tensor core 310, the area occupied by the tensor core 310 becomes smaller, so it is possible to keep the total area of the computing device 300 unchanged or even smaller.
[0049] like Figure 3 As shown in , the vector core 320 can read data from the off-chip memory 340 or write data to the off-chip memory 340.
[0050] Specifically, when a vector core 320 (e.g., vector core 320 - 1 ) reads data from an off-chip memory 340 , according to a read request of the vector core 320 - 1 , the data may reach the vector register 322 of the vector core 320 - 1 from the off-chip memory 340 via the on-chip memory 330 (via the L2 cache 336 and the L1 cache 332 in sequence).
[0051] In some embodiments, the read request of the vector core 320 - 1 may be a warp read request. In this case, the warp scheduled by the vector core 320 - 1 may directly read the data required by the warp from the off-chip memory 340 and write it into the vector register 322 .
[0052] In other embodiments, the multiple vector cores 320 can obtain a larger transmission bandwidth by merging the continuous read requests of all threads in the scheduled thread warp. In this case, the read request of the vector core 320 is a thread continuous read request, and the read data may be the data required by the vector core 320-1, or may be the data required by other vector cores 320 in the multiple vector cores 320. In this case, the vector core 320-1 can write the data read into its vector register 322 into the group shared memory 334, and the group shared memory 334 reorders the data read by the multiple vector cores 320, and then the vector core 320-1 reads the data required by the vector core 320-1 from the group shared memory 334. For example, the vector core 320-1 can read the data required by the vector core 320-1 from the address corresponding to the vector core 320-1 in the group shared memory 334.
[0053] The process of the vector core 320 writing data to the off-chip memory 340 is similar. Specifically, when a vector core 320 (e.g., the vector core 320-2) writes data to the off-chip memory 340, according to the write request of the vector core 320-2, the data can reach the off-chip memory 340 from the vector register 322 of the vector core 320-2 via the on-chip memory 330 (via the L1 cache 332 and the L2 cache 336 in sequence).
[0054] In some embodiments, the write request of the vector core 320 - 2 may be a warp write request. In this case, the warp scheduled by the vector core 320 - 2 may directly write data from its vector register 322 to the off-chip memory 340 .
[0055] In other embodiments, the vector core 320-2 can obtain a larger transmission bandwidth by merging the continuous write requests of all threads in the scheduled thread warp. In this case, the data in the vector register 322 of the vector core 320-2 may not be the data to be written by itself. In this case, the vector core 320-2 can write the data in its vector register 322 to the group shared memory 334, and the group shared memory 334 reorders the data written by multiple vector cores 320, and then the vector core 320-2 reads the data of the vector core 320-2 from the group shared memory 334. For example, the vector core 320-2 can read the data of the vector core 320-2 from the address corresponding to the vector core 320-2 in the group shared memory 334, and write the data to the off-chip memory 340.
[0056] For the tensor core 310, it can directly read the input data for performing matrix operations from the group shared memory 334 (such as Figure 4), performing matrix operations on the input data, and storing the result data of the matrix operations into the group shared memory 334 (as shown by arrows 312 and 314 in FIG. 3 ). Figure 4 Here, the input data for matrix operations stored in the group shared memory 334 may be the input data arranged by the group shared memory 334 based on the data arrangement requirements for performing matrix operations by the tensor core 310 (i.e., the hardware rules of the tensor core 310). Therefore, in this case, the tensor core 310 may not need to store the matrix operations as shown in FIG. Figure 2 Input buffers 212 and 214 are shown in FIG.
[0057] In this way, the tensor core 310 and the vector core 320 can directly exchange data through the on-chip memory, avoiding the disadvantages of the tensor core 310 reading data from the off-chip memory 340, such as having to arrange data on the off-chip memory 340 according to the specified rules of the tensor core 310. In addition, the vector core 320 can act as a data producer and the tensor core 310 can act as a data consumer, which can be more flexibly applied to the reverse calculation, loss calculation, optimizer solution, pruning quantization and sparsification operations of the AI model.
[0058] Figure 4 An exemplary flow chart of a data processing method 400 for a computing device 300 according to an embodiment of the present invention is shown. Here, the data processing method 400 may be a complete forward operation process or a part thereof of a training or reasoning process of an AI model executed in the computing device 300.
[0059] like Figure 4 As shown in FIG, in block 410, the plurality of vector cores 320 may read the input data required for the tensor core 310 to perform the matrix operation from the off-chip memory 340 of the computing device 300 based on the data requirements of the tensor core 310. Here, the data requirements of the tensor core 310 indicate the arrangement and size requirements of each dimension of the two input matrices of a matrix operation of the tensor core 310.
[0060] In some embodiments, the multiple vector cores 320 may send thread continuous read requests to the off-chip memory 340 by merging memory accesses. For example, the thread continuous read request requests the off-chip memory 340 to read input data Dt of a matrix size.
[0061] In this case, the multiple vector cores 320 may read the input data Dt from the off-chip memory 340, and load the input data Dt to the vector registers 322 of the multiple vector cores 320 via the on-chip memory 330. More specifically, each vector core 320 may read part of the input data Dt from the off-chip memory 340 (for example, at a granularity of 128B or 512B, etc.), and read the part of the data to the vector registers 322 of each vector core 320 via the L2 cache 336 and the L1 cache 332, respectively, wherein the size of the part of the data is 1 / N of the matrix size, and N is the number of vector cores 320 (i.e., the number of vector cores 320 included in one computing group).
[0062] As mentioned above, at this time, the data in the vector register 322 of the vector core 320 may not be the data required by the vector core 320. In this case, each vector core 320 can write part of the data in its vector register 322 to the group shared memory 334, and each vector core 320 reads part of the data it needs from the group shared memory 334 to the vector register 322 of the vector core 320 through coordinate mapping.
[0063] In some other embodiments, each vector core 320 may send a warp read request to the off-chip memory 340 , respectively. The warp read request requests the off-chip memory 340 to read a portion of the input data Dt corresponding to the vector core 320 .
[0064] In this case, each vector core 320 can directly read the partial data from the off-chip memory 340 and load it into the vector register 322 of the vector core 320 via the on-chip memory 330. In this way, the partial data read by multiple vector cores 320 together constitute the input data for the tensor core 310 to perform matrix operations.
[0065] Next, at block 420 , each vector core 320 may write the input data read at block 410 to the group shared memory 334 based on the data arrangement requirement (ie, the hardware rule described above) of the matrix operation of the tensor core 310 .
[0066] In block 430 , the tensor core 310 may read the arranged input data from the group shared memory 334 , perform a matrix operation based on the arranged input data, and write result data of the matrix operation to the group shared memory 334 .
[0067] In this way, the tensor core 310 can directly read the input data it needs from the on-chip memory 330 without reading the data from the off-chip memory 340, and the tensor core 310 can also write the generated result data to the on-chip memory 330 again for the vector core 320 to read and use, so that the on-chip memory 330 can be used to realize data interaction between the tensor core 310 and the vector core 320.
[0068] Further, the method 400 may also include (not shown in the figure): after block 430, the vector core 320 may read the result data of the matrix operation performed by the tensor core 310 from the group shared memory 334 into the vector register 322 of the vector core 320, and perform general calculations and coordinate transformations on the result data. For example, for the last hidden layer 120, the vector core 320 may perform an activation function operation on the result data (such as a feature matrix) of the matrix operation of the tensor core 310 at the output layer 130 using the activation function Softmax to map the result data to the interval [0, 1].
[0069] Since the training and reasoning process of the AI model involves the alternating execution of matrix operations and general operations, the above process can be repeated between the tensor core 310 and the vector core 320 through the group shared memory 334 until the final operation result is generated.
[0070] Finally, the vector core 320 may rearrange the final operation result and output it to the off-chip memory 340 .
[0071] By using the solution of the present invention, by increasing the data interaction using the on-chip memory 330 between the tensor core 310 and the vector core 320, the tensor core 310 and the vector core 320 can be producers and consumers of each other, so that the forward, reverse, optimizer, sparse calculation and other issues of the AI model can be more flexibly adapted. In addition, by arranging the data on the on-chip memory 330 (more specifically, the group shared memory 334), it is no longer necessary to arrange the data on the off-chip memory 340, so that it is more friendly to the programming model. The software can realize any data arrangement suitable for the tensor core 310 through the vector core 320, and interact with the data through the group shared memory 334, thereby greatly improving the memory access performance. In addition, the present invention also improves the on-chip fusion capability of the operator, and uses the flexibility of the vector core 320 to divide the large piece of data on the off-chip memory 340 into the local data required by the tensor core 310, and the output data of the tensor core 310 can be coordinate mapped and merged and written out.
[0072] Those skilled in the art will appreciate that the computing device 300 shown in the figure is merely illustrative and may include more or fewer components.
[0073] The computing device 300 and the data processing method for the computing device 300 according to the present disclosure are described above in conjunction with the accompanying drawings. However, those skilled in the art can understand that the computing device 300 does not necessarily include all the components shown in the drawings, and may only include some or more components necessary to perform the functions described in the present disclosure, and the connection method of these components is not limited to the form shown in the drawings, and the method 400 may also include more steps not shown in the drawings.
[0074] The present invention may be implemented as a method, a computing device, a control device of the computing device, a computer-readable storage medium and / or a computer program product. A computer-readable storage medium stores a computer program code, which is used to execute the method of the present disclosure when it is executed. The computer program product includes a computer program, which executes the method of the present disclosure when it is executed. The computing device and / or the computing device may include at least one processor and at least one memory coupled to the at least one processor, and the memory may store instructions for execution by the at least one processor. When the instruction is executed by the at least one processor, the computing device and / or the control device may execute the above method.
[0075] In one or more exemplary designs, the functions described in the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes on a computer-readable medium.
[0076] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present disclosure may be implemented as electronic hardware, computer software, or a combination of the two.
[0077] The above description of the present disclosure is intended to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications of the present disclosure are obvious to those of ordinary skill in the art, and the general principles defined herein may also be applied to other variations without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.
Claims
1. A computing device, characterized in that include: A tensor core, which is used to perform matrix operations; an on-chip memory, the on-chip memory comprising a group shared memory shared by a plurality of vector cores; The plurality of vector cores are configured to read input data of a matrix operation of the tensor core from an off-chip memory of the computing device based on a data requirement of the tensor core, and each vector core writes the input data to the group shared memory based on a data arrangement requirement of the matrix operation, wherein the data requirement of the tensor core indicates an arrangement and size requirement of each dimension of two input matrices of the matrix operation of the tensor core; The tensor core reads the arranged input data from the group shared memory, performs matrix operations based on the arranged input data, and writes result data of the matrix operations into the group shared memory.
2. The computing device of claim 1, wherein: The plurality of vector cores are used to read input data of matrix operations of the tensor cores from an off-chip memory of the computing device based on data requirements of the tensor cores, including: The multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses, so as to read the input data from the off-chip memory and load the input data into the vector registers of the multiple vector cores via the on-chip memory.
3. The computing device of claim 2, wherein: The multiple vector cores send thread continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory, and load the input data to the vector registers of the multiple vector cores via the on-chip memory, including: Each vector core reads part of the input data from the off-chip memory, and reads the part of the data into a vector register of the vector core via an L2 cache and an L1 cache of the on-chip memory, respectively, wherein the size of the part of the data is 1 / N of the matrix size, and N is the number of the plurality of vector cores; The multiple vector cores respectively write their respective parts of data into the group shared memory; and Each vector core reads part of the data it needs from the group shared memory to the vector register of the vector core through coordinate mapping.
4. The computing device of claim 1, wherein: The plurality of vector cores are used to read input data of matrix operations of the tensor cores from an off-chip memory of the computing device based on data requirements of the tensor cores, including: Each vector core sends a warp read request to the off-chip memory to request to read a portion of the input data corresponding to the vector core; and Each vector core reads a portion of the input data corresponding to the vector core from the off-chip memory, and loads the portion of the input data to a vector register of the vector core via the on-chip memory.
5. The computing device of claim 1, wherein: Each vector core also reads the result data of the matrix operation performed by the tensor core from the group shared memory into the vector register of the vector core, and performs general calculations on the result data.
6. A data processing method for a computing device, the computing device comprising: A tensor core, which is used to perform matrix operations; an on-chip memory, the on-chip memory comprising a group shared memory shared by a plurality of vector cores; and the plurality of vector cores, characterized in that the method comprises: The plurality of vector cores read input data of the matrix operation of the tensor core from the off-chip memory of the computing device based on the data requirements of the tensor core, wherein the data requirements of the tensor core indicate the arrangement and size requirements of each dimension of two input matrices of the matrix operation of the tensor core; Each vector core writes the input data to the group shared memory based on the data arrangement requirement of the matrix operation; and The tensor core reads the arranged input data from the group shared memory, performs a matrix operation based on the arranged input data, and writes result data of the matrix operation to the group shared memory.
7. The method according to claim 6, characterized in that The plurality of vector cores reading input data of matrix operations of the tensor cores from an off-chip memory of the computing device based on the data requirements of the tensor cores includes: The multiple vector cores send thread-continuous read requests to the off-chip memory by merging memory accesses, so as to read the input data from the off-chip memory and load the input data into the vector registers of the multiple vector cores via the on-chip memory.
8. The method according to claim 7, characterized in that The multiple vector cores send thread continuous read requests to the off-chip memory by merging memory accesses to read the input data from the off-chip memory, and load the input data to the vector registers of the multiple vector cores via the on-chip memory, including: Each vector core reads part of the input data from the off-chip memory, and reads the part of the data into a vector register of the vector core via an L2 cache and an L1 cache of the on-chip memory, respectively, wherein the size of the part of the data is 1 / N of the matrix size, and N is the number of the plurality of vector cores; The multiple vector cores respectively write their respective parts of data into the group shared memory; and Each vector core reads part of the data it needs from the group shared memory to the vector register of the vector core through coordinate mapping.
9. The method according to claim 6, characterized in that The plurality of vector cores reading input data of matrix operations of the tensor cores from an off-chip memory of the computing device based on the data requirements of the tensor cores includes: Each vector core sends a warp read request to the off-chip memory to request to read a portion of the input data corresponding to the vector core; and Each vector core reads a portion of the input data corresponding to the vector core from the off-chip memory, and loads the portion of the input data to a vector register of the vector core via the on-chip memory.
10. The method according to claim 6, characterized in that The method further comprises: Each vector core also reads the result data of the matrix operation performed by the tensor core from the group shared memory into the vector register of the vector core, and performs general calculations on the result data.
11. A computer-readable storage medium having a computer program code stored thereon, wherein the computer program code executes the method according to any one of claims 6 to 10 when executed.
12. A computer program product, comprising a computer program, which, when executed by a machine, performs the method according to any one of claims 6 to 10.
