AI chip, electronic equipment and convolution operation method

Abstract

The invention relates to an AI chip, electronic equipment and a convolution operation method, and belongs to the field of artificial intelligence. The AI chip comprises N convolution kernels and a storage control system, wherein the storage control system is connected with the N convolution kernels. The storage control system is configured to read input image data from the memory, distribute the input image data to each convolution kernel, read each weight block from the memory, divide each weight block into N parts of weight data and distribute the N parts of weight data to the N convolution kernels, each convolution kernel corresponds to one part of weight data, and each convolution kernel corresponds to one part of weight data; each weight block is a part of a complete weight; each convolution kernel is configured to perform convolution operation on the received weight data and the input image data, convolution operation results of each convolution kernel for the same weight block are added to obtain a convolution operation result of each weight block, and convolution operation of each weight block is added to obtain a final convolution operation result.

Description

An AI chip, electronic device, and convolution operation method Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to an AI chip, electronic device and convolution operation method. Background Art

[0002] In the related art, artificial intelligence (AI) chips, such as neural network processing units (NPUs), are often used to implement general matrix multiplication (GEMM). The structure of the NPU chip is shown in Figure 1. The memory control system (MSS) reads the input image data (such as InImage, which is used to perform convolution operations with weight data), weight data (such as kernel, which is used to perform convolution operations with input image data) and 2ndImage data (used to perform addition (tensor_add) or multiplication (tensor_mult) with the image data output by the convolution operation) from the memory (such as DDR (Double Data Rate) or SRAM (Static Random Access Memory)). Among them, the InImage data and the kernel data are convolved in the convolution kernel, and the convolved data enters the post-processing module (postprocess) for post-processing (including activation, pooling or tensor_add / tensor_mult with 2ndImage data, etc.), and the post-processed data is stored in the memory as the final output image data (such as OutImage).

[0003] In order to limit the hardware area and improve processing efficiency, InImage data, 2ndImage data, and OutImage data are all processed in the form of 3DTile (tile) segmentation. However, there is no concept of 3DTile when reading weight data (kernel) in the hardware. In order to perform correct GEMM operations with InImage Tile, the kernel data needs to be adjusted in order offline before being stored in the memory. In actual applications, the weight matrix P may come from the output data of a certain layer in the neural network. It needs to be rearranged in the external module to the correct GEMM multiplication and addition order before it can be convolved with the InImage data. Transferring the weight matrix P to the external module for matrix data rearrangement takes a long time, especially when processing larger matrices, and the efficiency is very low.

[0004] Summary of the Invention

[0005] In view of this, the purpose of this application is to provide an AI chip, an electronic device and a convolution operation method to improve the problem that the current AI chip needs to transmit the weight matrix P to the outside for matrix data rearrangement when performing general matrix multiplication, which is time-consuming and inefficient, especially when processing larger matrices.

[0006] The embodiment of the present application is implemented as follows:

[0007] In the first aspect, an embodiment of the present application provides an AI chip, comprising: N convolution cores, a storage control system, N convolution cores, where N is an integer greater than or equal to 2; the storage control system is electrically connected to the N convolution cores, and the storage control system is configured to read input image data from the memory and distribute the input image data to each convolution core, as well as read each weight block from the memory, and split each weight block into N portions of weight data and distribute them to the N convolution cores, wherein each convolution core corresponds to a portion of weight data, and each weight block is part of the complete weight; each convolution core is configured to perform a convolution operation on the received weight data with the input image data, wherein the convolution operation results of each convolution core for the same weight block are added to obtain the convolution operation result of each weight block, and the convolution operation of each weight block is added to obtain the final convolution operation result.

[0008] In an embodiment of the present application, the concept of Tile is introduced for weight data. The complete weight is divided into multiple weight blocks, each weight block is read, and each weight block is split into N parts of weight data and distributed to N convolution kernels. This eliminates the need for an external module to perform inefficient data rearrangement on the weight data, thereby reducing the number of cycles of the GEMM operation and improving the efficiency of the convolution operation.

[0009] In combination with a possible implementation manner of the first aspect embodiment, the storage control system is configured to: split each weight block of size A*B into N parts of weight data of size A*Kpc, or split each weight block of size B*A into N parts of weight data of size Kpc*A, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each round of convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each round of convolution.

[0010] In an embodiment of the present application, by splitting each weight block of size A*B into N parts of weight data of size A*Kpc, or splitting each weight block of size B*A into N parts of weight data of size Kpc*A, both the general matrix P*matrix Q and the matrix PT*matrix Q can maximize the processing efficiency, where the matrix PT is the transposed matrix of the matrix P.

[0011] In combination with a possible implementation manner of the embodiment of the first aspect, the storage control system includes: a first image data loading module, a second image data loading module, and a weight data processing module; the first image data loading module is configured to read the input image data from the memory and distribute the input image data to each convolution kernel; the second image data loading module is configured to read each weight block from the memory; the weight data processing module is electrically connected to the second image data loading module and the N convolution kernels, respectively, and the weight data processing module is configured to split each weight block of size A*B into N parts of weight data of size A*Kpc, and distribute them to the N convolution kernels, or, split each weight block of size B*A into N parts of weight data of size Kpc*A, and distribute them to the N convolution kernels.

[0012] In an embodiment of the present application, when performing a GEMM operation, the original weight data request channel is no longer used to request kernel data, but the second image data loading module is used to request the weight matrix P. Since the second image data loading module reads data in the form of 3DTile, each time the weight data is read, only a part of the complete weight, that is, a weight block, is read. This eliminates the need for an external module to perform inefficient data rearrangement on the weight data, thereby reducing the number of cycles of the GEMM operation.

[0013] In combination with a possible implementation method of the first aspect, if the weight data processing module is configured to split each weight block of size A*B into N portions of weight data of size A*Kpc; the weight data processing module includes: N registers, corresponding one-to-one to the N convolution kernels, the N registers are all electrically connected to the second image data loading module, and each register is configured to store a portion of weight data of size A*Kpc.

[0014] In an embodiment of the present application, N registers can be used to split each weight block of size A*B into N pieces of weight data of size A*Kpc, which can save costs and reduce the complexity of the AI ​​chip while achieving the purpose of the invention.

[0015] In combination with a possible implementation manner of the first aspect embodiment, the weight data processing module includes: N extractors, electrically connected to the second image data loading module, each of the extractors being configured to extract a piece of weight data of size Kpc*A from a weight block within one clock cycle when the size of the weight block is B*A and Kpc=1.

[0016] In combination with a possible implementation manner of the first aspect embodiment, the weight data processing module includes: N logic shifters, electrically connected to the second image data loading module, each of the shifters is configured to extract a piece of weight data of size Kpc*A from a weight block within A clock cycles when the size of the weight block is B*A and Kpc>1.

[0017] In combination with a possible implementation method of the first aspect, if the weight data processing module is configured to split each weight block of size B*A into N weight data of size Kpc*A; the weight data processing module includes: N extractors, N logical shifters, and N selectors; the N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract a weight data of size Kpc*A from a weight block within one clock cycle; the N logical shifters are electrically connected to the second image data loading module, and each of the shifters is configured to extract a weight data of size Kpc*A from a weight block within A clock cycles; the N selectors correspond one-to-one to the N convolution kernels, and each of the selectors is electrically connected to one extractor, one logical shifter and one convolution kernel, and each of the selectors is configured to send the output of the extractor to the corresponding convolution kernel when Kpc=1; and send the output of the logical shifter to the corresponding convolution kernel when Kpc>1.

[0018] In an embodiment of the present application, when Kpc=1, N extractors are used to split each weight block of size B*A into N portions of weight data of size Kpc*A; when Kpc>1, N logical shifters are used to split each weight block of size B*A into N portions of weight data of size Kpc*A; at the same time, N selectors are used to select the correct data to output to the corresponding convolution kernel. While achieving the purpose of the invention, it can save costs and reduce the complexity of the AI ​​chip, and make the AI ​​chip compatible with multiple Kpc division methods.

[0019] In combination with a possible implementation manner of the embodiment of the first aspect, the weight data processing module includes: N registers, N extractors, N logical shifters, N first selectors, and N second selectors; the N registers correspond one-to-one to the N convolution kernels, and the N registers are electrically connected to the second image data loading module, and each register is configured to store a weight data of size A*Kpc; the N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract a weight data of size Kpc*A from a weight block within one clock cycle; the N logical shifters are electrically connected to the second image data loading module, and each of the shifters is Configured to extract a piece of weight data of size Kpc*A from a weight block within A clock cycles; each of the first selectors is electrically connected to one of the extractors and one of the logic shifters, and each of the first selectors is configured to select the output data output of the corresponding extractor when Kpc=1; and select the output data output of the corresponding logic shifter when Kpc>1; each of the second selectors is electrically connected to one of the first selectors, one of the registers, and one of the convolution kernels, and each of the second selectors is configured to select the output data output of the corresponding register when the size of the weight block is A*B, and select the output data output of the corresponding first selector when the size of the weight block is B*A.

[0020] In an embodiment of the present application, the weight data processing module using the above structure can be compatible with a variety of scenarios. For example, when the size of the weight block is A*B, the second image data loading module will send the weight block to N registers via the path0 path; when the size of the weight block is B*A, and Kpc=1, the weight block will be sent to N extractors; when the size of the weight block is B*A, and Kpc>1, the weight block will be sent to N logical shifters, and then N first selectors will be used to select the correct data to output to the corresponding second selector, and then the second selector will select the correct data to output to the corresponding convolution kernel. While achieving the purpose of the invention, it can save costs and reduce the complexity of the AI ​​chip, and make the AI ​​chip compatible with a variety of scenarios, thereby improving the applicability of the solution.

[0021] In combination with a possible implementation of the embodiment of the first aspect, each convolution kernel is also configured to convert the weight data of size Kpc*A into weight data of size A*Kpc when receiving weight data of size Kpc*A, and then perform a convolution operation with the input image data.

[0022] In an embodiment of the present application, when weight data of size Kpc*A is received, the weight data of size Kpc*A is converted into weight data of size A*Kpc, and then a convolution operation is performed with the input image data, so that the operation of matrix PT*matrix Q can be implemented, so that the AI ​​chip can be applied not only to matrix P*matrix Q, but also to matrix PT*matrix Q. In a second aspect, an embodiment of the present application further provides an electronic device, comprising: a memory and an AI chip provided in the above-mentioned embodiment of the first aspect and / or any possible implementation method in combination with the embodiment of the first aspect, wherein the AI ​​chip is electrically connected to the memory.

[0023] On the third aspect, the embodiment of the present application also provides a convolution operation method, including: reading input image data, and reading each weight block, and splitting each weight block into N parts of weight data, wherein each weight block is part of the complete weight; performing convolution operation on each weight data with the input image data; adding the convolution operation results of each weight data belonging to the same weight block to obtain the convolution operation result of each weight block; adding the convolution operation results of each weight block to obtain the final convolution operation result.

[0024] Other features and advantages of the present application will be described in the following description. The purpose and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. The above and other purposes, features and advantages of the present application will be more clearly shown through the drawings.

[0026] FIG1 is a schematic diagram of the structure of the connection between an AI chip and a memory in the related art.

[0027] FIG2 shows a schematic structural diagram of an AI chip connected to a memory according to an embodiment of the present application.

[0028] FIG3 shows a schematic diagram illustrating the principle of a convolution operation provided in an embodiment of the present application.

[0029] FIG4 shows a schematic diagram of a principle of making a GEMM operation equivalent to the convolution operation shown in FIG3 , provided by an embodiment of the present application.

[0030] FIG5 shows a schematic diagram of the flow of data of a matrix P from a DDR memory to an SRAM and then to a convolution kernel according to an embodiment of the present application.

[0031] FIG6 a shows a schematic diagram of the flow of 2D data of another matrix P provided in an embodiment of the present application from a DDR memory to an SRAM and then to a convolution kernel.

[0032] FIG6 b shows a schematic diagram of the flow of 3D data of another matrix P provided in an embodiment of the present application from the DDR memory to the SRAM and then to the convolution kernel.

[0033] FIG7 shows a schematic structural diagram of a weight data processing module provided in an embodiment of the present application.

[0034] FIG8 shows a schematic structural diagram of another weight data processing module provided in an embodiment of the present application.

[0035] FIG9 shows a flow chart of a convolution operation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following embodiments can be used as examples to more clearly illustrate the technical solutions of the present application, and are not intended to limit the scope of protection of the present application. It will be understood by those skilled in the art that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other.

[0037] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in the subsequent figures. At the same time, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0038] Furthermore, the term "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0039] In the description of the embodiments of the present application, unless otherwise clearly specified or limited, the technical term "electrical connection" may refer to a direct electrical connection or an indirect electrical connection through an intermediate medium.

[0040] In view of the fact that the current AI chip needs to transfer the weight matrix P to the outside for matrix data rearrangement when performing general matrix multiplication, it takes a long time, especially when processing larger matrices, the efficiency is very low. The embodiment of the present application provides an AI chip, an electronic device and a convolution operation method, so that it is no longer necessary to transfer the weight matrix P to the outside for matrix data rearrangement, thereby improving the efficiency of the convolution operation. The AI ​​chip provided by the embodiment of the present application will be described below with reference to Figure 2. The AI ​​chip includes: N convolution kernels and a storage control system, the storage control system is electrically connected to the N convolution kernels, and N is an integer greater than or equal to 2. It can be understood that the AI ​​chip is also electrically connected to the memory. Among them, the memory and the storage control system are different devices, and the functions and uses of the two are different. The memory is a storage device specifically used to store data; and the storage control system is a read-write control system of the memory, which is used to control the reading and writing of data.

[0041] The storage control system is configured to read input image data from a memory (which may be an SRAM memory and / or a DDR memory), and distribute the input image data to each convolution kernel, and read each weight block (weight Tile) from the memory, and split each weight block into N portions of weight data and distribute them to N convolution kernels, wherein each convolution kernel corresponds to a portion of weight data, and each weight block is part of the complete weight. In an embodiment of the present application, the concept of Tile is introduced to the weight data, by dividing the complete weight into multiple weight blocks, and then reading each weight block, and splitting each weight block into N portions of weight data and distributing them to N convolution kernels, so that there is no need for an external module to perform inefficient data rearrangement on the weight data, thereby reducing the number of cycles (cycles) of the GEMM operation.

[0042] It is understandable that the input image data read from the memory by the storage control system may also be a portion of the complete input image data, which is split to fit memories of various sizes.

[0043] For a better understanding, the convolution operation shown in Figure 3 is explained below. As shown in Figure 3, the dimensions of InImage are outX*outY*kZ, and the dimensions of the kernel data are kX*kY*kZ*outZ. Among them, kZ represents the depth in the first direction, outZ represents the depth in the second direction, outX*outY represents the two-dimensional size of the input image data, outX represents the length of the image, outY represents the height of the image, kX*kY represents the size of the two-dimensional weight element group, kX represents the length of the two-dimensional weight element group, and kY represents the height of the two-dimensional weight element group.

[0044] The first three-dimensional (kX*kY*kZ) kernel array is multiplied with each point in the InImage of the same dimension to obtain kX*kY*kZ multiplication results. Adding all kX*kY*kZ multiplication results can obtain the first point of the first plane on the OutImage. The three-dimensional kernel array is slid from left to right and then from top to bottom in the InTile, and the same multiplication and addition operation is performed to obtain the first outTile of the first plane on the OutImage. There are outZ three-dimensional kernel arrays in the fourth dimension. The same operation is performed on each of them to obtain the OutImage results of different planes in the outZ dimension.

[0045] The GEMM operation of matrix P * matrix Q = matrix R is equivalent to the convolution shown in Figure 3. Its equivalent principle diagram is shown in Figure 4. Consider one of the GEMM matrices P (rows are kZ, columns are outZ, in this case, kX = kY = 1) as the kernel array, and the other matrix Q (rows are outX * outY, columns are kZ) as InImage. Then, GEMM can be regarded as the convolution of InImage and the kernel array.

[0046] When splitting weight data (kZ*outZ), first split the data along the kZ direction and then along the outZ direction, and split it into multiple weight blocks of size A*B or B*A, where A and B are both positive integers, A is the maximum height of all convolution kernels in the first direction (such as the kZ direction mentioned above), for example, the maximum value of A is 9, and B is the maximum number of weight element groups processed by all convolution kernels in each round of convolution. The size of each weight element group is kX*kY*A, that is, a weight element group contains kX*kY*A weight elements. When kX=kY=1, a weight element group contains A weight elements.

[0047] In which, the storage control system is configured to: split each weight block of size A*B into N parts of weight data of size A*Kpc, or split each weight block of size B*A into N parts of weight data of size Kpc*A, wherein Kpc is a positive integer, and Kpc is the maximum number of weight element groups processed by each convolution kernel in each round of convolution in the second direction (such as the outZ direction mentioned above).

[0048] When performing GEMM, tile data for matrix P must be first extracted in the kZ direction and then in the outZ direction. Depending on the order of the data, GEMM can be divided into two cases: matrix P * matrix Q and matrix PT * matrix Q, where matrix PT is the transposed matrix of matrix P. In the case of matrix PT * matrix Q, matrix P must be transposed to matrix PT before multiplication with matrix Q. This step can be performed within the convolution kernel. These two cases of matrix P have 2D and 3D forms, respectively.

[0049] When performing the matrix P * matrix Q transformation, the kZ direction of matrix P in the memory is continuous. Two forms of matrix P are shown in Figure 5. Figure 5 shows the data flow from DDR memory to SRAM (cache), and then to the convolution kernel (core). Data can be first stored in the DDR memory, then loaded into the SRAM, and then loaded into the convolution kernel. In this case, the memory in Figure 2 can include both DDR memory and SRAM, with the DDR memory connected to the storage control system via the SRAM.

[0050] The upper left side of Figure 5 is the 2D form of the matrix P stored in the DDR memory: in_x (the length of the matrix P in the x direction) = kZ; Tile_x (the length of the tile in the x direction) = A (the Tile_x of the last tile in the x direction may be smaller than the Tile_x of other tiles); in_y (the length of the matrix P in the y direction) = outZ; Tile_y (the length of the tile in the y direction) = B = Kpc*N (it can be understood that the Tile_y of the last tile in the y direction may be smaller than the Tile_y of other tiles). The arrow indicates the sliding direction of the tile, that is, first from left to right and then from top to bottom, where Figure 5 only shows the sliding direction from left to right. The lower left side of Figure 5 shows the 3D representation of matrix P stored in DDR memory: in_x = kZ; Tile_x = A (the Tile_x of the last tile in the x-direction is smaller than the Tile_x of the other tiles); in_y = outZ / B; Tile_y = 1; in_z (the length of matrix P in the z-direction) = B (the Tile_y of the last tile in the y-direction may be smaller than the Tile_y of the other tiles). Arrows indicate the direction of tile sliding, first from left to right, then from top to bottom. The middle portion of Figure 5 shows the SRAM cache representation of matrix P during processing. Similar to its 2D representation in DDR memory, weight tiles are read from left to right, then from top to bottom. The right side of Figure 5 shows the weight data received by each convolution kernel, with a size of A*Kpc.

[0051] When performing the matrix PT * matrix Q, the outZ direction of matrix P in the memory is continuous. Two forms of matrix P are shown in Figures 6a and 6b. Figure 6a shows the flow of 2D data from DDR memory to SRAM and then to the convolution core, while Figure 6b shows the flow of 3D data from DDR memory to SRAM and then to the convolution core.

[0052] The left part of Figure 6a is the 2D form of the matrix P stored in the DDR memory: in_x = outZ; Tile_x = B (the Tile_x of the last Tile in the x direction may be smaller than the Tile_x of other tiles); in_y = kZ; Tile_y = A (the Tile_y of the last Tile in the y direction may be smaller than the Tile_y of other tiles). The arrows indicate the sliding direction of the tile, which is first from top to bottom and then from left to right. The middle part of Figure 6a is the form of the matrix P cached in the SRAM, which is the same as the 2D form in the DDR memory. The arrows indicate the sliding direction of the tile, which is first read from top to bottom and then from left to right. The right part of Figure 6a is the weight data received by each convolution kernel. The size of the weight data is Kpc*A. When performing the convolution operation, it needs to be transposed to A*Kpc.

[0053] The left side of Figure 6b shows the 3D representation of matrix P stored in DDR memory: in_x = outZ; Tile_x = B (the Tile_x of the last tile in the x-direction may be smaller than the Tile_x of the other tiles); in_y = kZ / A; Tile_y = 1; in_z = B. The arrows indicate the direction of tile sliding, first from top to bottom, then from left to right. The middle section of Figure 6b shows the SRAM cache representation of matrix P during processing, which is identical to the 2D representation of matrix P * matrix Q. The arrows indicate the direction of tile sliding, first from left to right, then from top to bottom. The right side of Figure 6b shows the weight data received by each convolution kernel. The weight data is of size A*Kpc, and no additional transposition is required during convolution operations.

[0054] When reading the weight block, the storage control system will read the weight block in sequence from the DDR memory or SRAM memory according to the principle shown in Figure 5 or Figure 6a, Figure 6b. When reading the weight block, the storage control system gives priority to reading from the SRAM memory. If there is no corresponding weight block in the SRAM memory, it is read from the DDR memory. After reading the weight block from the DDR, it needs to be stored in the SRAM for caching. In an optional embodiment, the SRAM memory can be the memory inside the AI ​​chip, and the DDR memory is the memory outside the AI ​​chip.

[0055] In one embodiment, the storage control system includes: a first image data loading module, a second image data loading module, and a weight data processing module. The first image data loading module is electrically connected to the memory and N convolution cores, respectively; the second image data loading module is electrically connected to the memory and the weight data processing module, respectively; and the weight data processing module is electrically connected to the N convolution cores.

[0056] The first image data loading module is configured to read the input image data from the memory and distribute the input image data to each convolution kernel. The second image data loading module is configured to read each weight block from the memory and send each read weight block to the weight data processing module. The weight data processing module is electrically connected to the second image data loading module and the N convolution kernels respectively, and the weight data processing module is configured to split each weight block into N parts of weight data and distribute them to the N convolution kernels.

[0057] In an embodiment of the present application, when performing a GEMM operation, the original weight data request channel is no longer used to request kernel data, but the second image data loading module is used to request the weight matrix P. Since the second image data loading module reads data in the form of 3DTile, each time the weight data is read, only a part of the complete weight, that is, a weight block, is read. This eliminates the need for an external module to perform inefficient data rearrangement on the weight data, thereby reducing the number of cycles of the GEMM operation.

[0058] Specifically, the weight data processing module is configured to split each weight block of size A*B into N parts of weight data of size A*Kpc and distribute them to N convolution kernels, or to split each weight block of size B*A into N parts of weight data of size Kpc*A and distribute them to N convolution kernels.

[0059] In one embodiment, if the weight data processing module is configured to split each weight block of size A*B into N weight data of size A*Kpc; the weight data processing module includes: N registers. The N registers correspond one-to-one to the N convolution kernels, and the N registers are electrically connected to the second image data loading module, and each register is configured to store a weight data of size A*Kpc. Among them, the above-mentioned register can be a variable length input and fixed length output register (Variable Length Writer, VLM), which can receive weight data of non-fixed effective length, and the effective data is transmitted to the corresponding convolution kernel after accumulating enough output width.

[0060] To ensure performance, the maximum input width of the VLM is min(sram_data_width A*max_kpc*bpp), and the output width is max_kpc*bpp. The sram_data_width parameter indicates the width of the SRAM, and bpp indicates the number of bytes per weight element. After reading the weight block from DDR, it needs to be cached in SRAM.

[0061] In another embodiment, the weight data processing module is configured to split each weight block of size B*A into N pieces of weight data of size Kpc*A; the weight data processing module includes: N extractors, N logical shifters and N selectors, as shown in Figure 7. Each extractor and each logical shifter is connected to the second image data loading module, and one selector connects one extractor and one logical shifter. The N extractors are all electrically connected to the second image data loading module, and when Kpc=1, the second image data loading module sends the weight block data to the extractor. Each extractor is configured to extract a piece of weight data of size Kpc*A from a weight block within one clock cycle. In this embodiment, the second image loading module can send A pieces of B weight data to the extractor within one clock cycle.

[0062] When Kpc=1, adjacent weight elements belong to different convolution kernels, and the effective data transmitted in each cycle includes A B arrays (row arrays). Assuming Kpc=1, the number of convolution kernels N=24, then B=Kpc*N=1*24, assuming A=9, each weight element occupies one Byte, then the data width transmitted in each cycle is 216Byte, then Byte0, Byte24, Byte48, Byte72, Byte96, Byte120, Byte144, Byte168, Byte196 belong to core1; Byte1, Byte25, Byte49, Byte73, Byte97, Byte121, Byte145, Byte169, Byte197 belong to core2, and so on. Under this implementation method, one clock cycle allows the transmission of A B arrays.

[0063] N logical shifters are all electrically connected to the second image data loading module. When Kpc is greater than 1, the second image data loading module sends the weight block data to the logical shifter. Each shifter is configured to extract a piece of weight data of size Kpc*A from a weight block within A clock cycles. Under this embodiment, the second image loading module only sends 1 B of weight data to the logical shifter within one clock cycle, and a B*A weight block requires A clock cycles to be sent.

[0064] When Kpc is greater than 1, only one B array can be transmitted per cycle. Assuming Kpc = 2, the number of convolution kernels N = 24, and each weight element occupies one byte, then Kpc * N = 2 * 24 = 48. Therefore, the data width transmitted per cycle is 48 bytes. Byte 0 and Byte 1 belong to core 1; Byte 2 and Byte 3 belong to core 2, and so on. Byte 46 and Byte 47 belong to core 24. Logical shifter 1 is responsible for outputting Byte 0 and Byte 1; logical shifter 2 is responsible for outputting Byte 2 and Byte 3; and so on. Logical shifter 24 is responsible for outputting Byte 46 and Byte 47.

[0065] The N selectors correspond one-to-one to the N convolution cores. Each selector is electrically connected to an extractor, a logic shifter and a convolution core. Each selector is configured to send the output of the extractor to the corresponding convolution core when Kpc=1; and to send the output of the logic shifter to the corresponding convolution core when Kpc>1.

[0066] It can be understood that when the size of the weight block is B*A and Kpc=1, the weight data processing module can include only N extractors (excluding N logical shifters and N selectors). At this time, the N extractors are directly connected to the N convolution kernels. Similarly, when the size of the weight block is B*A and Kpc>1, the weight data processing module can include only N logical shifters (excluding N extractors and N selectors). At this time, the N logical shifters are directly connected to the N convolution kernels. The structure shown in Figure 7 is the structure of a weight data processing module compatible with multiple Kpc.

[0067] It is understandable that when the weight block size is B*A and Kpc>1, an extractor can also be used to split each weight block of size B*A into N weight data of size Kpc*A. At this time, due to the different sizes of Kpc configured in different cases, and the uneven distribution of Kpc for each convolution kernel at the edge tiles of matrix P, the design of splitting and reassembling introduces many variables. Therefore, in order to reduce the area overhead caused by these variables, when Kpc c>1, a logical shifter is used to implement it.

[0068] In another embodiment, as shown in Figure 8, the weight data processing module includes: N registers, N extractors, N logical shifters, N first selectors and N second selectors. It can be understood that when the second image data loading module sends the weight block, it will not send it to at least two of the extractors, logical shifters, and registers at the same time. When the size of the weight block is A*B, the second image data loading module will send the weight block to N registers via path0; when the size of the weight block is B*A and Kpc=1, the weight block will be sent to N extractors; when the size of the weight block is B*A and Kpc>1, the weight block will be sent to N logical shifters.

[0069] The N registers correspond one-to-one to the N convolution kernels, and the N registers are all electrically connected to the second image data loading module. Each register is configured to store a piece of weight data of size A*Kpc.

[0070] The N extractors are all electrically connected to the second image data loading module, and each extractor is configured to extract a piece of weight data with a size of Kpc*A from a weight block within one clock cycle.

[0071] The N logic shifters are all electrically connected to the second image data loading module, and each shifter is configured to extract a piece of weight data with a size of Kpc*A from a weight block within A clock cycles.

[0072] Each first selector is electrically connected to an extractor and a logic shifter. Each first selector is configured to select the output data of the corresponding extractor when Kpc=1; and to select the output data of the corresponding logic shifter when Kpc>1.

[0073] Each second selector is electrically connected to a first selector, a register, and a convolution kernel. Each second selector is configured to select the output data output of the corresponding register when the size of the weight block is A*B (or the size of the weight data is A*Kpc), and to select the output data output of the corresponding first selector when the size of the weight block is B*A (or the size of the weight data is Kpc*A).

[0074] Each convolution kernel is configured to perform a convolution operation on the received weight data with the input image data. Specifically, when receiving weight data of a size of A*Kpc, each convolution kernel is configured to perform a convolution operation on the received weight data with the input image data; when receiving weight data of a size of Kpc*A, the weight data of a size of Kpc*A is converted into weight data of a size of A*Kpc, and then convolution operation is performed with the input image data.

[0075] Among them, the convolution operation results of each convolution kernel for the same weight block are added according to the corresponding position to obtain the convolution operation result of each weight block, and the convolution operation of each weight block is added according to the corresponding position to obtain the final convolution operation result.

[0076] In some possible implementations, the AI ​​chip may further include a post-processing module, which is respectively connected to each convolution kernel and the memory. The post-processing module is used to add the convolution operation results of each convolution kernel for the same weight block according to the corresponding position to obtain the convolution operation result of each weight block, and add the convolution operation of each weight block according to the corresponding position to obtain the final convolution operation result.

[0077] Among them, the above-mentioned AI chip can be an integrated circuit chip with signal processing capabilities. The above-mentioned AI chip can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a microprocessor, etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. Or the AI ​​chip can also be any conventional processor, etc.

[0078] In addition, the AI ​​chip can also be a dedicated computing acceleration chip (or accelerator) designed to undertake heavy computing tasks, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processor, etc. Of course, it can also be other processors for AI computing tasks.

[0079] An embodiment of the present application further provides an electronic device comprising: a memory and the aforementioned AI chip, wherein the AI ​​chip is electrically connected to the memory. The memory is used to store data required for performing GEMM operations, such as input image data and weight data.

[0080] The electronic devices include but are not limited to tablets, notebooks, vehicle-mounted devices, and servers.

[0081] The present application also provides a convolution operation method that can be applied to the above-mentioned AI chip. The convolution operation method provided by the present application embodiment will be described below with reference to FIG9.

[0082] S1: Read the input image data, read each weight block, and split each weight block into N parts of weight data.

[0083] The first image data loading module may be used to read the input image data from the memory, and the second image data loading module may be used to read each weight block from the memory. The weight data processing module may be used to split each weight block into N portions of weight data. For example, the second image data loading module sends each read weight block to the weight data processing module, and the weight data processing module splits each weight block into N portions of weight data. Each weight block is part of the complete weight.

[0084] After reading the input image data, the first image data loading module distributes the input image data to N convolution kernels, where N is an integer greater than or equal to 2.

[0085] After splitting each weight block into N pieces of weight data, the weight data processing module will distribute the N pieces of weight data to N convolution kernels, with each convolution kernel corresponding to one piece of weight data.

[0086] When the weight data processing module splits each weight block into N parts of weight data, it can be to split each weight block of size A*B into N parts of weight data of size A*Kpc, or to split each weight block of size B*A into N parts of weight data of size Kpc*A, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each round of convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each round of convolution.

[0087] S2: Perform convolution operation on each weight data with the input image data.

[0088] The convolution kernel in the AI ​​chip may perform a convolution operation on each weight data received with the input image data.

[0089] S3: Add the convolution operation results of each weight data belonging to the same weight block to obtain the convolution operation result of each weight block.

[0090] The post-processing module may add the convolution operation results of each weight data belonging to the same weight block to obtain the convolution operation result of each weight block.

[0091] S4: Add the convolution operation results of each weight block to obtain the final convolution operation result.

[0092] The post-processing module may add the convolution operation results of each weight block to obtain the final convolution operation result.

[0093] The convolution operation method provided in the embodiment of the present application has the same implementation principle and technical effects as those in the aforementioned AI chip embodiment. For the sake of brief description, for matters not mentioned in the method embodiment, reference can be made to the corresponding content in the aforementioned AI chip embodiment.

[0094] An embodiment of the present application further provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium), on which a computer program is stored. When the computer program is run by a computer such as the above-mentioned electronic device, the convolution operation method shown above is executed.

[0095] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0097] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0098] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a laptop, a server, or an electronic device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An AI chip, characterized in that: include: N convolution kernels, where N is an integer greater than or equal to 2; A storage control system electrically connected to the N convolution cores, the storage control system being configured to read input image data from a memory and distribute the input image data to each convolution core, and to read each weight block from a memory and split each weight block into N portions of weight data and distribute them to the N convolution cores, wherein each convolution core corresponds to a portion of weight data and each weight block is a portion of a complete weight; Each convolution kernel is configured to perform a convolution operation on the received weight data and the input image data, wherein the convolution operation results of each convolution kernel for the same weight block are added to obtain the convolution operation result of each weight block, and the convolution operations of each weight block are added to obtain the final convolution operation result.

2. The AI ​​chip according to claim 1, characterized in that: The storage control system is configured to: split each weight block of size A*B into N parts of weight data of size A*Kpc, or split each weight block of size B*A into N parts of weight data of size Kpc*A, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each round of convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each round of convolution.

3. The AI ​​chip according to claim 2, characterized in that: The storage control system comprises: A first image data loading module is configured to read the input image data from a memory and distribute the input image data to each convolution kernel; a second image data loading module, configured to read each weight block from the memory; The weight data processing module is electrically connected to the second image data loading module and the N convolution cores, respectively. The weight data processing module is configured to split each weight block of size A*B into N weight data of size A*Kpc and distribute them to the N convolution cores, or to split each weight block of size B*A into N weight data of size Kpc*A and distribute them to the N convolution cores.

4. The AI ​​chip according to claim 3, characterized in that: If the weight data processing module is configured to split each weight block of size A*B into N weight data of size A*Kpc; the weight data processing module includes: N registers correspond one-to-one to the N convolution kernels, and the N registers are all electrically connected to the second image data loading module, and each register is configured to store a piece of weight data with a size of A*Kpc.

5. The AI ​​chip according to claim 3, characterized in that: The weight data processing module includes: N extractors, electrically connected to the second image data loading module, each of the extractors being configured to extract a piece of weight data of size Kpc*A from a weight block within one clock cycle when the size of the weight block is B*A and Kpc=1.

6. The AI ​​chip according to claim 3, characterized in that: The weight data processing module includes: N logic shifters, N logic shifters are electrically connected to the second image data loading module, and each of the shifters is configured to extract a piece of weight data with a size of Kpc*A from a weight block within A clock cycles when the size of the weight block is B*A and Kpc>1.

7. The AI ​​chip according to claim 3, characterized in that: If the weight data processing module is configured to split each weight block of size B*A into N pieces of weight data of size Kpc*A; The weight data processing module includes: N extractors, electrically connected to the second image data loading module, each of the extractors being configured to extract a piece of weight data of size Kpc*A from a weight block within one clock cycle; N logic shifters, electrically connected to the second image data loading module, each of the shifters being configured to extract a piece of weight data of size Kpc*A from a weight block within A clock cycles; N selectors correspond one-to-one to the N convolution cores, each of the selectors is electrically connected to an extractor, a logic shifter and a convolution core, and each of the selectors is configured to send the output of the extractor to the corresponding convolution core when Kpc=1; and to send the output of the logic shifter to the corresponding convolution core when Kpc>1.

8. The AI ​​chip according to claim 3, characterized in that: The weight data processing module includes: N registers, corresponding one to one with the N convolution kernels, the N registers are all electrically connected to the second image data loading module, and each register is configured to store a piece of weight data with a size of A*Kpc; N extractors, electrically connected to the second image data loading module, each of the extractors being configured to extract a piece of weight data with a size of Kpc*A from a weight block within one clock cycle; N logic shifters, electrically connected to the second image data loading module, each of the shifters being configured to extract a piece of weight data of size Kpc*A from a weight block within A clock cycles; N first selectors, each of which is electrically connected to one of the extractors and one of the logic shifters, and each of which is configured to select the output data of the corresponding extractor when Kpc=1; and select the output data of the corresponding logic shifter when Kpc>1; N second selectors, each of the second selectors is electrically connected to one of the first selectors, one of the registers, and one of the convolution cores, and each of the second selectors is configured to select the output data output of the corresponding register when the size of the weight block is A*B, and to select the output data output of the corresponding first selector when the size of the weight block is B*A.

9. The AI ​​chip according to any one of claims 1 to 8, characterized in that: Each convolution kernel is also configured to convert the weight data of size Kpc*A into weight data of size A*Kpc when receiving weight data of size Kpc*A, and then perform a convolution operation with the input image data.

10. An electronic device, characterized in that: include: A memory and an AI chip as described in any one of claims 1 to 9, wherein the AI ​​chip is electrically connected to the memory.

11. A convolution operation method, characterized in that: include: Read the input image data, read each weight block, and split each weight block into N weight data, where each weight block is part of the complete weight; Perform convolution operation on each weight data with the input image data; Add the convolution operation results of each weight data belonging to the same weight block to obtain the convolution operation result of each weight block; Add the convolution operation results of each weight block to get the final convolution operation result.

Images