AI chips, electronic devices and convolutional calculation methods

Abstract

This application relates to an AI chip, an electronic device, and a convolution calculation method, and is in the field of artificial intelligence. The AI ​​chip includes N convolution kernels and a memory subsystem electrically connected to the N convolution kernels. The memory subsystem is configured to read input image data from the memory and distribute it to each convolution kernel, read weight blocks from the memory, and divide each weight block into N weight data and distribute them to the N convolution kernels. Each convolution kernel corresponds to one weight data, and each weight block is a part of the total weights. Each convolution kernel is configured to perform a convolution calculation of the received weight data with the input image data, add the convolution calculation results of each convolution kernel for the same weight block to obtain the convolution calculation result of each weight block, and add the convolution calculation results of each weight block to obtain a final convolution calculation result.

Description

[Technical Field]

[0001] The present application belongs to the field of artificial intelligence, and specifically relates to AI chips, electronic devices and convolution calculation methods. [Background technology]

[0002] In related art, general matrix multiplication (GEMM) is typically implemented by artificial intelligence (AI) chips such as neural-network processing units (NPUs). The structure of an NPU chip, as shown in Figure 1, is such that a memory subsystem (MSS) reads input image data (e.g., represented as InImage and used for convolution with weight data), weight data (e.g., represented as kernel and used for convolution with input image data), and 2ndImage data (used for addition (tensor_add) or multiplication (tensor_mult) with image data output from the convolution operation) from a memory (e.g., double data rate (DDR) or static random access memory (SRAM)). Here, the InImage data and kernel data are convolved with a convolution kernel, and the convolved data is post-processed in the postprocessing module (postprocess) (including activation, pooling, or tensor_add / tensor_mult with 2ndImage data, etc.), and the post-processed data is stored in memory as the final output image data (e.g., represented by OutImage).

[0003] To reduce hardware area and improve processing efficiency, the InImage data, 2ndImage data, and OutImage data are all processed in the form of 3D Tile (tile) slices. However, the hardware does not have a concept of 3D Tile when reading weight data (kernels). Therefore, to perform correct GEMM calculations with InImageTile, the order of the kernel data must be adjusted offline before being stored in memory. In practice, the weight matrix P may originate from the output data of a layer in a neural network. Before convolving it with InImage data, it must be rearranged in an external module to have the correct GEMM multiplication and addition order. Furthermore, transferring the weight matrix P to an external module and rearranging the matrix data is time-consuming and inefficient, especially when dealing with large matrices. Summary of the Invention [Problem to be solved by the invention]

[0004] In light of this, the purpose of this application is to provide an AI chip, electronic device, and convolution calculation method to solve the problem that when current AI chips perform general matrix multiplication, they need to transfer the weight matrix P externally and rearrange the matrix data, which is time-consuming and inefficient, especially when dealing with large matrices. [Means for solving the problem]

[0005] An embodiment of the present application is realized as follows.

[0006] In a first aspect, an embodiment of the present application provides an AI chip, including N convolution kernels (cores) and a memory subsystem, where N is an integer greater than or equal to 2, electrically connected to the N convolution kernels and configured to read input image data from the memory and distribute the data to each of the convolution kernels, read weight blocks from the memory, and divide each weight block into N weight data and distribute the weight data to the N convolution kernels, where each convolution kernel corresponds to one weight data and each weight block is a part of the total weights, each convolution kernel performs a convolution operation on the received weight data and the input image data, adds convolution operation results of each convolution kernel for the same weight block to obtain a convolution operation result of each weight block, and adds the convolution operation results of each weight block to obtain a final convolution operation result.

[0007] In an embodiment of the present application, the concept of tiles (blocks) is introduced to weight data, and the entire weight is divided into multiple weight blocks. Each weight block is then read out and divided into N pieces of weight data, which are then distributed to N convolution kernels. This eliminates the need for inefficient data rearrangement of weight data by an external module, reduces the number of cycles for GEMM operations, and improves the efficiency of convolution calculations.

[0008] Referring to a possible embodiment of the first aspect, the memory subsystem is configured to divide each weight block of A×B size into N pieces of weight data of A×Kpc size, or to divide each weight block of B×A size into N pieces of weight data of Kpc×A size, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each convolution.

[0009] In the embodiment of the present application, by dividing each weight block of A×B size into N pieces of weight data of A×Kpc size, or by dividing each weight block of B×A size into N pieces of weight data of Kpc×A size, it is possible to maximize the processing efficiency for both general matrix P×matrix Q and matrix PT×matrix Q. Here, matrix PT is the transpose matrix of matrix P.

[0010] Referring to possible embodiments of the first aspect, the memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module, wherein the first image data loading module is configured to read the input image data from a memory and distribute the input image data to each of the convolution kernels; the second image data loading module is configured to read each of the weight blocks from a memory; and the weight data processing modules are electrically connected to the second image data loading module and the N convolution kernels, respectively, and are configured to divide each of the A×B weight blocks into N pieces of A×Kpc weight data to distribute to the N convolution kernels, or to divide each of the B×A weight blocks into N pieces of Kpc×A weight data to distribute to the N convolution kernels.

[0011] In an embodiment of the present application, when performing a GEMM operation, instead of requesting kernel data using a conventional weight data request means, a weight matrix P is requested using a second image data loading module. The second image data loading module reads data in accordance with the 3DTile format, so that each time weight data is read, only a portion of the total weights, i.e., one weight block, is read. This eliminates the need for inefficient data relocation of weight data by an external module and reduces the number of cycles required for GEMM operations.

[0012] Referring to a possible embodiment of the first aspect, when the weight data processing module is configured to divide each of the A×B sized weight blocks into N pieces of A×Kpc sized weight data, the weight data processing module includes N registers, each of which corresponds one-to-one to each of the N convolution kernels, and is electrically connected to the second image data loading module, and each register is configured to store one piece of A×Kpc sized weight data.

[0013] In the embodiment of the present application, N registers are used to divide each weight block of size A×B into N weight data of size A×Kpc, thereby achieving the objective of the present invention while saving costs and reducing the complexity of the AI ​​chip.

[0014] Referring to a possible embodiment of the first aspect, the weight data processing module includes N extractors, which are electrically connected to the second image data loading module, and each of the extractors is configured to extract one weight data of size Kpc×A from one weight block in one clock cycle when the size of the weight block is B×A and Kpc=1.

[0015] Referring to a possible embodiment of the first aspect, the weight data processing module includes N logical shifters, which are electrically connected to the second image data loading module, and each of the shifters is configured to extract one Kpc×A size weight data from one weight block in A clock cycles when the size of the weight block is B×A and Kpc>1.

[0016] Referring to possible embodiments of the first aspect, when the weight data processing module is configured to divide each of the weight blocks of B×A size into N weight data of Kpc×A size, the weight data processing module includes N extractors, N logical shifters, and N selectors, the N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one weight data of Kpc×A size from one weight block in one clock cycle, and the N logical shifters are connected to the second image data loading module. The N selectors are electrically connected to a convolution module, and each of the N shifters is configured to extract one Kpc×A size weight data from one weight block in A clock cycles. The N selectors correspond one-to-one to each of the N convolution kernels. Each of the selectors is electrically connected to one of the extractors, one of the logical shifters, and one convolution kernel, and is configured to send the output of the extractor to the corresponding convolution kernel when Kpc=1, and to send the output of the logical shifter to the corresponding convolution kernel when Kpc>1.

[0017] In the embodiment of the present application, when Kpc=1, N extractors are used to divide each B×A sized weight block into N Kpc×A sized weight data; when Kpc>1, N logical shifters are used to divide each B×A sized weight block into N Kpc×A sized weight data, and N selectors are used to select the correct data and output it to the corresponding convolution kernel, thereby achieving the objective of the present invention, saving costs, and reducing the complexity of the AI ​​chip, and allowing the AI ​​chip to support various Kpc division methods.

[0018] Referring to possible embodiments of the first aspect, the weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors, the N registers corresponding one-to-one to the N convolution kernels, and all of the N registers are electrically connected to the second image data loading module, each of the registers being configured to store one A×Kpc size weight data, the N extractors being electrically connected to the second image data loading module, and each of the extractors being configured to extract one Kpc×A size weight data from one weight block in one clock cycle, the N logical shifters being electrically connected to the second image data loading module, and the N selectors Each of the N first selectors is configured to extract one weight data of size Kpc×A from one weight block in A clock periods, each of the N first selectors is electrically connected to one of the extractors and one of the logical shifters, and is configured to select and output the output data of the corresponding extractor when Kpc=1, and to select and output the output data of the corresponding logical shifter when Kpc>1, each of the N second selectors is electrically connected to one of the first selectors, one of the registers, and one of the convolution kernels, and is configured to select and output the output data of the corresponding register when the weight block size is A×B, and to select and output the output data of the corresponding first selector when the weight block size is B×A.

[0019] In the embodiment of the present application, the weight data processing module using the above structure can be adapted to various cases. For example, if the size of the weight block is A×B, the second image data loading module sends the weight block to N registers via path0; if the size of the weight block is B×A and Kpc=1, it sends the weight block to N extractors; if the size of the weight block is B×A and Kpc>1, it sends the weight block to N logical shifters, and then N first selectors select the correct data and output it to the corresponding second selectors, and then the second selectors select the correct data and output it to the corresponding convolution kernel. This achieves the object of the present invention, saves costs, reduces the complexity of the AI ​​chip, allows the AI ​​chip to be adapted to various cases, and improves the applicability of the technical solution.

[0020] Referring to possible embodiments according to the first aspect, each convolution kernel is further configured to, upon receiving weight data of Kpc×A size, convert the weight data of Kpc×A size into weight data of A×Kpc size before performing a convolution operation with the input image data.

[0021] In an embodiment of the present application, when weight data of size Kpc×A is received, the weight data of size Kpc×A is converted into weight data of size A×Kpc, and then a convolution operation with input image data is performed, thereby realizing a matrix P×matrix Q operation, and this AI chip can be applied not only to matrix P×matrix Q, but also to matrix P×matrix Q. In a second aspect, an embodiment of the present application further provides an electronic device, which includes a memory and an AI chip according to any one of the embodiments of the first aspect and / or possible embodiments of the embodiments of the first aspect, and the AI ​​chip is electrically connected to the memory.

[0022] In a third aspect, an embodiment of the present application further provides a convolution method, which includes the steps of reading input image data, reading weight blocks that are part of all weights, and dividing each weight block into N pieces of weight data, performing a convolution operation on each weight data with the input image data, adding the convolution operation results of the weight data belonging to the same weight block to obtain the convolution operation result for each weight block, and adding the convolution operation results of the weight blocks to obtain a final convolution operation result.

[0023] Other features and advantages of the present application will be described hereinafter. The objectives and other advantages of the present application will be realized and attained by the structure particularly pointed out in the description and drawings. [Brief explanation of the drawings]

[0024] In order to more clearly describe the embodiments of the present application or the technical solutions of the prior art, the following will briefly explain the drawings necessary for describing the embodiments. The following drawings only show some embodiments of the present application, and those skilled in the art can obtain other related drawings based on these drawings without using inventive ability. The above and other objects, features and advantages of the present application will be more apparent by showing them in the drawings.

[0025] [Figure 1] This is a schematic diagram of a structure in which an AI chip and memory are connected in related technology. [Figure 2] 1 shows a schematic diagram of a structure in which an AI chip and memory are connected according to an embodiment of the present application. [Figure 3] 1 shows a principle diagram of a convolution operation according to an embodiment of the present application; [Figure 4] 4 shows a principle diagram of making the GEMM operation according to an embodiment of the present application equivalent to the convolution operation shown in FIG. 3. [Figure 5] 1 shows a schematic diagram of the flow of data of matrix P from DDR memory to SRAM and then to the convolution kernel according to an embodiment of the present application. [Figure 6a]1 shows a schematic diagram of the flow of 2D data of matrix P from DDR memory to SRAM and then to the convolution kernel according to an embodiment of the present application. [Figure 6b] 1 shows a schematic diagram of the flow of 3D data of matrix P from DDR memory to SRAM and then to the convolution kernel according to an embodiment of the present application. [Figure 7] 1 shows a structural schematic diagram of a weight data processing module according to an embodiment of the present application; [Figure 8] 1 shows a structural schematic diagram of a weight data processing module according to an embodiment of the present application; [Figure 9] 1 shows a flowchart of a convolution method according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0026] The technical solutions in the embodiments of the present application will be described below with reference to the drawings used in the embodiments of the present application. Of course, the described embodiments are only some of the embodiments of the present application, and do not represent all of the embodiments. The following embodiments are merely illustrative and are intended to more clearly explain the technical solutions of the present application, and do not limit the scope of protection of the present application. Those skilled in the art can combine the following embodiments and features in the embodiments as long as there is no contradiction.

[0027] Note that like reference numerals refer to like elements in the drawings, and a definition in one drawing does not require further definition or interpretation in other drawings. Furthermore, in the specification of this application, relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Note that terms such as "having" and "including," and any variations thereof, are intended to cover non-exclusive inclusions. Thus, a process, method, article, or apparatus comprising a set of elements is not necessarily limited to those elements and may include other elements not explicitly stated or inherent to those processes, methods, articles, or apparatus. Unless otherwise specified, an element limited by the term "comprising" does not exclude the situation in which the process, method, article, or apparatus comprising the element also comprises other similar elements.

[0028] Furthermore, the term "and / or" used in this application is merely for the purpose of explaining the relation between related objects, and indicates that three relationships can exist. For example, A and / or B can indicate three situations: A exists alone, A and B exist simultaneously, or B exists alone.

[0029] In the description of the embodiments of the present application, unless otherwise specified, the technical term "electrically connected" may mean direct electrical connection or indirect electrical connection via an intermediate.

[0030] When current AI chips perform general matrix multiplication, they need to transfer the weight matrix P externally and rearrange the matrix data, which is time-consuming and inefficient, especially when dealing with large matrices. In light of this, the embodiments of the present application provide an AI chip, electronic device, and convolution calculation method that eliminate the need to transfer the weight matrix P externally and rearrange the matrix data, thereby improving the efficiency of convolution calculations. An AI chip according to the embodiment of the present application will be described below with reference to FIG. 2. The AI ​​chip includes N convolution kernels and a memory subsystem, where the memory subsystem is electrically connected to the N convolution kernels, N being an integer greater than or equal to 2. The AI ​​chip is also electrically connected to the memory. The memory and the memory subsystem are different devices with completely different functions and uses. The memory is used as a dedicated storage component for storing data, and the memory subsystem is used as a memory read / write control system that controls the reading and writing of data.

[0031] The memory subsystem is configured to read input image data from memory (e.g., SRAM memory and / or DDR memory) and distribute the input image data to each of the convolution kernels, and also to read each weight block (weight tile) from memory and divide each weight block into N pieces of weight data and distribute them to the N convolution kernels. Here, each convolution kernel corresponds to one weight data, and each weight block is a part of the entire weights. In an embodiment of the present application, the concept of tiles is introduced to the weight data, the entire weights are divided into multiple weight blocks, and each weight block is read and divided into N pieces of weight data and distributed to the N convolution kernels. This eliminates the need for inefficient data rearrangement of weight data by an external module and reduces the number of cycles of GEMM operations.

[0032] It should be noted that the input image data read from memory by the memory subsystem may be a portion of the entire input image data that is divided to accommodate memories of various sizes.

[0033] For ease of understanding, the following description will be given with reference to the convolution operation shown in Fig. 3. As shown in Fig. 3, the dimensions of InImage are outX x outY x kZ, and the dimensions of kernel data are kX x kY x kZ x outZ. Here, kZ is the depth in the first direction, outZ is the depth in the second direction, outX x outY is the two-dimensional size of the input image data, outX is the length of the image, outY is the height of the image, kX x kY is the size of the two-dimensional weight element group, kX is the length of the two-dimensional weight element group, and kY is the height of the two-dimensional weight element group.

[0034] The first 3-dimensional (kX×kY×kZ) kernel array is multiplied with each point in InImage with the same dimensions, resulting in kX×kY×kZ multiplication results. All kX×kY×kZ multiplication results are added together to obtain the first point in the first plane of OutImage. The 3-dimensional kernel array is subjected to a similar multiplication and addition operation in a sliding window from left to right and top to bottom within InTile to obtain the first outTile in the first plane of OutImage. The fourth dimension contains outZ 3-dimensional kernel arrays, and a similar operation is performed on each to obtain the OutImage result for a different plane on the outZ dimension.

[0035] The GEMM operation of matrix P × matrix Q = matrix R is equivalent to the convolution shown in Figure 3, and Figure 4 shows the principle behind this equivalence. If the kZ × outZ matrix P in GEMM (where kX = kY = 1 if row is kZ and column is outZ) is the kernel array, and another outX × outY × kZ matrix Q (where row is outX × outY and column is kZ) is InImage, then GEMM can be considered as the convolution of InImage with the kernel array.

[0036] When dividing the weight data (kZ × outZ), the data is first divided along the kZ direction and then along the outZ direction, resulting in multiple weight blocks of A × B or B × A size. Here, A and B are both positive integers, A is the maximum height of all convolution kernels in the first direction (e.g., the above kZ direction), for example, the maximum value of A is 9, and B is the maximum number of weight element groups processed by all convolution kernels in each convolution. The size of each weight element group is kX × kY × A. That is, one weight element group includes kX × kY × A weight elements, and when kX = kY = 1, one weight element group includes A weight elements.

[0037] Here, the memory subsystem is configured to divide each weight block of A×B size into N weight data of A×Kpc size, or to divide each weight block of B×A size into N weight data of Kpc×A size, where Kpc is a positive integer and is the maximum number of weight element groups processed by each convolution kernel in the second direction (e.g., the above-mentioned outZ direction) in each convolution.

[0038] When performing GEMM, it is necessary to first take tile data for matrix P along the kZ direction and then along the outZ direction. Due to the difference in the order of data arrangement, GEMM is divided into two cases: matrix P × matrix Q and matrix PT × matrix Q, where matrix PT is the transpose of matrix P. Here, in the case of matrix PT × matrix Q, matrix P needs to be transposed to matrix PT in order to be multiplied by matrix Q, and this step can be performed within the convolution kernel. These two cases of matrix P have 2D and 3D formats, respectively.

[0039] When performing matrix P × matrix Q operations, the kZ direction of matrix P is continuous in memory. Figure 5 shows two forms of matrix P. Figure 5 also shows the data flow from DDR memory to SRAM (cache) and then to the convolution kernel (core). Here, data is first stored in DDR memory, then loaded into SRAM, and then loaded into the convolution kernel. In this case, the memory in Figure 2 includes both DDR memory and SRAM, and the DDR memory is connected to the memory subsystem via the SRAM.

[0040] The top left of Figure 5 shows the 2D format of matrix P stored in DDR memory. in_x (length of matrix P in x direction) = kZ, Tile_x (length of Tile in x direction) = A (Tile_x of the last Tile in x direction may be smaller than Tile_x of other Tile), in_y (length of matrix P in y direction) = outZ, Tile_y (length of Tile in y direction) = B = Kpc × N (Note that Tile_y of the last Tile in y direction may be smaller than Tile_y of other Tile). The arrows indicate the sliding direction of the Tile, that is, first from left to right, then from top to bottom. Note that Figure 5 only shows the sliding direction from left to right. The bottom left of Figure 5 shows the 3D format of matrix P stored in DDR memory. in_x=kZ, Tile_x=A (Tile_x of the last Tile in the x direction is smaller than Tile_x of other Tile), in_y=outZ / B, Tile_y=1, in_z (length of matrix P in the z direction)=B (Tile_y of the last Tile in the y direction may be smaller than Tile_y of other Tile). The arrows indicate the sliding direction of the Tile, that is, from left to right first and then from top to bottom. The part shown in the center of Figure 5 is the format of matrix P cached in SRAM memory during processing, which is the same as the 2D format in DDR memory. When reading out the weight Tile, it is also read out first from left to right and then from top to bottom. The part shown on the right side of Figure 5 is the weight data received by each convolution kernel, and the size of the weight data is A × Kpc.

[0041] When performing the matrix P × matrix Q operation, the outZ direction of matrix P is continuous in memory. Figures 6a and 6b show two forms of matrix P. Figure 6a shows the flow of 2D data from DDR memory to SRAM and then to the convolution kernel (core), and Figure 6b shows the flow of 3D data from DDR memory to SRAM and then to the convolution kernel (core).

[0042] The left side of Figure 6a shows the 2D format of matrix P stored in DDR memory: in_x=outZ, Tile_x=B (Tile_x of the last Tile in the x direction may be smaller than Tile_x of other Tile), in_y=kZ, Tile_y=A (Tile_y of the last Tile in the y direction may not be smaller than Tile_y of other Tile). The arrow indicates the sliding direction of Tile, i.e., from top to bottom and then from left to right. The center part of Figure 6a shows the format of matrix P cached in SRAM, which is the same as the 2D format in DDR memory. The arrow indicates the sliding direction of Tile, i.e., from top to bottom and then from left to right. The right side of Figure 6a shows the weight data received by each convolution kernel, and the size of the weight data is Kpc × A. It needs to be transposed to A × Kpc for convolution operation.

[0043] The left part of Figure 6b shows the 3D format of matrix P stored in DDR memory: in_x=outZ, Tile_x=B (Tile_x of the last Tile in the x direction may be smaller than Tile_x of other Tile), in_y=kZ / A, Tile_y=1, in_z=B. The arrow indicates the sliding direction of Tile, i.e., first from top to bottom and then from left to right. The center part of Figure 6b shows the format of matrix P cached in SRAM during processing, which is the same as the 2D format of matrix P × matrix Q. The arrow indicates the sliding direction of Tile, i.e., first from left to right and then from top to bottom. The right part of Figure 6b shows the weight data received by each convolution kernel, and the size of the weight data is A × Kpc. No additional transposition is required during convolution operations.

[0044] When reading a weight block, the memory subsystem reads the weight block from the DDR memory or the SRAM memory in order according to the principles shown in FIGS. 5, 6A, and 6B. When reading a weight block, the memory subsystem reads it from the SRAM memory first, and if the corresponding weight block is not available in the SRAM memory, it reads it from the DDR memory. The weight block read from the DDR must be stored and cached in the SRAM. In any embodiment, the SRAM memory may be internal to the AI ​​chip, and the DDR memory may be external to the AI ​​chip.

[0045] In one embodiment, the memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module, wherein the first image data loading module is electrically connected to the memory and the N convolution kernels, the second image data loading module is electrically connected to the memory and the weight data processing module, and the weight data processing module is electrically connected to the N convolution kernels, respectively.

[0046] The first image data loading module is configured to read input image data from the memory and distribute the input image data to each convolution kernel. The second image data loading module is configured to read each weight block from the memory and transmit each read weight block to the weight data processing module. The weight data processing module is electrically connected to the second image data loading module and the N convolution kernels, respectively, and is configured to divide each weight block into N pieces of weight data and distribute them to the N convolution kernels.

[0047] In an embodiment of the present application, when performing a GEMM operation, instead of requesting kernel data using a conventional weight data request means, a weight matrix P is requested using a second image data loading module. The second image data loading module reads data in accordance with the 3DTile format, so that each time weight data is read, only a portion of the total weights, i.e., one weight block, is read. This eliminates the need for inefficient data relocation of weight data by an external module and reduces the number of cycles required for GEMM operations.

[0048] Specifically, the weight data processing module is configured to divide each weight block of A×B size into N weight data of A×Kpc size and distribute them to the N convolution kernels, or to divide each weight block of B×A size into N weight data of Kpc×A size and distribute them to the N convolution kernels.

[0049] In one embodiment, the weight data processing module is configured to divide each of the A×B weight blocks into N A×Kpc weight data blocks, and includes N registers. The N registers correspond one-to-one to the N convolution kernels, and are all electrically connected to the second image data loading module. Each register is configured to store one A×Kpc weight data block. Here, the register is a variable length writer (VLM), which can receive weight data with a variable effective length and store the effective data until it reaches the output width before sending it to the corresponding convolution kernel.

[0050] To ensure performance, the maximum input width of the VLM is min(sram_data_width A × max_kpc × bpp), and the output width is max_kpc × bpp. The parameter sram_data_width represents the width of the SRAM, and bpp represents the number of bytes per weight element. Here, the weight blocks read from DDR must be stored and cached in SRAM.

[0051] In another embodiment, the weight data processing module is configured to divide each B×A weight block into N Kpc×A weight data. As shown in FIG. 7, the weight data processing module includes N extractors, N logical shifters, and N selectors. Each extractor and each logical shifter is connected to the second image data loading module, and one selector is connected to one extractor and logical shifter. All N extractors are electrically connected to the second image data loading module, and when Kpc=1, the second image data loading module sends weight block data to the extractors. Each extractor is configured to extract one Kpc×A weight data from one weight block per clock cycle. In this embodiment, the second image data loading module can send A B weight data to the extractors per clock cycle.

[0052] When Kpc=1, adjacent weight elements belong to different convolution kernels, and the valid data transmitted per cycle includes A B arrays (row arrays). Assuming Kpc=1 and the number of convolution kernels N=24, then B=Kpc×N=1×24. Assuming A=9 and each weight element occupies 1 byte, the data width transmitted per cycle is 216 bytes. In this case, Byte0, Byte24, Byte48, Byte72, Byte96, Byte120, Byte144, Byte168, and Byte196 belong to core1, and Byte1, Byte25, Byte49, Byte73, Byte97, Byte121, Byte145, Byte169, and Byte197 belong to core2, and so on. In this embodiment, A B arrays can be transmitted per clock cycle.

[0053] All of the N logical shifters are electrically connected to the second image data loading module, and when Kpc>1, the second image data loading module sends weight block data to the logical shifter. Each shifter is configured to extract one Kpc×A size weight data from one weight block in A clock cycles. In this embodiment, the second image data loading module sends only one B weight data to the logical shifter in one clock cycle. Therefore, one B×A weight block is sent in A clock cycles.

[0054] When Kpc>1, a maximum of one B array can be transmitted per cycle. Assuming Kpc=2, the number of convolution kernels N=24, and each weight element occupies 1 byte, Kpc×N=2×24=48, meaning the data width transmitted per cycle is 48 bytes. In this case, Byte0 and Byte1 belong to core1, Byte2 and Byte3 belong to core2, ..., Byte46 and Byte47 belong to core24. Therefore, logical shifter 1 is responsible for outputting Byte0 and Byte1, logical shifter 2 is responsible for outputting Byte2 and Byte3, ..., logical shifter 24 is responsible for outputting Byte46 and Byte47.

[0055] The N selectors correspond one-to-one to the N convolution kernels, and each selector is electrically connected to one extractor, one logical shifter, and one convolution kernel, and is configured to send the output of the extractor to the corresponding convolution kernel when Kpc=1, and to send the output of the logical shifter to the corresponding convolution kernel when Kpc>1.

[0056] Note that if the size of the weight block is B×A and Kpc=1, the weight data processing module can include only N extractors (not including N logical shifters and N selectors). At this time, the N extractors are directly connected to the N convolution kernels. Similarly, if the size of the weight block is B×A and Kpc>1, the weight data processing module can include only N logical shifters (not including N extractors and N selectors). At this time, the N logical shifters are directly connected to the N convolution kernels. The structure shown in FIG. 7 is the structure of a weight data processing module that can accommodate various Kpc.

[0057] If the size of a weight block is B×A and Kpc>1, an extractor can be used to divide each B×A weight block into N weight data blocks of Kpc×A size. Since the size of Kpc varies depending on the case, and the allocation of Kpc for each convolution kernel for tiles at the edges of matrix P may be uneven, this can lead to many variations in the design of the division and rearrangement. Therefore, to reduce the area overhead due to these variations, a logical shifter is used when Kpc>1.

[0058] 8, the weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors. When transmitting a weight block, the second image data loading module does not simultaneously transmit the weight block to at least two of the extractors, the logical shifters, and the registers. If the size of the weight block is A×B, the second image data loading module transmits the weight block to the N registers via path0. If the size of the weight block is B×A and Kpc=1, the second image data loading module transmits the weight block to the N extractors. If the size of the weight block is B×A and Kpc>1, the second image data loading module transmits the weight block to the N logical shifters.

[0059] The N registers correspond to the N convolution kernels one-to-one, and are electrically connected to the second image data loading module, each of which is configured to store one A×Kpc weight data.

[0060] The N extractors are all electrically connected to the second image data loading module, and each extractor is configured to extract one Kpc×A size weight data from one weight block in one clock cycle.

[0061] The N logical shifters are all electrically connected to the second image data loading module, and each shifter is configured to extract one Kpc×A size weight data from one weight block in A clock cycles.

[0062] Each of the first selectors is electrically connected to one extractor and one logical shifter, and is configured to select and output the output data of the corresponding extractor when Kpc=1, and to select and output the output data of the corresponding logical shifter when Kpc>1.

[0063] Each of the second selectors is electrically connected to one first selector, one register, and one convolution kernel, and is configured to select and output the output data of the corresponding register when the size of the weight block is A×B (or the size of the weight data is A×Kpc), and to select and output the output data of the corresponding first selector when the size of the weight block is B×A (or the size of the weight data is Kpc×A).

[0064] Each convolution kernel is configured to perform a convolution operation on the received weight data with the input image data. Specifically, when each convolution kernel receives weight data of A×Kpc size, it performs a convolution operation on the received weight data with the input image data, and when each convolution kernel receives weight data of Kpc×A size, it converts the weight data of Kpc×A size into weight data of A×Kpc size and then performs a convolution operation on the input image data.

[0065] Here, the convolution operation results of each convolution kernel for the same weight block are added according to the corresponding positions to obtain the convolution operation result of each weight block, and the convolution operation results of each weight block are added according to the corresponding positions to obtain the final convolution operation result.

[0066] In some possible embodiments, the AI ​​chip may further include a post-processing module connected to each convolution kernel and the memory, configured to add the convolution operation results of each convolution kernel for the same weight block according to corresponding positions to obtain the convolution operation result of each weight block, and to add the convolution operation results of each weight block according to corresponding positions to obtain a final convolution operation result.

[0067] Here, the AI ​​chip is an integrated circuit chip with signal processing capabilities. The AI ​​chip may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, capable of implementing or executing the methods, steps, and logic blocks disclosed in the embodiments of the present application. Alternatively, the AI ​​chip may be any conventional processor.

[0068] The AI ​​chip may be a dedicated computational accelerator chip (also called an accelerator) designed to handle massive computational tasks, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processor, or it may of course be a processor for other AI computational tasks.

[0069] An embodiment of the present application further provides an electronic device, the electronic device including a memory and the AI ​​chip electrically connected to the memory, the memory storing data necessary for GEMM operations, such as input image data and weight data.

[0070] The electronic devices include, but are not limited to, tablets, laptops, in-vehicle devices, and servers.

[0071] The embodiment of the present application further provides a convolution calculation method, which can be applied to the above AI chip. Hereinafter, the convolution calculation method according to the embodiment of the present application will be described with reference to FIG.

[0072] S1: Input image data is read, and each weight block is read and divided into N weight data.

[0073] The first image data loading module reads input image data from memory, the second image data loading module reads each weight block from memory, and the weight data processing module divides each weight block into N pieces of weight data. For example, the second image data loading module sends each read weight block to the weight data processing module, which divides each weight block into N pieces of weight data, where each weight block is a part of the entire weight.

[0074] The first image data loading module reads input image data and distributes the input image data to N convolution kernels, where N is an integer greater than or equal to 2.

[0075] The weight data processing module divides each weight block into N weight data, and then distributes the N weight data to N convolution kernels, each convolution kernel corresponding to one weight data.

[0076] When dividing each weight block into N pieces of weight data, the weight data processing module can divide each A×B sized weight block into N pieces of A×Kpc sized weight data, or can divide each B×A sized weight block into N pieces of Kpc×A sized weight data, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each convolution.

[0077] S2: A convolution operation is performed between each weight data and the input image data.

[0078] The convolution kernel in the AI ​​chip performs a convolution operation between each of the received weight data and the input image data.

[0079] S3: The convolution operation results of each weight data belonging to the same weight block are added together to obtain the convolution operation result of each weight block.

[0080] The post-processing module adds up the convolution results of the weight data belonging to the same weight block to obtain the convolution result of each weight block.

[0081] S4: The convolution results of each weight block are added together to obtain the final convolution result.

[0082] The post-processing module adds the respective convolution results of each weight block to obtain the final convolution result.

[0083] The convolution operation method according to the embodiments of the present application has the same implementation principle and technical effect as the above-mentioned AI chip embodiments, and for the sake of brevity, for parts not mentioned in the method embodiments, reference will be made to the corresponding content in the above-mentioned AI chip embodiments.

[0084] An embodiment of the present application provides a non-volatile computer-readable storage medium (hereinafter referred to as the storage medium), which stores a computer program that, when executed by the electronic device in a computer, executes the above-described convolution operation method.

[0085] In addition, each embodiment in this specification is described progressively, and in each embodiment, emphasis is placed on describing the parts that are different from other embodiments, and similar or similar parts between each embodiment may be cross-referenced.

[0086] In some embodiments of the present application, the described apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, flowcharts and block diagrams in the drawings illustrate possible architectures, functions, and operations based on apparatuses, methods, and computer program products according to various embodiments of the present invention. Each block in the flowchart or block diagram may represent a module, program segment, or part of code, including one or more executable instructions that can implement a specific logical function. In alternative implementations, the functions described in the blocks may be implemented in a different order than shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or may even be executed in the reverse order, depending on the required functionality. Each block in the block diagram and / or flowchart, and a combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs a specific function or operation, or by a combination of dedicated hardware and computer instructions.

[0087] Furthermore, each functional module in each embodiment of the present application may be an independent part formed by integration, or may be a stand-alone module, or may be an independent part formed by integration of two or more modules.

[0088] The functions can be realized in the form of software functional units and stored in a computer-readable storage medium when sold or used as an independent product. From this understanding, the technical solutions of the present application, or the parts that contribute to the prior art, or parts of the technical solutions, can be realized in the form of a software product. The computer software product is stored in a computer-readable storage medium and includes a plurality of commands for causing a computer device (such as a personal computer, laptop, server, or electronic device) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The computer-readable storage medium includes various media capable of storing program code, such as a USB disk, a portable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0089] The above are only specific embodiments of the present application, and the scope of protection of the present application is not limited thereto. Those skilled in the art may modify or replace the technical solutions within the technical scope disclosed in the present application, and these modifications or replacements are also included in the scope of protection of the present application. Therefore, the scope of protection of the present application is subject to the content set forth in the claims.

Claims

1. N convolution kernels and a memory subsystem; The N convolution kernels are an integer equal to or greater than 2, the memory subsystem is electrically connected to the N convolution kernels and configured to read input image data from the memory and distribute the input image data to each of the convolution kernels; read weight blocks from the memory, divide each weight block into N weight data and distribute the weight data to the N convolution kernels, where each convolution kernel corresponds to one weight data and each weight block is a portion of the entire weight; Each of the convolution kernels is configured to perform a convolution operation between the received weight data and the input image data, add the convolution operation results of each of the convolution kernels for the same weight block to obtain the convolution operation result of each weight block, and add the convolution operation results of each of the weight blocks to obtain the final convolution operation result. An AI chip characterized by the above.

2. The memory subsystem is configured to divide each weight block of A×B size into N weight data of A×Kpc size, or to divide each weight block of B×A size into N weight data of Kpc×A size, where A, B, and Kpc are all positive integers, B is the maximum number of weight element groups processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight element groups processed by each convolution kernel in the second direction in each convolution. The AI ​​chip according to claim 1 .

3. the memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module; the first image data loading module is configured to read the input image data from a memory and deliver the input image data to each of the convolution kernels; the second image data loading module is configured to read each of the weight blocks from a memory; The weight data processing module is electrically connected to the second image data loading module and the N convolution kernels, and is configured to divide each weight block of A×B size into N weight data of A×Kpc size and distribute them to the N convolution kernels, or to divide each weight block of B×A size into N weight data of Kpc×A size and distribute them to the N convolution kernels. The AI ​​chip according to claim 2 .

4. When the weight data processing module is configured to divide each of the weight blocks of A×B size into N weight data of A×Kpc size, the weight data processing module includes N registers; The N registers correspond one-to-one to the N convolution kernels, and are all electrically connected to the second image data loading module, and each register is configured to store one A×Kpc size weight data.

4. The AI ​​chip according to claim 3.

5. The weight data processing module includes N extractors; the N extractors are electrically connected to the second image data loading module; Each of the extractors is configured to extract one weight data of size Kpc×A from one weight block in one clock period when the size of the weight block is B×A and Kpc=1.

4. The AI ​​chip according to claim 3.

6. the weight data processing module includes N logical shifters; the N logical shifters are electrically connected to the second image data loading module; Each of the shifters is configured to extract one weight data of size Kpc×A from one weight block in A clock periods when the size of the weight block is B×A and Kpc>1.

4. The AI ​​chip according to claim 3.

7. When the weight data processing module is configured to divide each of the B×A size weight blocks into N Kpc×A size weight data, the weight data processing module includes N extractors, N logical shifters, and N selectors; The N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one Kpc×A size weight data from one weight block in one clock cycle; The N logical shifters are electrically connected to the second image data loading module, and each of the shifters is configured to extract one Kpc×A size weight data from one weight block in A clock cycles; The N selectors correspond one-to-one to the N convolution kernels, and each of the selectors is electrically connected to one of the extractors, one of the logical shifters, and one of the convolution kernels, and is configured to send the output of the extractor to the corresponding convolution kernel when Kpc=1, and to send the output of the logical shifter to the corresponding convolution kernel when Kpc>1.

4. The AI ​​chip according to claim 3.

8. the weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors; The N registers correspond to the N convolution kernels in a one-to-one manner, and are all electrically connected to the second image data loading module, and each register is configured to store one A×Kpc size weight data; The N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one Kpc×A size weight data from one weight block in one clock cycle; The N logical shifters are electrically connected to the second image data loading module, and each of the shifters is configured to extract one Kpc×A size weight data from one weight block in A clock cycles; each of the N first selectors is electrically connected to one of the extractors and one of the logical shifters, and is configured to select and output the output data of the corresponding extractor when Kpc=1, and to select and output the output data of the corresponding logical shifter when Kpc>1; Each of the N second selectors is electrically connected to one of the first selectors, one of the registers, and one of the convolution kernels, and is configured to select and output output data of the corresponding register when the size of the weight block is A×B, and to select and output output data of the corresponding first selector when the size of the weight block is B×A.

4. The AI ​​chip according to claim 3.

9. Each of the convolution kernels is further configured to, upon receiving weight data of Kpc×A size, convert the weight data of Kpc×A size into weight data of A×Kpc size, and then perform a convolution operation with the input image data. The AI ​​chip according to any one of claims 1 to 8.

10. A memory and the AI ​​chip according to any one of claims 1 to 9, wherein the AI ​​chip is electrically connected to the memory. An electronic device characterized by:

11. A step of reading input image data, reading each weight block that is a part of the entire weight, and dividing each weight block into N weight data; performing a convolution operation between each weight data and the input image data; adding up the convolution operation results of the weight data belonging to the same weight block to obtain the convolution operation result of each weight block; and adding the convolution results of each weight block to obtain a final convolution result. A convolution calculation method comprising:

Images