AI chip, electronic device, and convolutional computation method

Abstract

This application relates to an AI chip, an electronic device, and a convolution calculation method, and is in the field of artificial intelligence. The AI ​​chip includes N convolution kernels and a memory subsystem electrically connected to the N convolution kernels. The memory subsystem is configured to read input image data from the memory and distribute it to each convolution kernel, read weight blocks from the memory, and divide each weight block into N weight data and distribute them to the N convolution kernels. Each convolution kernel corresponds to one weight data, and each weight block is a part of the total weights. Each convolution kernel is configured to perform a convolution calculation of the received weight data with the input image data, add the convolution calculation results of each convolution kernel for the same weight block to obtain the convolution calculation result of each weight block, and add the convolution calculation results of each weight block to obtain a final convolution calculation result.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to AI chips, electronic devices, and convolutional operation methods.

Background Art

[0002] In related technologies, General Matrix Multiplication (GEMM) is usually realized by an artificial intelligence (AI) chip such as a neural-network processing unit (NPU) for neural networks. As shown in FIG. 1, the structure of the NPU chip is such that a memory subsystem (MSS) reads input image data (for example, represented by InImage and used for convolution operation with weight data), weight data (for example, represented by kernel and used for convolution operation with input image data), and 2ndImage data (used for addition (tensor_add) or multiplication (tensor_mult) with the image data output from the convolution operation) from a memory (such as DDR (Double Data Rate) or SRAM (Static Random Access Memory)). Here, the InImage data and the kernel data are convolved with a convolution kernel, and the convolved data is post-processed (including activation, pooling, or tensor_add / tensor_mult with 2ndImage data, etc.) by a post-processing module (postprocess), and the post-processed data is stored in the memory as the final output image data (for example, represented by OutImage).

[0003] To reduce hardware area and improve processing efficiency, InImage, 2ndImage, and OutImage data are all processed in the form of 3DTile slices. However, the hardware does not have the concept of 3DTiles when reading weight data (kernel), and in order to perform the correct GEMM operation with InImageTile, the array order of the kernel data must be adjusted offline before being stored in memory. In practice, the weight matrix P may originate from the output data of a layer of the neural network, and before convolution with the InImage data, it needs to be rearranged in the correct GEMM multiplication and addition order by an external module. Furthermore, transferring the weight matrix P to an external module and rearranging the matrix data is time-consuming and inefficient, especially when dealing with large matrices. [Overview of the project] [Problems that the invention aims to solve]

[0004] In light of this, the purpose of this application is to provide an AI chip, electronic device, and convolution method to improve the problem that current AI chips, when performing general matrix multiplication, need to transfer the weight matrix P to an external source and rearrange the matrix data, which is time-consuming and inefficient, especially when dealing with large matrices. [Means for solving the problem]

[0005] The embodiment of this application is realized as follows.

[0006] In a first embodiment, the present invention provides an AI chip. This AI chip includes N convolution kernels (cores) and a memory subsystem, wherein the N convolution kernels are integers N greater than or equal to 2, and the memory subsystem is electrically connected to the N convolution kernels and is configured to read input image data from memory and distribute it to each of the convolution kernels, read each weight block from memory, divide each weight block into N weight data and distribute them to the N convolution kernels, where each convolution kernel corresponds to one weight data, each weight block is a part of the whole weight, each convolution kernel performs a convolution operation between the received weight data and the input image data, adds the convolution results of each convolution kernel for the same weight block to obtain the convolution result for each weight block, and adds the convolution results of each weight block to obtain the final convolution result.

[0007] In the embodiment of this application, the concept of tile (block) is introduced to the weight data, the entire weight is divided into multiple weight blocks, each weight block is read out, and each weight block is divided into N weight data and distributed to N convolution kernels. This eliminates the need for inefficient data rearrangement of weight data by external modules, reduces the number of cycles in GEMM operations, and improves the efficiency of convolution operations.

[0008] Referring to a possible embodiment according to the first embodiment, the memory subsystem is configured to divide each A×B weight block into N A×Kpc weight data, or to divide each B×A weight block into N Kpc×A weight data, where A, B, and Kpc are all positive integers, B is the maximum number of weight elements processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight elements processed by each convolution kernel in the second direction in each convolution.

[0009] In the embodiment of this application, processing efficiency can be maximized for either a general matrix P × matrix Q or matrix PT × matrix Q by dividing each A × B size weight block into N A × Kpc size weight data, or by dividing each B × A size weight block into N Kpc × A size weight data. Here, matrix PT is the transpose of matrix P.

[0010] Referring to possible embodiments of the first embodiment, the memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module, wherein the first image data loading module is configured to read the input image data from memory and distribute the input image data to each of the convolutional kernels, the second image data loading module is configured to read each of the weight blocks from memory, and the weight data processing module is electrically connected to the second image data loading module and the N convolutional kernels, respectively, and is configured to divide each A×B weight block into N A×Kpc weight data and distribute them to the N convolutional kernels, or to divide each B×A weight block into N Kpc×A weight data and distribute them to the N convolutional kernels.

[0011] In the embodiment of this application, when performing a GEMM operation, the weight matrix P is requested using a second image data loading module, rather than requesting kernel data using a conventional weight data requesting means. Since the second image data loading module reads data in 3DTile format, each time weight data is read, only a portion of the entire weight, i.e., one weight block, is read. This eliminates the need for inefficient data rearrangement of weight data by an external module, and reduces the number of cycles in the GEMM operation.

[0012] Referring to a possible embodiment of the first embodiment, if the weight data processing module is configured to divide each A×B weight block into N A×Kpc weight data, the weight data processing module includes N registers, each of which corresponds one-to-one with each of the N convolution kernels and is electrically connected to the second image data loading module, and each register is configured to store one A×Kpc weight data.

[0013] In the embodiment of this application, by using N registers to divide each A×B sized weight block into N A×Kpc sized weight data, the objectives of the present invention can be achieved, costs can be saved, and the complexity of the AI ​​chip can be reduced.

[0014] Referring to possible embodiments of the first embodiment, the weight data processing module includes N extractors, the N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one Kpc × A weight data from one weight block in one clock cycle, where the weight block size is B × A and Kpc = 1.

[0015] Referencing a possible embodiment of the first embodiment, the weight data processing module includes N logical shifters, the N logical shifters are electrically connected to the second image data loading module, and each of the shifters is configured to extract one Kpc × A weight data from one weight block in A clock periods, provided that the size of the weight block is B × A and Kpc > 1.

[0016] Referring to a possible embodiment of the first embodiment, if the weight data processing module is configured to divide each B×A weight block into N Kpc×A weight data, the weight data processing module includes N extractors, N logical shifters, and N selectors, the N extractors being electrically connected to the second image data loading module, each of the extractors being configured to extract one Kpc×A weight data from one weight block in one clock cycle, and the N logical shifters being connected to the second image data loading module. The N selectors are electrically connected to the shifting module, and each of the shifters is configured to extract one Kpc × A size weight data from one weight block in A clock periods, with each of the N selectors corresponding one-to-one to each of the N convolution kernels, and each of the selectors is electrically connected to one of the extractors, one of the logical shifters, and one convolution kernel, and is configured to send the output of the extractor to the corresponding convolution kernel when Kpc = 1, and to send the output of the logical shifter to the corresponding convolution kernel when Kpc > 1.

[0017] In the embodiment of this application, when Kpc=1, each B×A weight block is divided into N Kpc×A weight data using N extractors, and when Kpc>1, each B×A weight block is divided into N Kpc×A weight data using N logic shifters, and the correct data is selected using N selectors and output to the corresponding convolution kernel. This achieves the objective of the present invention, saves costs, reduces the complexity of the AI ​​chip, and allows the AI ​​chip to support various Kpc division methods.

[0018] Referring to possible embodiments of the first embodiment, the weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors, wherein the N registers correspond one-to-one with each of the N convolution kernels and are all electrically connected to the second image data loading module, each of the registers is configured to store one A × Kpc size weight data, the N extractors are electrically connected to the second image data loading module and each of the extractors is configured to extract one Kpc × A size weight data from one weight block in one clock cycle, and the N logical shifters are electrically connected to the second image data loading module and each of the shifters Each of the N first selectors is configured to extract one Kpc × A size weight data from one weight block in A clock periods, and each of the N first selectors is electrically connected to one of the extractors and one of the logic shifters, and is configured to select and output the output data of the corresponding extractor when Kpc = 1, and to select and output the output data of the corresponding logic shifter when Kpc > 1, and each of the N second selectors is electrically connected to one of the first selectors, one of the registers and one of the convolution kernels, and is configured to select and output the output data of the corresponding register when the size of the weight block is A × B, and to select and output the output data of the corresponding first selector when the size of the weight block is B × A.

[0019] In the embodiment of this application, the weight data processing module using the above structure can handle various cases. For example, if the size of the weight block is A × B, the second image data loading module sends the weight block to N registers via the path0 path; if the size of the weight block is B × A and Kpc = 1, it sends the weight block to N extractors; if the size of the weight block is B × A and Kpc > 1, it sends the weight block to N logical shifters, then the correct data is selected by N first selectors and output to the corresponding second selectors, and further correct data is selected by the second selectors and output to the corresponding convolution kernel. This achieves the objectives of the present invention, saves costs, reduces the complexity of the AI ​​chip, allows the AI ​​chip to handle various cases, and improves the applicability of the technology.

[0020] Referring to possible embodiments of the first embodiment, each convolution kernel is further configured to receive weight data of size Kpc × A, convert the Kpc × A weight data to A × Kpc weight data, and then perform a convolution operation with the input image data.

[0021] In the embodiment of this application, upon receiving weight data of size Kpc × A, the weight data of size Kpc × A is converted to weight data of size A × Kpc, and then a convolution operation is performed with the input image data to realize a matrix PT × matrix Q operation. This AI chip can be applied not only to matrix P × matrix Q but also to matrix PT × matrix Q. In a second embodiment, the embodiment of this application further provides an electronic device comprising a memory and an AI chip according to any one of the embodiments of the first embodiment and / or an embodiment of the first embodiment, wherein the AI ​​chip is electrically connected to the memory.

[0022] In a third aspect, an embodiment of the present application further provides a convolution operation method. This convolution operation method includes steps of reading input image data, reading each weight block which is a part of the entire weights, dividing each of the weight blocks into N weight data, performing a convolution operation of each of the weight data with the input image data, adding the convolution operation results of the weight data belonging to the same weight block to obtain the convolution operation result of each weight block, and adding the convolution operation results of each weight block to obtain a final convolution operation result.

[0023] Other features and advantages of the present application will be described in the following part. The objectives and other advantages of the present application can be realized and obtained by the structure particularly shown in the specification and drawings.

Brief Description of the Drawings

[0024] To more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings necessary for the description of the embodiments will be briefly described below. The following drawings only show some embodiments of the present application, and those skilled in the art can obtain other related drawings based on these drawings without using inventive capabilities. By showing in the drawings, the above and other objectives, features, and advantages of the present application will become clearer.

[0025] [Figure 1] It is a schematic diagram of the structure in which an AI chip and a memory are connected in the related art. [Figure 2] A schematic diagram of the structure in which an AI chip and a memory are connected according to an embodiment of the present application is shown. [Figure 3] A schematic diagram of the principle of the convolution operation according to an embodiment of the present application is shown. [Figure 4] A schematic diagram of the principle of making the GEMM operation according to an embodiment of the present application equivalent to the convolution operation shown in FIG. 3 is shown. [Figure 5] A schematic diagram of the flow of the data of matrix P from the DDR memory to the SRAM and then to the convolution kernel according to an embodiment of the present application is shown. [Figure 6a]This diagram shows a schematic representation of the flow of 2D data of matrix P according to the embodiment of this application, from DDR memory to SRAM and then to the convolution kernel. [Figure 6b] This diagram shows a schematic representation of the flow of 3D data of matrix P according to the embodiment of this application, from DDR memory to SRAM and then to the convolution kernel. [Figure 7] This diagram shows a schematic structure of the weight data processing module according to an embodiment of this application. [Figure 8] This diagram shows a schematic structure of the weight data processing module according to an embodiment of this application. [Figure 9] A flowchart of the convolution operation method according to the embodiment of this application is shown. [Modes for carrying out the invention]

[0026] The technical concept in the embodiments of this application will be described below with reference to the drawings used in the embodiments of this application. It goes without saying that the embodiments described are only a selection of embodiments of this application, and not all embodiments. The following embodiments are illustrative and are intended only to more clearly illustrate the technical concept of this application, and do not limit the scope of protection of this application. Those skilled in the art can combine the following embodiments and features, insofar as they do not contradict each other.

[0027] Furthermore, similar reference numerals indicate similar things in the drawings, so if they are defined in one drawing, it is unnecessary to define or interpret them further in other drawings. Also, in the specification of this application, relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any factual relationship or order between such entities or actions. Furthermore, terms such as "having" and "including" and any variations thereof are intended to cover non-exclusive inclusion. Thus, a process, method, article or apparatus that includes a set of elements is not necessarily limited to these elements, and may include other elements that are not specified or are specific to these processes, methods, articles or apparatus. Unless otherwise specified, the expression "including" does not exclude situations in which the process, method, article or apparatus that includes such elements also has other similar elements.

[0028] Furthermore, the term "and / or" as used in this application is merely to describe the relationship between related subjects, indicating that three relationships are possible. For example, A and / or B can represent three situations: A existing alone, A and B existing simultaneously, or B existing alone.

[0029] In the description of the embodiments of this application, unless otherwise specified, the technical term "electrically connected" may mean directly electrically connected or indirectly electrically connected via an intermediate.

[0030] Current AI chips perform general matrix multiplication, which requires transferring the weight matrix P externally and rearranging the matrix data. This process is time-consuming and inefficient, especially when dealing with large matrices. In light of this problem, the embodiment of this application provides an AI chip, electronic equipment, and a convolution operation method that eliminates the need to transfer the weight matrix P externally and rearrange the matrix data, thereby improving the efficiency of the convolution operation. The AI ​​chip according to the embodiment of this application will be described below with reference to Figure 2. This AI chip includes N convolution kernels and a memory subsystem, the memory subsystem being electrically connected to the N convolution kernels, where N is an integer greater than or equal to 2. This AI chip is also electrically connected to memory. Here, memory and the memory subsystem are different devices, and their functions and applications are entirely different. Memory is used as a dedicated storage element for storing data, while the memory subsystem is used as a memory read / write control system that controls the reading and writing of data.

[0031] The memory subsystem is configured to read input image data from memory (e.g., SRAM memory and / or DDR memory), distribute the input image data to each convolutional kernel, read each weight block (weight tile) from memory, divide each weight block into N weight data, and distribute them to N convolutional kernels. Here, each convolutional kernel corresponds to one weight data, and each weight block is a part of the whole weight. In the embodiment of this application, by introducing the concept of tile to the weight data, dividing the whole weight into multiple weight blocks, reading each weight block, and dividing each weight block into N weight data to distribute them to N convolutional kernels, inefficient data rearrangement of weight data by an external module is eliminated, and the number of GEMM operation cycles is reduced.

[0032] Furthermore, the input image data read from memory by the memory subsystem may be only a portion of the entire input image data, which is divided to accommodate various memory sizes.

[0033] To facilitate understanding, the following explanation will refer to the convolution operation shown in Figure 3. As shown in Figure 3, the dimensions of InImage are outX × outY × kZ, and the dimensions of kernel data are kX × kY × kZ × outZ. Here, kZ is the depth in the first direction, outZ is the depth in the second direction, outX × outY is the 2D size of the input image data, outX is the length of the image, outY is the height of the image, kX × kY is the size of the 2D weight element group, kX is the length of the 2D weight element group, and kY is the height of the 2D weight element group.

[0034] The first 3D (kX×kY×kZ) kernel array is multiplied by each point in the InImage with the same dimension, resulting in kX×kY×kZ multiplication results. Adding all kX×kY×kZ multiplication results together gives the first point on the first plane of the OutImage. The 3D kernel array is then slid through the InTile from left to right and top to bottom, and similar multiplication and addition operations are performed to obtain the first outTile on the first plane of the OutImage. The fourth dimension has outZ 3D kernel arrays, and similar operations are performed on each to obtain OutImage results for different planes on the outZ dimension.

[0035] The GEMM operation of matrix P × matrix Q = matrix R is equivalent to the convolution shown in Figure 3, and Figure 4 shows the principle diagram that makes them equivalent. If we consider the kZ × outZ matrix P in GEMM (where the rows are kZ and the columns are outZ, so kX=kY=1) as the kernel array, and another outX × outY × kZ matrix Q (where the rows are outX × outY and the columns are kZ) as the InImage, then GEMM can be considered as a convolution of the InImage and the kernel array.

[0036] When dividing weight data (kZ × outZ), the data is first divided along the kZ direction, then along the outZ direction, and finally divided into multiple weight blocks of size A × B or B × A. Here, A and B are both positive integers, where A is the maximum height of all convolution kernels in the first direction (e.g., the kZ direction above), for example, the maximum value of A is 9, and B is the maximum number of weight element groups processed by all convolution kernels in each convolution. The size of each weight element group is kX × kY × A. That is, one weight element group contains kX × kY × A weight elements, and when kX = kY = 1, one weight element group contains A weight elements.

[0037] Here, the memory subsystem is configured to either divide each A×B weight block into N A×Kpc weight data, or divide each B×A weight block into N Kpc×A weight data, where Kpc is a positive integer and is the maximum number of weight elements processed by each convolution kernel in the second direction (e.g., the outZ direction above) in each convolution.

[0038] When performing GEMM, it is necessary to obtain tile data for matrix P first along the kZ direction and then along the outZ direction. Due to the difference in the order of data arrangement, GEMM can be divided into two cases: matrix P × matrix Q and matrix PT × matrix Q, where matrix PT is the transpose of matrix P. In the case of matrix PT × matrix Q, matrix P must be transposed to matrix PT in order to multiply it with matrix Q, and this step can be performed within the convolution kernel. These two cases of matrix P have a 2D form and a 3D form, respectively.

[0039] When performing an operation involving matrix P × matrix Q, the kZ direction of matrix P is contiguous within memory. Figure 5 shows two forms of matrix P. Figure 5 shows the data flow from DDR memory to SRAM (cache) and then to the convolution kernel (core). Here, data is first stored in DDR memory, then loaded into SRAM, and finally loaded into the convolution kernel. In this case, the memory in Figure 2 includes both DDR memory and SRAM, and the DDR memory is connected to the memory subsystem via SRAM.

[0040] The upper left of Figure 5 shows the 2D form of matrix P stored in DDR memory. in_x (length of matrix P in the x direction) = kZ, Tile_x (length of tile in the x direction) = A (the Tile_x of the last tile in the x direction may be smaller than the Tile_x of other tiles), in_y (length of matrix P in the y direction) = outZ, Tile_y (length of tile in the y direction) = B = Kpc × N (note that the Tile_y of the last tile in the y direction may be smaller than the Tile_y of other tiles). The arrows indicate the sliding direction of the tile, that is, from left to right first, and then from top to bottom. Here, Figure 5 only shows the sliding direction from left to right. The lower left of Figure 5 shows the 3D form of matrix P stored in DDR memory. in_x=kZ, Tile_x=A (the Tile_x of the last Tile in the x-direction is smaller than the Tile_x of the other Tiles), in_y=outZ / B, Tile_y=1, in_z (length of matrix P in the z-direction)=B (the Tile_y of the last Tile in the y-direction may be smaller than the Tile_y of the other Tiles). The arrows indicate the sliding direction of the Tiles, that is, from left to right first, and then from top to bottom. The central part of Figure 5 shows the format of matrix P cached in the SRAM memory during processing, which is the same as the 2D format in DDR memory. When reading weight Tiles, they are also read from left to right first, and then from top to bottom. The right side of Figure 5 shows the weight data received by each convolution kernel, with a size of A × Kpc.

[0041] When performing the operation of matrix PT × matrix Q, the outZ direction of matrix P is contiguous within memory. Figures 6a and 6b show two forms of matrix P. Figure 6a shows the 2D data flow from DDR memory to SRAM and then to the convolution kernel (core), while Figure 6b shows the 3D data flow from DDR memory to SRAM and then to the convolution kernel (core).

[0042] The left side of Figure 6a shows the 2D format of matrix P stored in DDR memory. in_x=outZ, Tile_x=B (the Tile_x of the last Tile in the x-direction may be smaller than the Tile_x of other Tiles), in_y=kZ, Tile_y=A (the Tile_y of the last Tile in the y-direction may not be smaller than the Tile_y of other Tiles). The arrows indicate the sliding direction of the Tiles, i.e., from top to bottom, and then from left to right. The center of Figure 6a shows the format of matrix P cached in SRAM, which is the same as the 2D format in DDR memory. The arrows indicate the sliding direction of the Tiles, i.e., read from top to bottom, and then from left to right. The right side of Figure 6a shows the weight data received by each convolution kernel, with a size of Kpc × A. It is necessary to transpose this to A × Kpc when performing the convolution operation.

[0043] The left side of Figure 6b shows the 3D form of matrix P stored in DDR memory. in_x=outZ, Tile_x=B (the Tile_x of the last Tile in the x direction may be smaller than the Tile_x of other Tiles), in_y=kZ / A, Tile_y=1, in_z=B. The arrows indicate the sliding direction of the Tiles, i.e., from top to bottom, and then from left to right. The center of Figure 6b shows the form of matrix P cached in SRAM during processing, which is the same as the 2D form of matrix P × matrix Q. The arrows indicate the sliding direction of the Tiles, i.e., from left to right, and then from top to bottom. The right side of Figure 6b shows the weight data received by each convolution kernel, with a size of A × Kpc. No additional transposition is required when performing the convolution operation.

[0044] When reading weight blocks, the memory subsystem reads them sequentially from DDR memory or SRAM memory according to the principles shown in Figures 5, 6a, and 6b. When reading weight blocks, the memory subsystem prioritizes reading from SRAM memory, and if there are no corresponding weight blocks in SRAM memory, it reads from DDR memory. The weight blocks read from DDR need to be stored in SRAM for caching. In any embodiment, the SRAM memory may be internal memory of the AI ​​chip, and the DDR memory may be external memory of the AI ​​chip.

[0045] In one embodiment, the memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module. The first image data loading module is electrically connected to memory and N convolution kernels, respectively; the second image data loading module is electrically connected to memory and a weight data processing module, respectively; and the weight data processing module is electrically connected to N convolution kernels, respectively.

[0046] The first image data loading module is configured to read input image data from memory and distribute the input image data to each convolution kernel. The second image data loading module is configured to read each weight block from memory and send each read weight block to the weight data processing module. The weight data processing module is electrically connected to the second image data loading module and to N convolution kernels, and is configured to divide each weight block into N weight data and distribute them to the N convolution kernels.

[0047] In the embodiment of this application, when performing a GEMM operation, the weight matrix P is requested using a second image data loading module, rather than requesting kernel data using a conventional weight data requesting means. Since the second image data loading module reads data in 3DTile format, each time weight data is read, only a portion of the entire weight, i.e., one weight block, is read. This eliminates the need for inefficient data rearrangement of weight data by an external module, and reduces the number of cycles in the GEMM operation.

[0048] Specifically, the weight data processing module is configured to either divide each A×B weight block into N A×Kpc weight data and distribute them to N convolution kernels, or to divide each B×A weight block into N Kpc×A weight data and distribute them to N convolution kernels.

[0049] In one embodiment, the weight data processing module is configured to divide each A×B weight block into N A×Kpc weight data blocks and includes N registers. The N registers correspond one-to-one with each of the N convolution kernels and are all electrically connected to a second image data loading module. Each register is configured to store one A×Kpc weight data block. Here, the registers are variable length writers (VLMs) that can receive weight data of variable effective length, store the effective data until it reaches the output width, and then transmit it to the corresponding convolution kernel.

[0050] To ensure performance, the maximum input width of the VLM is set to min(sram_data_width A × max_kpc × bpp), and the output width is set to max_kpc × bpp. The parameter sram_data_width represents the width of the SRAM, and bpp represents the number of bytes for each weight element. Here, it is necessary to store the weight blocks read from the DDR in the SRAM for caching.

[0051] In another embodiment, the weight data processing module is configured to divide each B×A weight block into N Kpc×A weight data. As shown in Figure 7, the weight data processing module includes N extractors, N logical shifters, and N selectors. Each extractor and each logical shifter is connected to a second image data loading module, and one selector is connected to one extractor and one logical shifter. All N extractors are electrically connected to the second image data loading module, and when Kpc=1, the second image data loading module transmits weight block data to the extractors. Each extractor is configured to extract one Kpc×A weight data from one weight block in one clock cycle. In this embodiment, the second image data loading module can transmit A weight data of B to the extractors in one clock cycle.

[0052] When Kpc=1, adjacent weight elements belong to different convolution kernels, and the valid data transmitted in each cycle contains A B arrays (row arrays). Assuming Kpc=1 and the number of convolution kernels N=24, B=Kpc×N=1×24, so A=9. Assuming each weight element occupies 1 byte, the data width transmitted in each cycle is 216 bytes. In this case, Byte0, Byte24, Byte48, Byte72, Byte96, Byte120, Byte144, Byte168, Byte196 belong to core1, Byte1, Byte25, Byte49, Byte73, Byte97, Byte121, Byte145, Byte169, Byte197 belong to core2, and so on. In this embodiment, A B arrays can be transmitted in one clock cycle.

[0053] Each of the N logical shifters is electrically connected to a second image data loading module, and if Kpc > 1, the second image data loading module transmits weight block data to the logical shifters. Each shifter is configured to extract one Kpc × A size weight data from one weight block in A clock cycles. In this embodiment, the second image data loading module transmits only one B weight data to the logical shifter in one clock cycle. Thus, one B × A weight block is transmitted in A clock cycles.

[0054] If Kpc > 1, a maximum of one B array can be transmitted per cycle. Assuming Kpc = 2, the number of convolution kernels N = 24, and each weight element occupies 1 byte, then Kpc × N = 2 × 24 = 48, and the data width transmitted per cycle is 48 bytes. In this case, Byte0 and Byte1 belong to core1, Byte2 and Byte3 belong to core2, ... Byte46 and Byte47 belong to core24. Therefore, logical shifter 1 is responsible for the output of Byte0 and Byte1, logical shifter 2 is responsible for the output of Byte2 and Byte3, ..., logical shifter 24 is responsible for the output of Byte46 and Byte47.

[0055] N selectors correspond one-to-one with each of the N convolution kernels, and each selector is electrically connected to one extractor, one logic shifter, and one convolution kernel, and is configured to send the output of the extractor to the corresponding convolution kernel when Kpc=1, and to send the output of the logic shifter to the corresponding convolution kernel when Kpc>1.

[0056] Note that when the size of the weight block is B × A and Kpc = 1, the weight data processing module can contain only N extractors (excluding N logical shifters and N selectors). In this case, the N extractors are directly connected to N convolution kernels. Similarly, when the size of the weight block is B × A and Kpc > 1, the weight data processing module can contain only N logical shifters (excluding N extractors and N selectors). In this case, the N logical shifters are directly connected to N convolution kernels. The structure shown in Figure 7 is a weight data processing module structure that can accommodate various Kpc values.

[0057] Furthermore, if the weight block size is B×A and Kpc>1, each B×A weight block can be divided into N Kpc×A weight data using an extractor. In this case, the size of Kpc varies depending on the different cases, and the allocation of Kpc per convolution kernel to the edges of matrix P may be uneven, which can lead to many variations in the design of the division and rearrangement. Therefore, to reduce the area overhead caused by these variations, when Kpc>1, a logical shifter is used.

[0058] In another embodiment, as shown in Figure 8, the weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors. The second image data loading module does not simultaneously send weight blocks to at least two of the extractors, logical shifters, and registers. If the size of the weight block is A × B, the second image data loading module sends the weight block to N registers via path0; if the size of the weight block is B × A and Kpc = 1, it sends the weight block to N extractors; and if the size of the weight block is B × A and Kpc > 1, it sends the weight block to N logical shifters.

[0059] N registers correspond one-to-one with each of the N convolution kernels and are all electrically connected to the second image data loading module. Each register is configured to store one A × Kpc size weight data.

[0060] Each of the N extractors is electrically connected to the second image data loading module. Each extractor is configured to extract one Kpc × A size weight data from one weight block in one clock cycle.

[0061] Each of the N logical shifters is electrically connected to the second image data loading module. Each shifter is configured to extract one Kpc × A size weight data from one weight block in A clock cycles.

[0062] Each of the first selectors is electrically connected to one extractor and one logical shifter, and is configured to select and output the output data of the corresponding extractor when Kpc=1, and to select and output the output data of the corresponding logical shifter when Kpc>1.

[0063] Each of the second selectors is electrically connected to one first selector, one register, and one convolution kernel, and is configured to select and output the output data of the corresponding register when the size of the weight block is A × B (or the size of the weight data is A × Kpc), and to select and output the output data of the corresponding first selector when the size of the weight block is B × A (or the size of the weight data is Kpc × A).

[0064] Each convolution kernel is configured to perform a convolution operation on received weight data with input image data. Specifically, each convolution kernel is configured to perform a convolution operation on the received weight data with input image data when it receives weight data of size A × Kpc, and to convert the Kpc × A weight data to A × Kpc weight data before performing a convolution operation with input image data when it receives weight data of size Kpc × A.

[0065] Here, the convolution results of each convolution kernel for the same weight block are added according to their corresponding positions to obtain the convolution result for each weight block. The final convolution result is obtained by adding the convolution results of each weight block according to their corresponding positions.

[0066] In some possible embodiments, the AI ​​chip may further include a post-processing module. The post-processing module is connected to each convolution kernel and to memory, and is configured to obtain the convolution result for each weight block by adding the convolution results of each convolution kernel for the same weight block according to their corresponding positions, and to obtain the final convolution result by adding the convolution results of each weight block according to their corresponding positions.

[0067] Here, the AI ​​chip is an integrated circuit chip having signal processing capabilities. The AI ​​chip may be a general-purpose processor including a Central Processing Unit (CPU), Network Processor (NP), microprocessor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and may implement or execute each of the methods, steps and logic blocks disclosed in the embodiments of this application. Alternatively, the AI ​​chip may be any conventional processor, etc.

[0068] This AI chip could be a dedicated computational acceleration chip (also called an accelerator) designed to handle massive computational tasks, such as a Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), or neural network processor, or of course, a processor for other AI computational tasks.

[0069] Embodiments of this application further provide an electronic device which includes a memory and the above-mentioned AI chip electrically connected to the memory. The memory stores data necessary for GEMM operations, such as input image data and weight data.

[0070] This electronic device includes, but is not limited to, tablets, laptops, in-car devices, and servers.

[0071] The embodiments of this application further provide a convolution operation method that is applicable to the AI ​​chip described above. The convolution operation method according to the embodiments of this application will be described below with reference to Figure 9.

[0072] S1: Read the input image data, read each weight block, and divide each weight block into N weight data.

[0073] The first image data loading module reads the input image data from memory, the second image data loading module reads each weight block from memory, and the weight data processing module divides each weight block into N weight data. For example, the second image data loading module sends each read weight block to the weight data processing module, which divides each weight block into N weight data. Here, each weight block is part of the whole weight.

[0074] The first image data loading module, upon reading the input image data, distributes the input image data to N convolution kernels, where N is an integer greater than or equal to 2.

[0075] The weight data processing module divides each weight block into N weight data points, then distributes these N weight data points to N convolution kernels, with each convolution kernel corresponding to one weight data point.

[0076] The weight data processing module can divide each weight block into N weight data by either dividing each A×B weight block into N A×Kpc weight data, or dividing each B×A weight block into N Kpc×A weight data. Here, A, B, and Kpc are all positive integers, where B is the maximum number of weight elements processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight elements processed by each convolution kernel in the second direction in each convolution.

[0077] S2: A convolution operation is performed between each weight data and the input image data.

[0078] The convolution kernel in the AI ​​chip performs a convolution operation between each of the received weight data and the input image data.

[0079] S3: Add the convolution results of each weight data belonging to the same weight block to obtain the convolution result for each weight block.

[0080] The post-processing module adds the convolution results of each weight data belonging to the same weight block to obtain the convolution result for each weight block.

[0081] S4: The convolution results of each weight block are added together to obtain the final convolution result.

[0082] The post-processing module adds up the convolution results of each weight block to obtain the final convolution result.

[0083] The convolution operation method according to the embodiment of this application has the same implementation principle and technical effect as the embodiment of the AI ​​chip described above. For the sake of brevity, parts of the method not mentioned in the embodiment of the method will be referred to in the corresponding content of the embodiment of the AI ​​chip described above.

[0084] Embodiments of this application provide a non-volatile computer-readable storage medium (hereinafter referred to as the storage medium) which stores a computer program that, when executed on the electronic device in a computer, performs the convolution operation method described above.

[0085] In this specification, each example is described progressively, with emphasis on explaining the differences between each example and the others. Similar or identical parts between the examples should be referred to alternately.

[0086] In some embodiments of this application, the described apparatus and methods can also be implemented in other ways. The embodiments of the apparatus described above are illustrative only, and for example, flowcharts and block diagrams in the drawings illustrate implementable architectures, functions and operations based on the apparatus, methods and computer program products according to some embodiments of the present invention, where each block in the flowchart or block diagram can represent a module, program segment or part of code containing one or more executable instructions capable of implementing a predetermined logical function. In an interchangeable implementation, the implementation of the functions described in the blocks may differ from the order shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or in some cases in reverse order. This is determined by the required functions. Each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated, hardware-based system that performs a predetermined function or operation, or by a combination of dedicated hardware and computer instructions.

[0087] Furthermore, each functional module in each embodiment of this application may be a single independent part formed by integration, a single module, or a single independent part formed by integration of two or more modules.

[0088] The aforementioned functions can be implemented in the form of a software function unit, which, when sold or used as an independent product, can be stored on a computer-readable storage medium. Under this understanding, the technical proposal of this application, or any part thereof that contributes to the prior art, or any part thereof, can be implemented in the form of a software product. This computer software product is stored on a computer-readable storage medium and includes a plurality of commands for a computer device (such as a personal computer, laptop computer, server, or electronic device) to perform all or part of the steps of the above-described method in each embodiment of this application. The computer-readable storage medium includes various media capable of storing program code, such as USB disks, portable hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The above describes only specific embodiments of the present application, and the scope of protection of this application is not limited thereto. A person skilled in the art can modify or substitute the technical invention within the scope of the technical invention disclosed herein, and such modifications or substitutions are also included within the scope of protection of this application. Therefore, the scope of protection of this application is as set forth in the claims.

Claims

1. It includes N convolution kernels and a memory subsystem, The aforementioned N convolution kernels are such that N is an integer greater than or equal to 2. The memory subsystem is electrically connected to the N convolution kernels and is configured to read input image data from memory and distribute it to each of the convolution kernels, read each weight block from memory, divide each weight block into N weight data and distribute them to the N convolution kernels, where each convolution kernel corresponds to one weight data, and each weight block is a part of the whole weight. Each convolution kernel performs a convolution operation between the received weight data and the input image data, and the convolution results of each convolution kernel for the same weight block are added together to obtain the convolution result for each weight block. It is configured to obtain the final convolution result by adding the convolution results of the following operations. The memory subsystem is configured to either divide each A × B weight block into N A × Kpc weight data, or divide each B × A weight block into N Kpc × A weight data, where A, B, and Kpc are all positive integers, B is the maximum number of weight elements processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight elements processed by each convolution kernel in the second direction in each convolution. An AI chip characterized by the following features.

2. The memory subsystem includes a first image data loading module, a second image data loading module, and a weight data processing module. The first image data loading module is configured to read the input image data from memory and distribute the input image data to each of the convolution kernels. The second image data loading module is configured to read each weight block from memory, The weight data processing module is electrically connected to the second image data loading module and the N convolution kernels, and is configured to either divide each A x B weight block into N A x Kpc weight data and distribute them to the N convolution kernels, or divide each B x A weight block into N Kpc x A weight data and distribute them to the N convolution kernels. The AI ​​chip according to feature 1.

3. If the weight data processing module is configured to divide each A × B size weight block into N A × Kpc size weight data, the weight data processing module includes N registers, The N registers correspond one-to-one with each of the N convolution kernels and are all electrically connected to the second image data loading module, and each register is configured to store one A × Kpc size weight data. The AI ​​chip according to feature 2.

4. The aforementioned weight data processing module includes N extractors, The N extractors are electrically connected to the second image data loading module. Each of the extractors is configured to extract one Kpc × A size weight data from one weight block in one clock cycle, provided that the weight block size is B × A and Kpc = 1. The AI ​​chip according to feature 2.

5. The aforementioned weight data processing module includes N logical shifters, The N logical shifters are electrically connected to the second image data loading module. Each of the aforementioned logic shifters is configured to extract one Kpc × A size weight data from one weight block in A clock cycles, provided that the size of the weight block is B × A and Kpc > 1. The AI ​​chip according to feature 2.

6. If the weight data processing module is configured to divide each B × A size weight block into N Kpc × A size weight data, then the weight data processing module includes N extractors, N logical shifters, and N selectors. The N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one Kpc × A size weight data from one weight block in one clock cycle. The N logical shifters are electrically connected to the second image data loading module, and each of the logical shifters is configured to extract one Kpc × A size weight data from one weight block in A clock cycles. The N selectors correspond one-to-one with each of the N convolution kernels, and each of the selectors is electrically connected to one extractor, one logic shifter, and one convolution kernel, and is configured to transmit the output of the extractor to the corresponding convolution kernel when Kpc = 1, and to transmit the output of the logic shifter to the corresponding convolution kernel when Kpc > 1. The AI ​​chip according to feature 2.

7. The weight data processing module includes N registers, N extractors, N logical shifters, N first selectors, and N second selectors. The N registers correspond one-to-one with each of the N convolution kernels and are all electrically connected to the second image data loading module, and each register is configured to store one A × Kpc size weight data. The N extractors are electrically connected to the second image data loading module, and each of the extractors is configured to extract one Kpc × A size weight data from one weight block in one clock cycle. The N logical shifters are electrically connected to the second image data loading module, and each of the logical shifters is configured to extract one Kpc × A size weight data from one weight block in A clock cycles. Each of the N first selectors is electrically connected to one of the extractors and one of the logic shifters, and is configured to select and output the output data of the corresponding extractor when Kpc = 1, and to select and output the output data of the corresponding logic shifter when Kpc > 1. Each of the N second selectors is electrically connected to one of the first selectors, one of the registers, and one of the convolution kernels, and is configured to select and output the output data of the corresponding register when the size of the weight block is A × B, and to select and output the output data of the corresponding first selector when the size of the weight block is B × A. The AI ​​chip according to feature 2.

8. Each convolution kernel is further configured to receive weight data of size Kpc × A, convert that Kpc × A weight data to A × Kpc weight data, and then perform the convolution operation with the input image data. The AI ​​chip according to any one of claims 1 to 7.

9. The invention comprises a memory and an AI chip according to any one of claims 1 to 7, wherein the AI ​​chip is electrically connected to the memory. An electronic device characterized by the following features.

10. The system includes a memory and an AI chip as described in claim 8, wherein the AI ​​chip is electrically connected to the memory. An electronic device characterized by the following features.

11. A convolution operation method, which is performed by an AI chip, The steps include reading the input image data, reading each weight block which is part of the whole weight, and dividing each weight block into N weight data, The steps include performing a convolution operation between each weight data and the input image data, The steps include adding the convolution results of each weight data belonging to the same weight block to obtain the convolution result for each weight block, The process includes the step of adding the convolution results of each weight block to obtain the final convolution result, In the step of dividing each weight block into N weight data, each A × B weight block is divided into N A × Kpc weight data, or each B × A weight block is divided into N Kpc × A weight data, where A, B, and Kpc are all positive integers, B is the maximum number of weight elements processed by all convolution kernels in each convolution, A is the maximum height of all convolution kernels in the first direction, and Kpc is the maximum number of weight elements processed by each convolution kernel in the second direction in each convolution. A convolution operation method characterized by the following.

Images