Convolution calculation method, system, device and storage medium
By chunking the input images by blocking the size of the unit memory block and the convolution kernel, and depositing them row by row into the unit memory block by shifting and intercepting a fixed length for matrix multiplication, the existing convolution calculation methods are solved in the memory space and bandwidth. The requirements are insufficient, efficient convolutional calculation is realized.
Patent Information
- Application Number
- CN202110997622.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-27
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-08-27
AI Technical Summary
The existing convolutional calculation methods have shortcomings in storage space and bandwidth requirements, and it is difficult to efficiently support convolution operations in deep learning, resulting in high computing efficiency and cost.
By blocking the input images by blocking the size of the unit memory block and the convolution kernel, and depositing them into the unit memory block row by row by row by shifting and intercepting a fixed length, matrix multiplication is performed to achieve convolutional calculations, saving storage and bandwidth costs.
It effectively reduces the demand for storage space and transmission bandwidth, improves computing efficiency, and reduces storage and transmission costs.
Smart Images
Figure CN113870091B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of convolution computing technology, and in particular to a convolution computing method, system, device and storage medium. Background Art
[0002] Convolution operation is the most important operation in deep learning. Convolutional networks have pushed deep learning to the forefront of almost all machine learning tasks. How to efficiently support convolution operations plays a vital role in the operation of neural network models, including the absolute running time of the model, model latency, throughput, power consumption, cost, off-chip bandwidth requirements and other aspects.
[0003] Since convolution operations usually expand the multi-channel input image and the corresponding convolution kernel into two-dimensional matrices, and obtain the calculation results by matrix multiplication, due to hardware limitations, in order to improve the computing energy efficiency, it is necessary to reduce the demand for storage space and storage transfer bandwidth. Summary of the invention
[0004] Based on this, it is necessary to provide a convolution calculation method, system, device and storage medium to save storage space and reduce bandwidth requirements.
[0005] In order to achieve the purpose of this application, this application adopts the following technical solutions:
[0006] A convolution calculation method, comprising:
[0007] Dividing each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed; wherein the maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel;
[0008] Each sub-block to be processed is stored in a unit storage block; wherein, for each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel by shifting and intercepting a fixed length, and stored in the unit storage block row by row;
[0009] For each sub-block to be processed, the data to be convolved is read from at least one of the unit storage blocks within the limit of a single calculation capacity and a matrix multiplication operation is performed to sequentially obtain the portion of each row of the output image corresponding to the sub-block to be processed.
[0010] A convolution computing system, comprising:
[0011] A block division module, used for dividing each channel of the input image in the same way according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel, to obtain a sub-block to be processed; wherein the maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel;
[0012] A sub-block storage module is used to store each sub-block to be processed in a unit storage block; wherein, for each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel in a manner of shifting and intercepting a fixed length, and stored in the unit storage block row by row;
[0013] A computing module is used to read the data to be convolved from at least one of the unit storage blocks and perform matrix multiplication operations for each sub-block to be processed, within the limit of a single computing capacity, so as to sequentially obtain the part of each row of the output image corresponding to the sub-block to be processed.
[0014] A convolution computing device comprises a memory, a processor and a convolution computing program stored in the memory and executable on the processor, wherein the convolution computing program implements the steps of the convolution computing method described above when executed by the processor.
[0015] A computer-readable storage medium stores a convolution calculation program, which implements the steps of the convolution calculation method described above when executed by a processor.
[0016] The above-mentioned convolution calculation method, system, device and computer-readable storage medium divide the input image into blocks, and then each sub-block to be processed is shifted and truncated to a fixed length and stored in a unit storage block row by row, and the calculation of the blocks is gradually completed in units of a single calculation capacity, thereby completing the convolution calculation of the entire input image. The storage method after shifting and truncating by a fixed length and the subsequent calculation method in the above steps can also save storage costs and transmission bandwidth costs because a large amount of data can be reused. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A schematic diagram of the structure of a convolution computing device in the hardware operating environment involved in the embodiment of the present application;
[0018] Figure 2a This is the convolution calculation principle diagram;
[0019] Figure 2b Schematic diagram for multi-channel convolution calculation;
[0020] Figure 2c A schematic diagram of converting a convolution kernel into a two-dimensional matrix;
[0021] Figure 2d A schematic diagram for converting an input image into a two-dimensional matrix;
[0022] Figure 2e A schematic diagram of obtaining the result of a convolution operation through two-dimensional matrix multiplication;
[0023] Figure 3a A flow chart of a convolution calculation method according to an embodiment;
[0024] Figure 3b A structural diagram of a convolution computing device according to an embodiment;
[0025] Figure 4a It is a 7×17 image pixel distribution map;
[0026] Figure 4b for Figure 4a The pixel distribution map of the feature image obtained by 3×3 convolution operation on the image;
[0027] Figure 4c For the general Figure 4a Schematic diagram of storing the input of 2 channels of the 7×10 sub-block into the unit storage block;
[0028] Figure 4d For Figure 4c Part of the data used to calculate the first row of the output image is read from the data;
[0029] Figure 4e Schematic diagram of the process of calculating each pixel of the output image through a sliding window;
[0030] Figure 4f The embodiment of the present application is converted into a matrix multiplication operation to obtain Figure 4e Schematic diagram of the same results;
[0031] Figure 4g For Figure 4c Part of the data used to calculate the second row of the output image is read from the data;
[0032] Figure 5 A module diagram of a convolution computing system according to an embodiment. DETAILED DESCRIPTION
[0033] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present application are given in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.
[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0035] Figure 1 It is a structural diagram of a convolution computing device 100 of a hardware operating environment involved in an embodiment of the present application.
[0036] The convolution computing device of the embodiment of the present application can be, for example, a server, a personal computer, a smart phone, a tablet computer, a portable computer, etc. As long as it has a certain general data processing capability.
[0037] like Figure 1 As shown, the convolution computing device 100 includes: a memory 104, a processor 102 and a network interface 106.
[0038] In some embodiments, the processor 102 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, configured to run program codes stored in the memory 104 or process data, such as executing a convolution calculation program.
[0039] The memory 104 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 104 may be an internal storage unit of the convolution computing device 100, such as a hard disk of the convolution computing device 100. In other embodiments, the memory 104 may also be an external storage device of the convolution computing device 100, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the convolution computing device 100.
[0040] Furthermore, the memory 104 may also include an internal storage unit of the convolution computing device 100. The memory 104 may be used not only to store application software and various data installed in the convolution computing device 100, such as codes for training a face recognition model, but also to temporarily store data that has been output or is to be output.
[0041] The network interface 106 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface), which is typically used to establish a communication connection between the convolution computing device 100 and other electronic devices.
[0042] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth (Blue Tooth) communication protocol or a combination thereof.
[0043] Figure 1 Only the convolution computing device 100 having components 102-106 is shown, and those skilled in the art will appreciate that Figure 1 The structure shown does not constitute a limitation on the convolution computing device 100, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0044] In the field of machine learning, especially in the field of convolutional neural networks, convolution operations are often involved. Figure 2a The convolution operation uses the convolution kernel 202 to perform sliding window calculations on the input image 204 one by one to extract features in the input image to obtain the output image 206. Figure 2aIn the example, the size of the input image 204 is 8×8, and the size of the convolution kernel 202 is 2×2. In order to calculate the first value of the output image 206, the convolution kernel 202 is convolved with the 4 pixels in the upper left corner of the input image 204, that is, the value of each pixel is multiplied by the value of the corresponding position of the convolution kernel 202 and then the 4 products are added, 2×0+5×1+7×1+4×0=12, and the value of the first pixel in the upper left corner of the output image 206 is obtained. In order to obtain the value of the pixel in the first row and second column of the output image 206, the position corresponding to the convolution kernel 202 and the input image 204 is moved to the right by one grid, and then calculated in the same way as above. By gradually moving to the right and calculating in the area corresponding to the convolution kernel 202 with a step size of 1, all values of the first row on the output image 206 can be obtained. It can be understood that when the area corresponding to the convolution kernel 202 moves to the 4 pixels in the upper right corner, the value of the first row and last column of the input image 204 is calculated. In the same way, the area corresponding to the convolution kernel 202 is shifted down by one row, and all the values of the second row on the output image 206 can be calculated. It can be seen that when the input image 204 is 8×8 and the convolution kernel 202 is 2×2, the output image 206 is 7×7. When different convolution kernel 202 sizes or sliding steps are set, the size of the output image 206 will also change. In addition, the output image 206 can also be pooled for further compression.
[0045] The input image 204 is usually multi-channel, for example, the original image is generally RGB three-channel, and the feature image in the middle can be set with different numbers of channels according to the situation. The above convolution process is for the convolution operation of one channel, and the convolution operation method of each channel is the same. For each channel, the convolution kernel can be the same or different.
[0046] like Figure 2b As shown, the result of the operation of a multi-channel (the number of channels is C) input image 208 and a group of convolution kernels (i.e., the convolution kernels with the same number of channels together form a filter 210) is an output image of one channel, and multiple filters (the number is Cout) output a multi-channel output image 212, and the size of each channel image in the output image 212 is H'×W'. The size of the output image 212 varies depending on the size of the convolution kernel, the sliding step, whether pooling is performed, and the pooling method.
[0047] When performing convolution operations in a computer, matrix multiplication is used. This calculation method requires converting the input image and convolution kernel into a two-dimensional matrix for matrix multiplication. Figure 2cAs shown, there are Cout filters, each filter has C channels corresponding to the input image, and each channel has a size of K×K. It is converted into a two-dimensional matrix of size Cout×(C×K×K), that is, the height is Cout and the width is C×K×K. For each filter, one channel is expanded into one dimension by row, resulting in a row of K×K, and C channels are concatenated into a row of C×K×K. Cout filters form a Cout row.
[0048] like Figure 2d As shown, the input image has C channels, and the size of each channel is H × W. Convert it into a two-dimensional matrix of size (H'×W') × (C×K×K), that is, the height is (H'×W') and the width is (C×K×K).
[0049] Afterwards, if Figure 2e As shown, you can Figure 2d The two-dimensional matrix and Figure 2c The result of the convolution calculation is obtained by multiplying the transpose of the two-dimensional matrix in. That is:
[0050] ((H'×W')×(C×K×K))×(Cout×(C×K×K)) T =(H'×W')×Cout
[0051] However, the above calculation process is limited by memory size or computing power, and sometimes it is impossible to complete the multiplication of two matrices at one time. To this end, this application proposes a convolution calculation method, which can divide the calculation into multiple small parts and make full use of the data transmitted each time, saving bandwidth and improving calculation efficiency.
[0052] Figure 3a FIG. 1 is a flow chart of a convolution calculation method according to an embodiment. The method includes:
[0053] Step S302: Divide each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain sub-blocks to be processed.
[0054] like Figure 3b As shown, when performing a basic matrix multiplication operation, the weight data is read from the data storage area 302 by the weight reading unit 304 and then sent to the weight cache unit 306, and the feature data is read by the feature data reading unit 308 and then sent to the feature cache unit 310. The matrix calculation unit 312 reads the corresponding data from the weight cache unit 306 and the feature cache unit 310, and then performs a matrix multiplication operation, outputs the operation result to the output cache unit 314, and stores it in the data storage area 302 through the output data reading and writing unit 316.
[0055] The unit storage block may be a basic storage block in the memory, each storage block having 8×8 16-bit storage spaces, and 8 16-bits in each row constitute a bank. Each 16-bit storage space may store one pixel data of a feature image. The unit storage block is also a basic storage unit in the weight cache unit 306 and the feature cache unit 310.
[0056] The maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel. For example, when the size of the convolution kernel is 3×3, the width of the bank is 8 pixels, and the maximum width of the sub-block to be processed is 10 pixels.
[0057] Each channel is divided in the same way, which means that after one channel is divided in a certain way, the other channels should be divided in the same way. For example, for a 2-channel 11×16 input image, each channel is divided into 2 3×8 and 2 8×8 sub-blocks.
[0058] In one embodiment, the input image is an initial input image or an intermediate feature image. The initial input image is generally an RGB three-channel image, and the intermediate feature image is an image after convolution processing. That is, the method of the present application is applicable to any convolution calculation process.
[0059] Step S304: Each sub-block to be processed is stored in a unit storage block. For each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel by shifting and intercepting a fixed length, and stored in the unit storage block row by row.
[0060] For example, when the size of the convolution kernel is 3×3 and the width of the sub-block to be processed is 10 pixels, each row of the sub-block to be processed can be divided into 3 rows by shifting and intercepting 8 pixels. Assuming that the first row of the sub-block to be processed is a0~a9, the intercepted 3 rows are a0~a7, a1~a8, and a2~a9 respectively.
[0061] For another example, when the size of the convolution kernel is 2×2 and the width of the sub-block to be processed is 9 pixels, each row of the sub-block to be processed can be divided into 2 rows by shifting and intercepting 8 pixels. Assuming that the first row of the sub-block to be processed is a0~a8, the intercepted 2 rows are a0~a7 and a1~a8 respectively.
[0062] For another example, when the size of the convolution kernel is 2×2, and the width of the sub-block to be processed is 8 pixels, each row of the sub-block to be processed can be divided into 2 rows by shifting and intercepting 7 pixels. Assuming that the first row of the sub-block to be processed is a0~a7, the intercepted 2 rows are a0~a6 and a1~a7 respectively. At this time, each intercepted row cannot completely fill a row of the unit storage block. This happens when the divided sub-block is the last remaining sub-block.
[0063] When the data of a sub-block fills up a unit memory block bank, it continues to be stored in a new unit memory block bank.
[0064] Step S306: For each sub-block to be processed, the data to be convolved is read from at least one of the unit storage blocks within the limit of a single calculation capacity and a matrix multiplication operation is performed to sequentially obtain the portion of each row of the output image corresponding to the sub-block to be processed.
[0065] The single calculation capacity is to read 8×8 data in a unit storage block at one time, and the data to be convolution calculated can be obtained by multiple readings. After the data is multiplied by the weight matrix, a part of a row of the output image can be obtained. After the parts of multiple sub-blocks located in the same row are multiplied by the weight matrix, the other parts of the row of the output image are obtained, thereby forming a row of the output image. In one embodiment, the single calculation capacity is the amount of data that the processor can calculate per clock cycle; the processor is a single processor or a multi-processor.
[0066] In the above embodiment, the input image is divided into blocks, and each sub-block to be processed is shifted and cut into a fixed length and stored in a unit storage block row by row, and the calculation of the blocks is gradually completed in units of a single calculation capacity, thereby completing the convolution calculation of the entire input image. The storage method after shifting and cutting into a fixed length and the subsequent calculation method in the above step S304 can reuse a large amount of data, thereby saving storage costs and transmission bandwidth costs.
[0067] The following is an explanation through specific examples.
[0068] Assume that the size of each input feature value channel is 7×17, and the output image is 5×15 after convolution with a 3×3 convolution kernel, and the convolution step is 1. The input image is as follows Figure 4a As shown, the output image is Figure 4b shown.
[0069] Assume that the input image has 2 channels. Each channel of the input image is divided into 7×10 sub-blocks and 7×7 sub-blocks. To process the 7×10 sub-blocks ( Figure 4a As an example, combine Figure 3b, the data in the sub-block is transferred to the feature cache unit 310 through the feature data reading unit 308, and stored in the following format: Figure 4c As shown, in the bank with address addr0, the 0th / 1st / 2nd row is a0,0~a0,9 obtained by sliding window. The feature data reading unit 308 reads a0,0~a0,9 at one time, and then outputs it to the feature cache unit 310 through shifting, and then reads a1,0~a1,9 of the next row, and then outputs it to the 3rd to 5th rows of the bank with address addr0 in the feature cache unit 310 through shifting. After all rows are read and stored in the feature cache unit 310, the input feature value of the next channel is read.
[0070] To ensure that each matrix multiplication (matrix multiplication in each clock cycle) can read 8 rows of data in the bank without bank conflicts, in one embodiment, the step S304: storing each sub-block to be processed in a unit storage block may include:
[0071] When the data storage of one channel of the input image is completed, if the remaining space of the last unit storage block used by the data of the one channel meets the preset conditions, the data of the next channel is stored starting from the remaining space of the last unit storage block; otherwise, the remaining space that meets the preset conditions is created in a new unit storage block, and the data of the next channel is stored starting from the remaining space of the new unit storage block.
[0072] In this embodiment, the preset condition is that the remaining space of the last unit storage block used for the data of one channel includes the row: (C×K×K)%8, that is, the remainder of the product of the number of channels C and the height and width of the convolution kernel divided by the width of the bank 8. In this embodiment, the number of channels is 2 and the convolution kernel is 3×3, then the remainder is 2, so the first row of the second channel stored in the feature cache unit 310 is bank1, and the address is the address after the first channel. If the first channel is stored in the last address of bank1 and subsequent banks in the feature cache unit 310, they are not occupied, the same address as the last address of the previous channel can be used directly. In this embodiment, the last bank stored in the last address of the first channel is bank4 (bank4 of addr2), so the second channel needs to start another address (addr3 after the first channel) to store the data of the first bank (bank1 of addr3). Figure 4c As shown, the data of the second channel is stored between bank1 of addr3 and bank5 of addr5.
[0073] After the feature cache unit 310 has finished storing, the matrix calculation unit 312 will spend 3 clock cycles to read in sequence: 1) addr0 bank0 to 7; 2) addr1 bank0, addr3 bank1 to bank7; 3) addr4 bank0 to 1, and then accumulate to obtain the final result of the 8 pixels in the first row of the output feature value channel. The data read once in 3 clock cycles is as follows: Figure 4d As shown in the figure, since the above matrix multiplication operation is based on the multiplication of two 8×8 matrices, 8 channels and 8 pixels in each channel will be obtained at the same time.
[0074] Please refer to Figure 4e According to the convolution kernel sliding window calculation principle, each pixel of the output image is the sum of the product of the sliding window and the corresponding pixel area. For example:
[0075] c0,0=b1*a0,0+b2*a0,1+b3*a0,2+b4*a1,0+b5*a1,1+b6*a1,2+b7*a2,0+b8*a2,1+b9*a2,2
[0076] c0,1=b1*a0,1+b2*a0,2+b3*a0,3+b4*a1,1+b5*a1,2+b6*a1,3+b7*a2,1+b8*a2,2+b9*a2,2
[0077] When there are multiple channels, c0,0 is the sum of the convolution operation results of the convolution kernels of the same pixel area in multiple channels.
[0078] In the method of this embodiment, the above operation can be converted into a matrix multiplication operation. Figure 4f The process shown.
[0079] Similarly, to calculate the 8 pixels of the second row of the output image, the matrix calculation unit 312 reads in sequence: 1) addr0 bank3-7, addr1 bank0-2; 2) addr1 bank3, addr3 bank4-bank7, addr4 0-2, 3) addr4 bank3-4. The data read once in 3 clock cycles is as follows: Figure 4g shown.
[0080] It can be seen from the above embodiments that: 1) in the process of sliding the window, each sliding window requires a total of 9 banks of data, and 6 banks of data are reused between two sliding windows, which greatly saves the storage demand of the feature cache unit 310, and the feature cache unit 310 saves 6 / 9=66%; 2) when transferring from the data storage area 302 to the feature cache unit 310, one data is transferred once and reused 9 times, and the bandwidth is saved to 1 / 9 of the original bandwidth.
[0081] In one of the embodiments, the step S302: dividing each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain sub-blocks to be processed can be: dividing into blocks with the maximum width until the width of the remaining sub-blocks is less than the maximum width or is just divided with the maximum width.
[0082] For example, if the convolution kernel is 3×3 and the bank width is 8, the maximum width is 10. If the width of the input image is 33, it is divided into 3 sub-blocks with a width of 10 and 1 sub-block with a width of 3. The width of the remaining sub-block is 3 and cannot be further divided. If the width of the input image is 40, it is divided into 4 sub-blocks with a width of 10, which is just enough. The height of the sub-block is not limited.
[0083] When the 7×17 input image is divided into blocks, the maximum width is 10, so that 7×10 sub-blocks and 7×7 sub-blocks are obtained. This division method can maximize the use of storage space.
[0084] In one of the embodiments, the step S302: dividing each channel of the input image in the same way according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed, can be: dividing the blocks with a width greater than the width of the convolution kernel and not exceeding the maximum width, so that the widths of any two sub-blocks to be processed are the same, or the widths differ by no more than 2 pixels.
[0085] For example, if the convolution kernel is 3×3 and the bank width is 8, the maximum width is 10 and the minimum width is 4. If the width of the input image is 33, it can be divided into 3 sub-blocks with a width of 8 and 1 sub-block with a width of 9. If the width of the input image is 35, it can be divided into 5 sub-blocks with a width of 7. The height of the sub-block is not limited. This division method can make the size of the processed sub-blocks close, which is convenient for designing the calculation matrix.
[0086] In addition, based on the same inventive concept, a convolution computing system is provided. Figure 5 As shown, the convolution computing system 500 includes:
[0087] The blocking module 502 is used to divide each channel of the input image in the same way according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed; wherein the maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel.
[0088] like Figure 3b As shown, when performing a basic matrix multiplication operation, the weight data is read from the data storage area 302 by the weight reading unit 304 and then sent to the weight cache unit 306, and the feature data is read by the feature data reading unit 308 and then sent to the feature cache unit 310. The matrix calculation unit 312 reads the corresponding data from the weight cache unit 306 and the feature cache unit 310, and then performs a matrix multiplication operation, outputs the operation result to the output cache unit 314, and stores it in the data storage area 302 through the output data reading and writing unit 316.
[0089] The unit storage block may be a basic storage block in the memory, each storage block having 8×8 16-bit storage spaces, and 8 16-bits in each row constitute a bank. Each 16-bit storage space may store one pixel data of a feature image. The unit storage block is also a basic storage unit in the weight cache unit 306 and the feature cache unit 310.
[0090] The maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel. For example, when the size of the convolution kernel is 3×3, the width of the bank is 8 pixels, and the maximum width of the sub-block to be processed is 10 pixels.
[0091] Each channel is divided in the same way, which means that after one channel is divided in a certain way, the other channels should be divided in the same way. For example, for a 2-channel 11×16 input image, each channel is divided into 2 3×8 and 2 8×8 sub-blocks.
[0092] In one embodiment, the input image is an initial input image or an intermediate feature image. The initial input image is generally an RGB three-channel image, and the intermediate feature image is an image after convolution processing. That is, the method of the present application is applicable to any convolution calculation process.
[0093] The sub-block storage module 504 is used to store each sub-block to be processed in a unit storage block; wherein, for each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel by shifting and truncating the fixed length, and stored in the unit storage block row by row.
[0094] For example, when the size of the convolution kernel is 3×3 and the width of the sub-block to be processed is 10 pixels, each row of the sub-block to be processed can be divided into 3 rows by shifting and intercepting 8 pixels. Assuming that the first row of the sub-block to be processed is a0~a9, the intercepted 3 rows are a0~a7, a1~a8, and a2~a9 respectively.
[0095] For another example, when the size of the convolution kernel is 2×2 and the width of the sub-block to be processed is 9 pixels, each row of the sub-block to be processed can be divided into 2 rows by shifting and intercepting 8 pixels. Assuming that the first row of the sub-block to be processed is a0~a8, the intercepted 2 rows are a0~a7 and a1~a8 respectively.
[0096] For another example, when the size of the convolution kernel is 2×2, and the width of the sub-block to be processed is 8 pixels, each row of the sub-block to be processed can be divided into 2 rows by shifting and intercepting 7 pixels. Assuming that the first row of the sub-block to be processed is a0~a7, the intercepted 2 rows are a0~a6 and a1~a7 respectively. At this time, each intercepted row cannot completely fill a row of the unit storage block. This happens when the divided sub-block is the last remaining sub-block.
[0097] When the data of a sub-block fills up a unit memory block bank, it continues to be stored in a new unit memory block bank.
[0098] The computing module 506 is used to read the data to be convolved from at least one of the unit storage blocks and perform matrix multiplication operations for each sub-block to be processed, within the limit of a single computing capacity, so as to sequentially obtain the part of each row of the output image corresponding to the sub-block to be processed.
[0099] The single calculation capacity is to read 8×8 data in a unit storage block at one time, and the data to be convolution calculated can be obtained by multiple readings. After the data is multiplied by the weight matrix, a part of a row of the output image can be obtained. After the parts of multiple sub-blocks located in the same row are multiplied by the weight matrix, the other parts of the row of the output image are obtained, thereby forming a row of the output image. In one embodiment, the single calculation capacity is the amount of data that the processor can calculate per clock cycle; the processor is a single processor or a multi-processor.
[0100] The above embodiment divides the input image into blocks, and then each sub-block to be processed is shifted and cut into a fixed length and stored in a unit storage block row by row, and the calculation of the blocks is gradually completed in units of a single calculation capacity, thereby completing the convolution calculation of the entire input image. The above sub-blocks are stored in the module 504 and then shifted and cut into fixed lengths and then stored, and the subsequent calculation method can reuse a large amount of data, thereby saving storage costs and transmission bandwidth costs.
[0101] The following is an explanation through specific examples.
[0102] Assume that the size of each input feature value channel is 7×17, and the output image is 5×15 after convolution with a 3×3 convolution kernel, and the convolution step is 1. The input image is as follows Figure 4a As shown, the output image is Figure 4b shown.
[0103] Assume that the input image has 2 channels. Each channel of the input image is divided into 7×10 sub-blocks and 7×7 sub-blocks. To process the 7×10 sub-blocks ( Figure 4a As an example, combine Figure 3b , the data in the sub-block is transferred to the feature cache unit 310 through the feature data reading unit 308, and stored in the following format: Figure 4c As shown, in the bank with address addr0, the 0th / 1st / 2nd row is a0,0~a0,9 obtained by sliding window. The feature data reading unit 308 reads a0,0~a0,9 at one time, and then outputs it to the feature cache unit 310 through shifting, and then reads a1,0~a1,9 of the next row, and then outputs it to the 3rd to 5th rows of the bank with address addr0 in the feature cache unit 310 through shifting. After all rows are read and stored in the feature cache unit 310, the input feature value of the next channel is read.
[0104] To ensure that each matrix multiplication (matrix multiplication in each clock cycle) can read 8 rows of data in the bank without bank conflicts, in one embodiment, the step S304: storing each sub-block to be processed in a unit storage block may include:
[0105] When the data storage of one channel of the input image is completed, if the remaining space of the last unit storage block used by the data of the one channel meets the preset conditions, the data of the next channel is stored starting from the remaining space of the last unit storage block; otherwise, the remaining space that meets the preset conditions is created in a new unit storage block, and the data of the next channel is stored starting from the remaining space of the new unit storage block.
[0106] In this embodiment, the preset condition is that the remaining space of the last unit storage block used for the data of one channel includes the row: (C×K×K)%8, that is, the remainder of the product of the number of channels C and the height and width of the convolution kernel divided by the width of the bank 8. In this embodiment, the number of channels is 2 and the convolution kernel is 3×3, then the remainder is 2, so the first row of the second channel stored in the feature cache unit 310 is bank1, and the address is the address after the first channel. If the first channel is stored in the last address of bank1 and subsequent banks in the feature cache unit 310, they are not occupied, the same address as the last address of the previous channel can be used directly. In this embodiment, the last bank stored in the last address of the first channel is bank4 (bank4 of addr2), so the second channel needs to start another address (addr3 after the first channel) to store the data of the first bank (bank1 of addr3). Figure 4c As shown, the data of the second channel is stored between bank1 of addr3 and bank5 of addr5.
[0107] After the feature cache unit 310 has finished storing, the matrix calculation unit 312 will spend 3 clock cycles to read in sequence: 1) addr0 bank0 to 7; 2) addr1 bank0, addr3 bank1 to bank7; 3) addr4 bank0 to 1, and then accumulate to obtain the final result of the 8 pixels in the first row of the output feature value channel. The data read once in 3 clock cycles is as follows: Figure 4d As shown in the figure, since the above matrix multiplication operation is based on the multiplication of two 8×8 matrices, 8 channels and 8 pixels in each channel will be obtained at the same time.
[0108] Please refer to Figure 4e According to the convolution kernel sliding window calculation principle, each pixel of the output image is the sum of the product of the sliding window and the corresponding pixel area. For example:
[0109] c0,0=b1*a0,0+b2*a0,1+b3*a0,2+b4*a1,0+b5*a1,1+b6*a1,2+b7*a2,0+b8*a2,1+b9*a2,2
[0110] c0,1=b1*a0,1+b2*a0,2+b3*a0,3+b4*a1,1+b5*a1,2+b6*a1,3+b7*a2,1+b8*a2,2+b9*a2,2
[0111] When there are multiple channels, c0,0 is the sum of the convolution operation results of the convolution kernels of the same pixel area in multiple channels.
[0112] In the method of this embodiment, the above operation can be converted into a matrix multiplication operation. Figure 4f The process shown.
[0113] Similarly, to calculate the 8 pixels of the second row of the output image, the matrix calculation unit 312 reads in sequence: 1) addr0 bank3-7, addr1 bank0-2; 2) addr1 bank3, addr3 bank4-bank7, addr4 0-2, 3) addr4 bank3-4. The data read once in 3 clock cycles is as follows: Figure 4g shown.
[0114] It can be seen from the above embodiments that: 1) in the process of sliding the window, each sliding window requires a total of 9 banks of data, and 6 banks of data are reused between two sliding windows, which greatly saves the storage demand of the feature cache unit 310, and the feature cache unit 310 saves 6 / 9=66%; 2) when transferring from the data storage area 302 to the feature cache unit 310, one data is transferred once and reused 9 times, and the bandwidth is saved to 1 / 9 of the original bandwidth.
[0115] The block division module 502 is specifically used to divide the blocks according to the maximum width until the width of the remaining sub-blocks is smaller than the maximum width or is exactly divided according to the maximum width.
[0116] Or the blocking module 502 is specifically used to: block with a width greater than the width of the convolution kernel and not exceeding the maximum width, so that the widths of any two sub-blocks to be processed are the same, or the width difference does not exceed 2 pixels.
[0117] For example, if the convolution kernel is 3×3 and the bank width is 8, the maximum width is 10 and the minimum width is 4. If the width of the input image is 33, it can be divided into 3 sub-blocks with a width of 8 and 1 sub-block with a width of 9. If the width of the input image is 35, it can be divided into 5 sub-blocks with a width of 7. The height of the sub-block is not limited. This division method can make the size of the processed sub-blocks close, which is convenient for designing the calculation matrix.
[0118] The sub-block storage module 504 is specifically used for:
[0119] When the data storage of one channel of the input image is completed, if the remaining space of the last unit storage block used by the data of the one channel meets the preset condition, the data of the next channel is stored starting from the remaining space of the last unit storage block;
[0120] Otherwise, a remaining space satisfying the preset condition is created in a new unit storage block, and data of the next channel is stored starting from the remaining space of the new unit storage block.
[0121] The above modules are virtual device modules corresponding to the methods one by one, and their specific execution processes have been described in the method embodiments, which will not be repeated here. It can be understood that the contents described in the above method embodiments can be appropriately introduced into the system embodiments to support them.
[0122] In addition, an embodiment of the present application also proposes a computer-readable storage medium, on which the above-mentioned convolution calculation program is stored. When the convolution calculation program is executed by a processor, the steps of the convolution calculation method described above are implemented.
[0123] The specific implementation of the computer-readable storage medium of the present application is basically the same as the above-mentioned embodiments of the convolution calculation method, and will not be repeated here.
[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0129] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A convolution calculation method, comprising: Dividing each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed; wherein the maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel; Each sub-block to be processed is stored in a cache unit in a unit storage block; wherein, for each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel in a manner of shifting and intercepting a fixed length, and stored in the unit storage block row by row; For each sub-block to be processed, data to be convolved that is less than or equal to the single calculation capacity is read from at least one of the unit storage blocks each time and a matrix multiplication operation is performed to obtain the portion of each row of the output image corresponding to the sub-block to be processed.
2. The convolution calculation method according to claim 1, characterized in that: The input image is an initial input image or an intermediate feature image.
3. The convolution calculation method according to claim 1, characterized in that: The step of dividing each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed includes: The blocks are divided according to the maximum width until the width of the remaining sub-blocks is smaller than the maximum width or is exactly divided according to the maximum width.
4. The convolution calculation method according to claim 1, characterized in that: The step of dividing each channel of the input image in the same manner according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel to obtain a sub-block to be processed includes: The blocks are divided with a width greater than the width of the convolution kernel and not exceeding the maximum width, so that the widths of any two sub-blocks to be processed are the same, or the widths differ by no more than 2 pixels.
5. The convolution calculation method according to claim 1, characterized in that: The single computing capacity is the amount of data that can be calculated by the processor in each clock cycle; the processor is a single processor or a multi-processor.
6. The convolution calculation method according to claim 1, characterized in that: The step of storing each sub-block to be processed in a unit storage block comprises: When the data storage of one channel of the input image is completed, if the remaining space of the last unit storage block used by the data of the one channel meets the preset condition, the data of the next channel is stored starting from the remaining space of the last unit storage block; Otherwise, a remaining space satisfying the preset condition is created in a new unit storage block, and data of the next channel is stored starting from the remaining space of the new unit storage block.
7. The convolution calculation method according to claim 1, characterized in that: The unit storage block has 8×8 16-bit storage spaces, and each 16-bit storage space is used to store one pixel data of the input image.
8. A convolution computing system, characterized in that: include: A block division module, used for dividing each channel of the input image in the same way according to the size of the unit storage block in the cache unit of the input image and the size of the convolution kernel, to obtain a sub-block to be processed; wherein the maximum width of the sub-block to be processed is less than the sum of the width of the unit storage block and the width of the convolution kernel; A sub-block storage module is used to store each sub-block to be processed in a unit storage block; wherein, for each row of data of the sub-block to be processed, it is divided into the same number of rows as the height value of the convolution kernel in a manner of shifting and intercepting a fixed length, and stored in the unit storage block row by row; A computing module is used to read the data to be convolved from at least one of the unit storage blocks and perform matrix multiplication operations for each sub-block to be processed, within the limit of a single computing capacity, so as to sequentially obtain the part of each row of the output image corresponding to the sub-block to be processed.
9. A convolution computing device, characterized in that: It includes a memory, a processor, and a convolution calculation program stored in the memory and executable on the processor, wherein the convolution calculation program implements the steps of the convolution calculation method as described in any one of claims 1 to 7 when executed by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a convolution calculation program, which, when executed by a processor, implements the steps of the convolution calculation method according to any one of claims 1 to 7.