Coarse-grained reconfigurable processor convolution operator bandwidth optimization design method

Abstract

The invention relates to a coarse-grained reconfigurable processor convolution operator bandwidth optimization design method. The method comprises the following steps: performing matrix conversion processing on target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from a shared memory; the target original data in the shared memory is carried from the dynamic random access memory; and performing matrix multiplication operation on the matrix data to obtain a convolution calculation result of the target original data. The method comprises the following steps: carrying out matrix conversion processing on target original data carried in from a shared memory to obtain matrix data, carrying out matrix multiplication operation on the matrix data of the target original data to obtain a convolution calculation result of the target original data, and carrying out convolution calculation on the target original data to obtain a convolution calculation result of the target original data, the data is not expanded data after the target original data matrix is converted, so that the data carrying amount from the dynamic random access memory to the shared memory is reduced, and the bandwidth overhead of carrying the data from the dynamic random access memory to the shared memory is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a bandwidth optimization design method for a coarse-grained reconfigurable processor convolution operator. Background Art

[0002] A coarse-grained reconfigurable processor is a flexible and programmable processor that can dynamically reconfigure its hardware structure according to application requirements.

[0003] The coarse-grained reconfigurable processor includes a computing array, a dynamic random access memory, and a shared memory. When a coarse-grained reconfigurable processor is used for convolution calculation, in the related art, the matrix data after matrix transformation and expansion is usually transferred from the dynamic random access memory to the shared memory for matrix multiplication operation to complete the convolution calculation of the convolution data.

[0004] However, the convolution calculation method in the related art has a technical problem of large data transmission bandwidth overhead. Summary of the invention

[0005] Based on this, it is necessary to provide a coarse-grained reconfigurable processor convolution operator bandwidth optimization design method to address the above technical problems, which can reduce the bandwidth overhead of data transfer.

[0006] In a first aspect, the present application provides a method for optimizing the bandwidth of a convolution operator in a coarse-grained reconfigurable processor, which is applied to a computing unit in a coarse-grained reconfigurable processor. The method comprises:

[0007] Perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0008] Perform matrix multiplication on the matrix data to obtain the convolution calculation result of the target original data.

[0009] In one embodiment, the target original data includes feature original data and weight data; matrix conversion processing is performed on the target original data to obtain matrix data of the target original data, including:

[0010] According to the convolution kernel size and convolution step size, the original data of the feature map is transformed into a matrix to obtain the feature matrix corresponding to the original data of the feature map;

[0011] Perform matrix conversion processing on the weight data to obtain a weight matrix corresponding to the weight data;

[0012] The feature matrix corresponding to the feature map original data and the weight matrix corresponding to the weight data are determined as the matrix data of the target original data.

[0013] In one embodiment, before performing matrix conversion processing on the target original data to obtain matrix data of the target original data, the method further includes:

[0014] Determine the single-cycle output matrix dimension of the computing unit according to the single-cycle computing capability of the computing unit;

[0015] According to the single-cycle output matrix dimension, the target original data is moved from the dynamic random access memory.

[0016] In one embodiment, moving target original data from a dynamic random access memory according to a single-cycle output matrix dimension includes:

[0017] According to the single-cycle output matrix dimension, determine the feature map matrix size and weight matrix size;

[0018] According to the size of the feature map matrix and the weight matrix, the target original data is moved from the dynamic random access memory.

[0019] In one embodiment, the target raw data includes feature map raw data and weight data; the method further includes:

[0020] Obtain the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit;

[0021] According to the storage capacity and the single-cycle computing capacity, the feature map carrying capacity of the feature map original data and the weight carrying capacity of the weight data are determined.

[0022] In one embodiment, the storage capacity is capable of storing complete feature map original data and complete weight data; according to the storage capacity and the single-cycle computing capacity, determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data includes:

[0023] The data size of the original data of the feature map is determined as the feature map carrying amount, and the data size of the weight data is determined as the weight carrying amount.

[0024] In one embodiment, the storage capacity is capable of storing feature map original data of a first preset data size and complete weight data; determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity, including:

[0025] Determine the number of cycles of the matrix multiplication operation according to the complete output data size of the matrix multiplication operation and the single-cycle output matrix dimension;

[0026] Determine the feature map transfer amount of the feature map original data according to the number of cycles and the first preset data size;

[0027] According to the data size of the weight data, the weight carrying amount of the weight data is determined.

[0028] In one embodiment, determining the feature map transfer amount of the feature map original data according to the number of cycles and the first preset data size includes:

[0029] According to the number of cycles, characteristic graph cycle data related to the characteristic graph original data is obtained;

[0030] The feature map transport volume of the feature map original data is determined according to the feature map circulation data, the first preset data size and the matrix conversion expansion coefficient.

[0031] In one embodiment, the storage capacity is capable of storing feature map original data and complete weight data of a second preset data size; determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity, including:

[0032] Obtaining feature map cycle data related to feature map original data;

[0033] Determine the product of the characteristic graph circulation data and the second preset data size as the characteristic graph carrying amount of the characteristic graph original data;

[0034] The data size of the weight data is determined as the weight carrying amount of the weight data.

[0035] In one embodiment, the storage capacity is capable of storing feature map original data of a second preset data size and weight data of a third preset data size; determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity, including:

[0036] Get the preset loop order for matrix multiplication operations;

[0037] According to the preset loop order, the second preset data size and the third preset data size, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined.

[0038] In one embodiment, the preset loop order is that the original feature map data is fixed and the weight data is looped each time; according to the preset loop order, the second preset data size and the third preset data size, the feature map carrying amount of the original feature map data and the weight carrying amount of the weight data are determined, including:

[0039] Obtaining feature map cycle data related to feature map original data;

[0040] Determine the product of the characteristic graph circulation data and the second preset data size as the characteristic graph carrying amount of the characteristic graph original data;

[0041] The product of the number of cycles and the third preset data size is determined as the weighted carrying amount of the weighted data.

[0042] In one embodiment, the preset loop order is that the weight data is fixed and the feature map original data is looped each time; according to the preset loop order, the second preset data size and the third preset data size, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined, including:

[0043] According to the number of cycles of the matrix multiplication operation, weight cycle data related to the weight data is obtained;

[0044] The product of the number of cycles and the second preset data size is determined as the feature map transport amount of the feature map original data;

[0045] The product of the weight circulation data and the third preset data size is determined as the weight carrying amount of the weight data.

[0046] In one embodiment, the storage capacity is capable of storing data of feature map original data of a second preset data size after being segmented in the channel dimension and data of weight data of a third preset data size after being segmented in the channel dimension; determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity, including:

[0047] Get the number of loops of the matrix multiplication operation;

[0048] The product of the number of loops and the second preset data size is determined as the feature map carrying amount of the feature map original data, and the product of the number of loops and the third preset data size is determined as the weight carrying amount of the weight data.

[0049] In a second aspect, the present application also provides a coarse-grained reconfigurable processor convolution operator bandwidth optimization design device, which is applied to a computing unit in a coarse-grained reconfigurable processor, and the device includes:

[0050] The matrix conversion module is used to perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0051] The matrix multiplication calculation module is used to perform matrix multiplication operations on matrix data to obtain the convolution calculation results of the target original data.

[0052] In a third aspect, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any one of the embodiments of the first aspect when executing the computer program.

[0053] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in any one of the embodiments of the first aspect are implemented.

[0054] In a fifth aspect, the embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in any one of the embodiments in the first aspect are implemented.

[0055] The above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design method obtains the matrix data of the target original data by performing matrix transformation processing on the target original data, and then performs matrix multiplication operation on the matrix data to obtain the convolution calculation result of the target original data, wherein the target original data is obtained by the computing unit from the shared memory, and the target original data in the shared memory is moved from the dynamic random access memory. In this method, the target original data is moved from the dynamic random access memory to the shared memory. When performing convolution calculation, the computing unit can obtain the moved target original data from the shared memory for matrix transformation processing to obtain the matrix data of the target original data, and then perform matrix multiplication operation on the matrix data of the target original data, that is, to obtain the convolution calculation result of the target original data, wherein the data moved from the dynamic random access memory to the shared memory is the target original data, rather than the data expanded after the matrix transformation of the target original data, which is equivalent to reducing the amount of data moved from the dynamic random access memory to the shared memory, thereby reducing the bandwidth overhead of data moved from the dynamic random access memory to the shared memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 FIG. 1 is an application environment diagram of a bandwidth optimization design method for a coarse-grained reconfigurable processor convolution operator in one embodiment;

[0058] Figure 2 is a schematic diagram of the architecture of a coarse-grained reconfigurable processor in one embodiment;

[0059] Figure 3 A schematic diagram of a flow chart of a method for optimizing bandwidth design of a convolution operator of a coarse-grained reconfigurable processor in one embodiment;

[0060] Figure 4 A schematic diagram of a data transfer process in one embodiment;

[0061] Figure 5 A schematic diagram of a process for determining the data handling amount in one embodiment;

[0062] Figure 6 A schematic diagram of a flow chart for determining the data handling amount in another embodiment;

[0063] Figure 7 A schematic diagram of a flow chart for determining the data handling amount in another embodiment;

[0064] Figure 8 A schematic diagram of a flow chart for determining the data handling amount in another embodiment;

[0065] Fig. 9 A schematic diagram of a flow chart for determining the data handling amount in another embodiment;

[0066] Fig.10 It is a schematic diagram of the structure of a device for optimizing bandwidth design of a convolution operator of a coarse-grained reconfigurable processor in one embodiment;

[0067] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The coarse-grained reconfigurable processor convolution operator bandwidth optimization design method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. 1 , the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The server 104 can be a computing unit in a coarse-grained reconfigurable processor. The architecture of the coarse-grained reconfigurable processor can be as follows: Figure 2As shown, it includes a coarse-grained reconfigurable computing array, which includes multiple computing units (Processing Element, PE), a user-managed on-chip memory (shard memory, shared memory) that interacts directly with the double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR) via a bus, and a possible memory management unit array (Memory Management Unit, MMU). Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0070] The technical background of this application is described below.

[0071] A coarse-grained reconfigurable processor is a processor with flexibility and programmability, and its hardware structure can be dynamically reconfigured according to application requirements. Neural network, also known as artificial neural network, is a computing model that simulates human brain neurons and is completed after a large amount of data training. It is used to output corresponding judgment / computation results when new data is input. Neural networks have a wide range of applications in various fields, such as object detection, data prediction, etc. The application of neural networks on coarse-grained reconfigurable processors is an important scenario. In high-computation applications such as machine learning, neural network training and reasoning are required. At present, the computing power and number of computing units of the hardware based on which neural networks run have been significantly improved, while memory and data handling have become the bottleneck of the entire system. The high consumption of bandwidth not only blocks the operation of the entire system, but also occupies a major part of the total network reasoning time. Especially for models with high-precision data as input and large computing input and output at each layer, reducing unnecessary data handling, reducing bandwidth overhead, and increasing computing unit utilization are very critical optimization technologies.

[0072] In the related art, the convolution calculation of the coarse-grained reconfigurable processor is equivalent to matrix calculation after the original data img2col is implemented. Due to the memory limitation of the chip where the computing unit is located, the original data is cut into blocks in the C dimension, H dimension, and W dimension, and then the blocks are calculated. The existing data handling scheme for the calculation and the data required for the calculation is not optimized, and the data handling is repeated many times, which occupies a lot of bandwidth.

[0073] Based on this, an embodiment of the present application provides a bandwidth optimization design method for a coarse-grained reconfigurable processor convolution operator, in which the target original data is moved from the dynamic random access memory to the shared memory. When performing convolution calculations, the computing unit can obtain the moved target original data from the shared memory for matrix conversion processing to obtain matrix data of the target original data, and then perform matrix multiplication operations on the matrix data of the target original data to obtain the convolution calculation result of the target original data, wherein the data moved from the dynamic random access memory to the shared memory is the target original data, rather than the data expanded after the matrix conversion of the target original data, which is equivalent to reducing the amount of data moved from the dynamic random access memory to the shared memory, thereby reducing the bandwidth overhead of moving data from the dynamic random access memory to the shared memory.

[0074] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0075] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] In an exemplary embodiment, Figure 3 As shown, a method for optimizing the bandwidth design of a convolution operator of a coarse-grained reconfigurable processor is provided, and the method is applied to a computing unit in a coarse-grained reconfigurable processor as an example for explanation, including the following steps 201 to 202. Among them:

[0077] S201, performing matrix conversion processing on target original data to obtain matrix data of target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory.

[0078] The target raw data refers to the data input to the convolution operator, including the feature map raw data and weight data. The feature map raw data is the output data processed by the convolution kernel of the previous convolution operator, and the weight data refers to the parameters in the convolution kernel. These parameters are learned during the training process and are used for weighted summation in the convolution operation to generate the feature map.

[0079] When performing convolution calculation, the original data of Feature Map is usually img2col, the elements in each convolution window are expanded into a row, and the convolution window is slid to the right and downward, and finally the three-dimensional original data of Feature Map is converted into a two-dimensional matrix; then the Weight data is img2col, and each convolution kernel is expanded into a column, which is consistent with the number of single-row elements of the original data of Feature Map, and multiple convolution kernels are combined into a two-dimensional matrix; then the two-dimensional matrix after img2col of the original data of Feature Map and the matrix after img2col of the Weight data are matrix multiplied, and the output results are sorted in memory to obtain equivalent convolution calculation results. It can be seen that the convolution calculation process is to perform img2col first and then perform matrix multiplication operation. Among them, the matrix conversion processing in the embodiment of the present application refers to img2col. img2col is an accelerated calculation strategy for realizing convolution operation, which shortens the convolution calculation time to the maximum extent by converting the convolution operation into the form of general matrix multiplication (GEneral Matrix Multiply, GEMM).

[0080] Shared memory refers to shard memory in coarse-grained reconfigurable processors, and dynamic random access memory refers to DDR in coarse-grained reconfigurable processors.

[0081] In the related art, the required data is configured according to the single-cycle computing capability of the computing unit, and the data is directly moved from the DDR, the data that has completed img2col is moved, or img2col is performed during the transfer process, and the data moved into the shared memory is the data that has expanded after img2col. In this way, it is equivalent to transferring the original data after expansion. Since the original data expands after img2col, the bandwidth overhead of transferring data from DDR to shared memory increases.

[0082] In order to reduce the bandwidth overhead of moving data from DDR to shared memory, in an embodiment of the present application, the target original data is directly moved from DDR, that is, the target original data is directly moved from DDR to the shared memory. When performing convolution calculations, the computing unit can obtain the moved target original data from the shared memory to perform img2col and matrix multiplication operations.

[0083] Usually, the storage capacity of shared memory is limited, and all target original data cannot be directly moved to the shared memory. Therefore, the target original data can be cut into pieces when moving. The cutting dimensions of the target original data can be spatial dimensions and channel dimensions. The spatial dimension refers to the W dimension and the H dimension, and the channel dimension refers to the C dimension.

[0084] According to the preset slicing method, the target original data is sliced, and the sliced ​​target original data is moved from the dynamic random access memory to the shared memory. The computing unit obtains the sliced ​​target original data from the shared memory, and performs matrix conversion processing on the sliced ​​target original data to obtain matrix data. Among them, one sliced ​​target original data corresponds to one matrix data.

[0085] Exemplarily, a method for performing matrix transformation processing on the target original data to obtain the matrix data of the target original data can be to perform matrix transformation processing on the feature map original data according to the convolution kernel size and the convolution step size to obtain the feature matrix corresponding to the feature map original data; perform matrix transformation processing on the weight data to obtain the weight matrix corresponding to the weight data; and determine the feature matrix corresponding to the feature map original data and the weight matrix corresponding to the weight data as the matrix data of the target original data.

[0086] In an embodiment of the present application, the convolution kernel size and the parameters in the convolution kernel are used as the convolution window, and the convolution window is slid in the feature map original data in sequence according to the convolution step length. The feature map original data in each convolution window is expanded into a row, and the convolution window is slid rightward and downward according to the convolution step length to convert the feature map original data into a two-dimensional matrix, that is, the feature matrix corresponding to the feature map original data is obtained. Similarly, the weight parameters of each convolution kernel are also expanded into a column, and the data expanded by multiple convolution kernels are combined into a two-dimensional matrix, that is, the weight matrix corresponding to the weight data is obtained. After obtaining the feature matrix corresponding to the feature map original data and the weight matrix corresponding to the weight data, the feature matrix corresponding to the feature map original data and the weight matrix corresponding to the weight data are used as the matrix data of the target original data.

[0087] S202, performing a matrix multiplication operation on the matrix data to obtain a convolution calculation result of the target original data.

[0088] The target original data includes feature map original data and weight data, and the matrix data includes a matrix corresponding to the feature map original data and a matrix corresponding to the weight data.

[0089] After obtaining the matrix data of the target original data, matrix multiplication is performed on the matrix corresponding to the feature map original data and the matrix corresponding to the weight data in the matrix data of the target original data to obtain a new matrix, which is the convolution calculation result of the target original data.

[0090] In the bandwidth optimization design method of the convolution operator of the coarse-grained reconfigurable processor provided in the embodiment of the present application, the matrix data of the target original data is obtained by performing matrix conversion processing on the target original data, and then the matrix data is subjected to matrix multiplication operation to obtain the convolution calculation result of the target original data, wherein the target original data is obtained by the computing unit from the shared memory, and the target original data in the shared memory is transported from the dynamic random access memory. In this method, the target original data is moved from the dynamic random access memory to the shared memory. When performing the convolution calculation, the computing unit can obtain the moved target original data from the shared memory for matrix conversion processing to obtain the matrix data of the target original data, and then the matrix data of the target original data is subjected to matrix multiplication operation, that is, the convolution calculation result of the target original data is obtained, wherein the data moved from the dynamic random access memory to the shared memory is the target original data, rather than the data expanded after the matrix conversion of the target original data, which is equivalent to reducing the amount of data transported from the dynamic random access memory to the shared memory, thereby reducing the bandwidth overhead of data transported from the dynamic random access memory to the shared memory.

[0091] Based on the above embodiment, an embodiment is provided to illustrate the data handling process before performing matrix conversion processing on the target original data to obtain the matrix data of the target original data.

[0092] In an exemplary embodiment, Figure 4 As shown, before performing matrix conversion processing on the target original data to obtain matrix data of the target original data, the method further includes:

[0093] S301, determining a single-cycle output matrix dimension of a computing unit according to the single-cycle computing capability of the computing unit.

[0094] The single-cycle computing capability of a computing unit refers to the number of computing operations that the computing unit can perform in a single clock cycle. The single-cycle output matrix dimension refers to the size of the matrix that the computing unit can output after the matrix multiplication operation in one computing cycle.

[0095] Different single-cycle computing capabilities correspond to different single-cycle output matrix dimensions. Exemplarily, the single-cycle output matrix dimension of the computing unit can be determined according to a capability-dimension mapping relationship pre-stored in a database.

[0096] For example, the database stores the mapping relationship between different single-cycle computing capabilities and different single-cycle output matrix dimensions. The single-cycle computing capability of the computing unit can be obtained first, and then the corresponding single-cycle output matrix dimension can be matched from the mapping relationship in the above database according to the single-cycle computing capability of the computing unit, and the matched single-cycle output matrix dimension can be determined as the single-cycle output matrix dimension of the computing unit.

[0097] S302, moving target original data from the dynamic random access memory according to the single-cycle output matrix dimension.

[0098] After determining the single-cycle output matrix dimension of the computing unit, the target original data can be moved from the dynamic random access memory according to the single-cycle output matrix dimension.

[0099] Exemplarily, according to the single-cycle output matrix dimension, the method of moving the target original data from the dynamic random access memory can be to determine the feature map matrix size and the weight matrix size according to the single-cycle output matrix dimension; and according to the feature map matrix size and the weight matrix size, move the target original data from the dynamic random access memory.

[0100] Assuming that the target original data of a convolution operator has a Feature Map original data size of N*H*W*Cin, a Weight data size of Cout*KH*KW*Cin, a Stride of (strideH, strideW), and a Padding of (PaddingT, PaddingB, PaddingLt, PaddingR), then the complete output data size (Nout, Hout, Wout, Cout) is

[0101]

[0102] Take the single-cycle output matrix dimension of the computing unit as M0*N0 as an example, where M0 is the product of the dimensions of the single-cycle output matrix in the H and W directions, and N0 is the C dimension of the single-cycle output matrix. Then the calculation of the single-cycle output matrix requires the input of the Feature Map matrix after img2col of size M0*K0 and the input of the Weight matrix of size K0*N0. According to the characteristics of img2col, M0 is the number of sliding expanded convolution windows, and K0 is the product of Cin*KH*kW, and N0 is the number of convolution kernels of the expanded Weight. Among them, M0*K0 is the size of the feature map matrix, and the feature map matrix refers to the matrix obtained after img2col is performed on the original feature map data. K0*N0 is the size of the weight matrix, and the weight matrix size refers to the matrix obtained after img2col is performed on the weight data.

[0103] After obtaining the size of the feature map matrix and the size of the weight matrix, the size of the required feature map original data and the size of the weight data can be inferred based on the size of the feature map matrix and the size of the weight matrix. According to the inferred size of the feature map original data and the size of the weight data, the target original data is moved from the dynamic random access memory. Still taking the single-cycle output matrix dimension of the computing unit as M0*N0 as an example, to output a matrix of size M0*N0, the size of the corresponding feature map original data is M0*K0 / (KH*kW)=M0*Cin, and the size of the weight data is K0*N0.

[0104] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in the embodiment of the present application, the single-cycle output matrix dimension of the computing unit is determined according to the single-cycle computing power of the computing unit, and then the target original data is moved from the dynamic random access memory according to the single-cycle output matrix dimension. In this method, an optional way to move the target original data is provided; the single-cycle output matrix dimension of the computing unit can be determined according to the single-cycle computing power of the computing unit, and the required target original data size can be inferred according to the single-cycle output matrix dimension, so that the target original data can be quickly and accurately moved from the dynamic random access memory for calculation.

[0105] Based on the above embodiment, an embodiment is provided to illustrate the process of determining the data transfer amount when transferring the target original data.

[0106] In an exemplary embodiment, Figure 5 As shown, the target original data includes feature map original data and weight data; the method also includes:

[0107] S401, obtaining the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit.

[0108] The storage capacity of the shared memory refers to the size of the feature map raw data and the weight raw data that can be stored in the shared memory.

[0109] The storage capacity of the shared memory and the single-cycle computing capacity of the computing unit can be obtained by testing the shared memory and the computing unit through specific means. For example, the shared memory and the computing unit can be tested using corresponding performance testing tools to obtain the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit.

[0110] S402, determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity.

[0111] Under different storage capacities, the feature map carrying capacity of the feature map original data and the weight carrying capacity of the weight data are different. Optionally, the feature map carrying capacity of the feature map original data and the weight carrying capacity of the weight data can be calculated based on a preset carrying capacity calculation formula, combined with the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit.

[0112] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in the embodiment of the present application, the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit are obtained, and then the feature map handling volume of the feature map original data and the weight handling volume of the weight data are determined according to the storage capacity and the single-cycle computing capacity. In this method, an optional way to quickly determine the data handling volume is provided; by obtaining the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit, the feature map handling volume of the feature map original data and the weight handling volume of the weight data are calculated according to the storage capacity and the single-cycle computing capacity, providing significant data support for the reduction of data handling volume.

[0113] Based on the above embodiment, an embodiment is provided to illustrate the process of determining the feature map carrying amount of feature map original data and the weight carrying amount of weight data according to the storage capacity and the single-cycle computing capacity.

[0114] In an exemplary embodiment, the storage capacity is capable of storing complete feature map original data and complete weight data; according to the storage capacity and the single-cycle computing capacity, determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data includes:

[0115] The data size of the original data of the feature map is determined as the feature map carrying amount, and the data size of the weight data is determined as the weight carrying amount.

[0116] When the memory capacity in the shared memory is large enough to accommodate the complete feature map original data and the complete weight data, the feature map original data and weight data only need to be moved once, and the data moving amount corresponding to the feature map original data and weight data is their own data size. At this time, the data size of the feature map original data is determined as the feature map moving amount, and the data size of the weight data is determined as the weight moving amount.

[0117] In the embodiment of the present application, the shared memory is sufficient to hold all the original Feature Map data and two Feature Map matrices of size M0*K0 after the original Feature Map data are expanded, as well as complete Weight data of size Cout*K0.

[0118] In the shared memory, expand a copy of the original data img2col of M0*Cin into a Feature Map matrix of M0*K0 at a time, and generate a Feature Map matrix and Weight matrix input of M0*K0 and K0*N0 in each cycle and calculate and generate a complete output of M0*N0 size as the inner loop (output a matrix of M0*N0 size in a single cycle), and the outer loop needs to be Times: Each time after the Feature Map original data is expanded, the Feature Map matrix is ​​kept unchanged, and different Weight matrices are replaced for calculation until the complete output of M0*Cout is completed. At the same time, the second block of Feature Map original data input is expanded during calculation using the pingpong mechanism. After the second block of Feature Map original data input is expanded, the computing unit uses the block and different Weight matrices to complete the complete output of the next block of M0*Cout... Because the required data are already in the on-chip memory, this process does not consume on-chip and off-chip bandwidth, and the pingpong mechanism can be used to overlap the on-chip memory transfer time and calculation as much as possible, so that the processor computing utilization rate is maximized. The amount of Feature Map original data carried is N*H*W*Cin, the amount of Weight carried is Cout*K0, and the total input data carried is N*H*W*Cin+Cout*K0.

[0119] If each time after the Feature Map original data is expanded, the Feature Map matrix is ​​kept unchanged, and different Weight matrices are replaced for calculation until the complete output of M0*Cout is completed, the calculation unit waits for the second block of Feature Map original data input to be expanded, and then uses the block and different Weight matrices to complete the complete output of the next block of M0*Cout... At this time, the amount of Feature Map original data carried is N*H*W*Cin, the amount of Weight carried is Cout*K0, and the total input data carried is N*H*W*Cin+Cout*K0. This case has the same data carrying volume as the previous case, but the difference is that in this case, the pingpong mechanism cannot be used to expand the img2col of the next Feature Map original data while calculating, and this process may cause blocking.

[0120] In the coarse-grained reconfigurable processor convolution operator bandwidth optimization design method provided in the embodiment of the present application, the storage capacity is to be able to store complete feature map original data and complete weight data, the data size of the feature map original data is determined as the feature map handling amount, and the data size of the weight data is determined as the weight handling amount. In this method, when the shared memory can store the complete feature map original data and the complete weight data, the data size of the feature map original data is the handling amount of the feature map original data, and the data size of the weight data is the handling amount of the weight data.

[0121] Based on the above embodiment, another embodiment is provided to illustrate the process of determining the feature map carrying amount of feature map original data and the weight carrying amount of weight data according to the storage capacity and single-cycle computing capacity.

[0122] In an exemplary embodiment, Figure 6 As shown, the storage capacity is capable of storing feature map original data of a first preset data size and complete weight data; according to the storage capacity and the single-cycle computing capacity, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined, including:

[0123] S501, determining the number of cycles of the matrix multiplication operation according to the complete output data size of the matrix multiplication operation and the single-cycle output matrix dimension.

[0124] The complete output data size of the matrix multiplication operation refers to the matrix size of the final convolution calculation result. Taking the Feature Map original data size of N*H*W*Cin and the Weight data size of Cout*KH*KW*Cin as an example, the complete output data size of the matrix multiplication operation is Nout*Hout*Wout*Cout. The first preset data size is M0*K0.

[0125] In the embodiment of the present application, the ratio of the complete output data size to the single-cycle output matrix dimension can be determined as the number of cycles of the matrix multiplication operation, that is, in, Circulating data for feature graphs related to feature graph original data; Loop through the weights data associated with the weights data.

[0126] S502, determining a feature map transfer amount of the feature map original data according to the number of cycles and the first preset data size.

[0127] S503, determining the weight carrying amount of the weight data according to the data size of the weight data.

[0128] Exemplarily, a method for determining the feature map transport volume of the feature map original data according to the number of loops and the first preset data size may be to obtain feature map loop data related to the feature map original data according to the number of loops; and determine the feature map transport volume of the feature map original data according to the feature map loop data, the first preset data size and the matrix conversion expansion coefficient.

[0129] Among them, the feature map cycle data related to the feature map original data is The matrix conversion expansion coefficient is KH*KW.

[0130] Optionally, the product of the feature map loop data and the first preset data size can be determined, and then the ratio of the product to the matrix conversion expansion coefficient is determined as the feature map handling capacity of the feature map original data. Since the shared memory can store complete weight data, the data size of the weight data is determined as the weight handling capacity of the weight data.

[0131] In the embodiment of the present application, the shared memory is sufficient to hold two copies of Feature Map original data of size M0*K0, a Feature Map matrix of size M0*K0 after one copy of the Feature Map original data is expanded, and complete Weight data.

[0132] In the shared memory, expand a copy of the original M0*K0 / (KH*kW) Feature Map data img2col into a M0*K0 Feature Map matrix at a time, and generate a Feature Map matrix and Weight matrix input of M0*K0 and K0*N0 size in each cycle and calculate a complete output of M0*N0 size as the inner loop, and the outer loop needs to be Each time the Feature Map original data is expanded, the Feature Map matrix is ​​kept unchanged, and different Weight matrices are replaced for calculation until the complete output of M0*Cout is completed. The calculation unit waits for the second block of Feature Map original data input to be expanded, and then uses the block and different Weight matrices to complete the complete output of the next block of M0*Cout... In this calculation process, the amount of Feature Map original data moved is about The total weight transported is K0*Cout, and the total data transported is approximately Because the pingpong mechanism can be carried out: that is, the second input can be moved in while the first input is being calculated, which can improve the utilization of the computing unit in the processor and reduce the total running time of the entire operator. Compared with the aforementioned situation where the complete feature map original data and weight data can be placed, although the pingpong mechanism is used to move the Feature Map original data, compared with the original data being able to be stored in shared memory, there may still be a situation where the system is blocked by the transfer. If the pingpong mechanism cannot be used in this case, the img2col expansion of the next Feature Map original data is performed while the calculation is being performed, and this process may cause blocking.

[0133] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in the embodiment of the present application, the storage capacity is to store the feature map original data and the complete weight data of the first preset data size, and the number of cycles of the matrix multiplication operation is determined according to the complete output data size of the matrix multiplication operation and the single-cycle output matrix dimension, and then the feature map carrying volume of the feature map original data is determined according to the number of cycles and the first preset data size, and then the weight carrying volume of the weight data is determined according to the data size of the weight data. In this method, when the shared memory can hold the feature map original data and the complete weight data of the first preset data size, the number of cycles of the matrix multiplication operation is first obtained, and then the feature map carrying volume of the feature map original data is calculated according to the number of cycles combined with the first preset data size, and the weight carrying volume of the weight data is calculated according to the data size of the weight data, which provides an optional way to quickly determine the carrying volume of the target original data.

[0134] Based on the above embodiment, another embodiment is provided to illustrate the process of determining the feature map carrying amount of feature map original data and the weight carrying amount of weight data according to the storage capacity and single-cycle computing capacity.

[0135] In an exemplary embodiment, Figure 7 As shown, the storage capacity is capable of storing feature map original data and complete weight data of a second preset data size; according to the storage capacity and the single-cycle computing capacity, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined, including:

[0136] S601, obtaining feature map cycle data related to feature map original data.

[0137] As mentioned above, the feature map cycle data related to the feature map original data can be obtained according to the number of cycles of the matrix multiplication operation, that is,

[0138] S602: Determine the product of the feature map circulation data and the second preset data size as the feature map carrying amount of the feature map original data.

[0139] The second preset data size is M0*Cin, and the feature map transfer volume of the original feature map data is

[0140] S603, determining the data size of the weight data as the weight carrying amount of the weight data.

[0141] The data size of the weight data is K0*Cout, and the weight carrying capacity of the weight data is K0*Cout.

[0142] In the embodiment of the present application, the shared memory is sufficient to hold two copies of M0*Cin Feature Map original data and the Feature Map matrix after the C channel portion of one copy of the Feature Map original data is expanded, as well as the complete Weight data.

[0143] In the shared memory, each time the original Feature Map data img2col is used, part of the original Feature Map data is expanded on the C channel with DevidedCin as the granularity, and the corresponding Weight matrix is ​​used for calculation to obtain the partial sum output of the size of M0*N0, and the expansion of the Feature Map part on the C channel is repeated until a complete output of the size of M0*N0 is completed. The internal loop is to generate a Feature Map matrix and Weight matrix input of 1 size of M0*K0 and K0*N0 per cycle and calculate to generate a complete output of the size of M0*N0, and the external loop needs to be looped In this calculation process, the amount of original Feature Map data moved is about The amount of Weight data transported is K0*Cout, and the total amount of data transported is approximately Because the pingpong mechanism is implemented, that is, the second input can be moved in while the first input is being calculated, the utilization rate of the processor computing unit can be improved and the total running time of the entire operator can be reduced. The difference from the previous case is that each FeatureMap original data may need to be repeatedly expanded within the chip, which may bring additional load to the on-chip memory.

[0144] If the shared memory is large enough to hold a copy of the M0*Cin Feature Map original data, a Feature Map matrix after the C channel portion of one of the Feature Map original data is expanded, and the complete Weight data.

[0145] In the shared memory, each time the original Feature Map data img2col is used, part of the original Feature Map data is expanded on the C channel with DevidedCin as the granularity, and the corresponding Weight matrix is ​​used for calculation to obtain the partial sum output of the size of M0*N0, and the expansion of the Feature Map part on the C channel is repeated until a complete output of the size of M0*N0 is completed. The internal loop is to generate a Feature Map matrix and Weight matrix input of 1 size of M0*K0 and K0*N0 per cycle and calculate to generate a complete output of the size of M0*N0, and the external loop needs to be looped In this calculation process, the amount of original Feature Map data moved is about The amount of Weight data transported is K0*Cout, and the total amount of data transported is approximately Because the pingpong mechanism can be implemented, that is, the second input can be moved in while the first input is being calculated, it can improve the utilization of the processor computing unit and reduce the total running time of the entire operator. The difference from the previous case is that the pingpong mechanism cannot be implemented. Each time the processor completes the calculation, it starts to move in the next Feature Map original data. The calculation can only continue after the next Feature Map original data is moved in. Therefore, the bandwidth from dynamic random access memory to shared memory will block the calculation and increase the calculation time of the operator.

[0146] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in an embodiment of the present application, the storage capacity is to store feature map original data and complete weight data of a second preset data size, and the feature map loop data related to the feature map original data is obtained, and then the product of the feature map loop data and the second preset data size is determined as the feature map handling volume of the feature map original data, and the data size of the weight data is determined as the weight handling volume of the weight data. In this method, when the shared memory can accommodate the feature map original data and complete weight data of the second preset data size, the product of the feature map loop data and the second preset data size is determined as the feature map handling volume of the feature map original data, and the data size of the weight data is determined as the weight handling volume of the weight data, which provides another optional way to quickly determine the handling volume of the target original data.

[0147] Based on the above embodiment, another embodiment is provided to illustrate the process of determining the feature map carrying amount of feature map original data and the weight carrying amount of weight data according to the storage capacity and single-cycle computing capacity.

[0148] In an exemplary embodiment, Figure 8As shown, the storage capacity is capable of storing feature map original data of a second preset data size and weight data of a third preset data size; according to the storage capacity and the single-cycle computing capacity, determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data, including:

[0149] S701, obtaining a preset loop order of a matrix multiplication operation.

[0150] In the embodiment of the present application, the third preset data size is K0*N0. The preset loop order includes two situations: the original data of the feature map is fixed and the weight data is looped each time, and the weight data is fixed and the original data of the feature map is looped each time.

[0151] S702, determining a feature map carrying amount of feature map original data and a weight carrying amount of weight data according to a preset loop order, a second preset data size, and a third preset data size.

[0152] Exemplarily, when the preset loop order is that the feature map original data is fixed, each time the weight data is looped, feature map loop data related to the feature map original data is obtained; the product of the feature map loop data and the second preset data size is determined as the feature map carrying capacity of the feature map original data; the product of the number of loops and the third preset data size is determined as the weight carrying capacity of the weight data.

[0153] Among them, according to the number of cycles of the matrix multiplication operation, the feature map cycle data related to the feature map original data is obtained, and the feature map cycle data is obtained as follows: The second preset data size is M0*Cin. The third preset data size is K0*N0. The number of cycles is

[0154] The feature map transfer volume of the original feature map data is The weighted transport volume of weighted data is

[0155] Exemplarily, when the preset loop order is that the weight data is fixed, each time the feature map original data is looped, weight loop data related to the weight data is obtained according to the number of loops of the matrix multiplication operation; the product of the number of loops and the second preset data size is determined as the feature map transport volume of the feature map original data; the product of the weight loop data and the third preset data size is determined as the weight transport volume of the weight data.

[0156] The number of cycles is The weight cycle data related to the weight data is

[0157] The amount of feature map data carried The weighted transport volume of weighted data is

[0158] In the embodiment of the present application, the shared memory is sufficient to hold a copy of the original Feature Map data, a Feature Map matrix after the original Feature Map data is expanded in the C channel, and Weight data of size K0*N0. In this case, different repeated transfers will be generated according to different loop orders.

[0159] In the shared memory, each time the Feature Map original data img2col is used, part of the Feature Map original data is expanded on the C channel with DevidedCin as the granularity, and the corresponding Weight matrix is ​​used for calculation to obtain the partial sum output of the size of M0*N0, and the expansion of the Feature Map part on the C channel is repeated until a complete output of the size of M0*N0 is completed. During this period, no data transfer between the chip and the original Feature Map data and the Weight data is generated. The internal loop is to generate a Feature Map matrix and Weight matrix input of 1 size of M0*K0 and K0*N0 per cycle and calculate to generate a complete output of the size of M0*N0, and the external loop needs to be Second-rate.

[0160] The original data of Feature Map is fixed, and the Weight matrix corresponding to different convolution kernels is used each time to complete all the blocks of M0*N0 size output on the Cout channel as the inner loop, and the blocks in HW space as the outer loop. In this calculation process, the amount of original data of Feature Map moved is about The amount of Weight data carried is The total data volume is approximately

[0161]

[0162] The Weight data remains unchanged, and different Feature Map original data is used each time to complete all the blocks of M0*N0 size output in the HW space dimension as the inner loop, and the blocks on the Cout channel as the outer loop. In this calculation process, the Feature Map handling volume is about The total weight is The total data volume is approximately According to the size of the Weight matrix and Feature Map matrix of the specific operator, you can choose the solution with the least amount of data to be moved.

[0163] In addition, when the shared memory is large enough to hold a copy of the original Feature Map data, a Feature Map matrix after the original data is expanded in the C channel, and a Weight matrix of DevidedCin*KH*KW*N0.

[0164] In the shared memory, each time the original Feature Map data img2col is used, part of the original Feature Map data is expanded on the C channel with DevidedCin as the granularity, and the corresponding Weight matrix is ​​used for calculation to obtain the partial sum output of M0*N0 size, and the expansion of the Feature Map part on the C channel is repeated until a complete output of M0*N0 size is completed. During this period, the original Feature Map data does not generate data transfer between the chip and the outside, and the Weight data needs to be transferred multiple times. The internal loop generates a Feature Map and Weight matrix input of M0*K0 and K0*N0 size in each cycle and calculates to generate a complete output of M0*N0 size, and the external loop needs to be looped In this calculation process, the Feature Map handling capacity is about The amount of Weight data carried is The total data volume is approximately

[0165]

[0166] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in the embodiment of the present application, the storage capacity is to store the feature map original data of the second preset data size and the weight data of the third preset data size, and the preset loop order of the matrix multiplication operation is obtained, and then the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined according to the preset loop order, the second preset data size and the third preset data size. In this method, when the feature map original data of the second preset data size and the weight data of the third preset data size can be placed in the shared memory, the feature map original data and the weight data need to be cut into pieces and then carried. Through different loop orders, combined with the second preset data size and the third preset data size, the second preset data size and the third preset data size are calculated respectively, which provides another optional way to quickly determine the carrying amount of the target original data.

[0167] Based on the above embodiment, another embodiment is provided to illustrate the process of determining the feature map carrying amount of feature map original data and the weight carrying amount of weight data according to the storage capacity and single-cycle computing capacity.

[0168] In an exemplary embodiment, Fig. 9As shown, the storage capacity is capable of storing data after the feature map original data of the second preset data size is segmented on the channel dimension and the weight data of the third preset data size is segmented on the channel dimension; according to the storage capacity and the single-cycle computing capacity, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined, including:

[0169] S801, obtaining the number of cycles of the matrix multiplication operation.

[0170] As mentioned earlier, the number of cycles for the matrix multiplication operation is

[0171] S802, determining the product of the number of loops and the second preset data size as the feature map carrying amount of the feature map original data, and determining the product of the number of loops and the third preset data size as the weight carrying amount of the weight data.

[0172] The feature map transfer volume of the original feature map data is The weighted transport volume of weighted data is

[0173] When the original Feature Map data and Weight data need to be split in the C dimension before being stored in the shared memory, the number of repeated transfers of the original Feature Map data and Weight data will become the largest.

[0174] The inner loop generates a Feature Map and Weight matrix input of size M0*K0 and K0*N0 per cycle and calculates a complete output of size M0*N0. The outer loop needs to be In this calculation process, the amount of original Feature Map data moved is about The amount of Weight data carried is The total data volume is approximately

[0175]

[0176] In the bandwidth optimization design method for the convolution operator of a coarse-grained reconfigurable processor provided in an embodiment of the present application, the storage capacity is to be able to store the data after the feature map original data of the second preset data size is segmented in the channel dimension and the data after the weight data of the third preset data size is segmented in the channel dimension, and the number of cycles of the matrix multiplication operation is obtained, and then the product of the number of cycles and the second preset data size is determined as the feature map handling amount of the feature map original data, and the product of the number of cycles and the third preset data size is determined as the weight handling amount of the weight data. In this method, when the data that can be stored in the shared memory storage is the segmented feature map original data and the segmented weight data, the handling amount increases and reaches the maximum. At this time, the product of the number of cycles and the second preset data size is determined as the feature map handling amount of the feature map original data, and the product of the number of cycles and the third preset data size is determined as the weight handling amount of the weight data, providing another optional way to determine the handling amount of the target original data.

[0177] In order to clearly show that the improved solution of the embodiment of the present application reduces the bandwidth overhead, the amount of Feature Map matrix and the amount of Weight matrix carried after img2col in various cases in the related art are explained.

[0178] ① The shared memory is large enough to hold the complete Feature Map matrix, complete Weight data, and M0*N0 block matrix output after img2col.

[0179] This is generally a convolution operator with a small amount of data. In this case, each cycle generates a Feature Map matrix and Weight matrix input of size M0*K0 and K0*N0 and calculates to generate a complete output of size M0*N0 as the inner loop, and the outer loop needs to be However, because the required data are already in the shared memory, this process does not consume on-chip and off-chip bandwidth, and the processor computing utilization rate reaches the highest. The Feature Map matrix transport volume is Nout*Hout*Wout*K0, and the Weight matrix transport volume is Cout*K0. The total transport volume is Nout*Hout*Wout*K0+Cout*K0.

[0180] ② The shared memory is large enough to hold two Feature Map matrices of size M0*K0 and a complete Weight matrix.

[0181] This operator with a small amount of Weight data can store the Weight matrix completely in the on-chip memory to reduce repeated transfers. Each cycle generates a Feature Map matrix and Weight matrix input with a size of M0*K0 and K0*N0 and calculates to generate a complete output of size M0*N0 as the internal loop, and the external loop needs to be Each time after expanding the original Feature Map data of size M0*K0, keep the Feature Map unchanged, use different N0-sized Weight blocks for multiple calculations until the complete output of M0*Cout is completed; use the pingpong mechanism to carry out the next block of Feature Map of size M0*K0 in the above calculation process; use the second block of Feature Map to complete the complete output of the next block of M0*Cout... In such a calculation process, the amount of Feature Map matrix carried is The transport capacity of the Weight matrix is ​​K0*Cout, and the total transport capacity is Because the pingpong mechanism can be carried out, that is, the second input can be moved in while the first input is being calculated, the utilization rate of the processor computing unit can be improved and the total running time of the entire operator can be reduced.

[0182] ③The shared memory is large enough to hold one M0*K0 Feature Map matrix and a complete Weight matrix.

[0183] The inner loop generates a Feature Map matrix and Weight matrix input of size M0*K0 and K0*N0 per cycle and calculates a complete output of size M0*N0. The outer loop needs to be Each time the Feature Map of size M0*K0 is expanded, the Feature Map is kept unchanged, and multiple calculations are performed using different Weight blocks of size N0 until the complete output of M0*Cout is completed; the pingpong mechanism is used to carry the next block of Feature Map of size M0*K0 during the above calculation process; the second block of Feature Map is used to complete the complete output of the next block of M0*Cout... In such a calculation process, the amount of Feature Map matrix carried is The transport capacity of the Weight matrix is ​​K0*Cout, and the total transport capacity is The amount of data moved is no different from that in the second case, but because pingpong cannot be performed, the processor starts to move in the next Feature Map matrix after each calculation is completed. The calculation can only continue after the next Feature Map matrix is ​​moved in. Therefore, the bandwidth from DDR to shared memory will block the calculation and increase the calculation time of the operator.

[0184] ④ The shared memory is large enough to hold two M0*K0 Feature Map matrices and one K0*N0 Weight matrix.

[0185] At this time, the inner loop generates a Feature Map matrix and Weight matrix input of size M0*K0 and K0*N0 per cycle and calculates a complete output of size M0*N0. The outer loop needs Each time after the original Feature Map data is expanded, the Feature Map matrix is ​​kept unchanged, and different Weight matrices are replaced for calculation until the complete output of M0*Cout is completed. The pingpong mechanism is used to carry the next M0*K0 Feature Map block in the above calculation process; the second Feature Map block is used to complete the complete output of the next M0*Cout... In such a calculation process, the amount of Feature Map matrix carried is The transport volume of the Weight matrix is The total transport volume is Because the pingpong mechanism can be carried out, that is, the second Feature Map matrix can be moved in while the first Feature Map matrix is ​​calculated, from this perspective, the Feature Map matrix transportation time can be overlapped with the calculation time to reduce the total calculation time. However, in the process of calculating a single M0*N0 output, the Weight matrix needs to be repeatedly moved. Compared with storing the complete Weight matrix in shared memory, the amount of Weight matrix transportation is increased, and the bandwidth load on and off the chip is increased. Therefore, the limited on-chip memory is used for Feature Mappingpong optimization or to reduce the repeated transportation of the Weight matrix as much as possible. It may be necessary to trade off on specific operators and specific hardware.

[0186] ⑤The shared memory is large enough to hold a Feature Map matrix and a Weight matrix of size M0*K0 and K0*N0.

[0187] At this time, the inner loop generates a Feature Map matrix and Weight matrix input of size M0*K0 and K0*N0 per cycle and calculates a complete output of size M0*N0. The outer loop needs Each time the original Feature Map data is expanded, the Feature Map matrix is ​​kept unchanged, and different Weight matrices are replaced for calculation until the complete output of M0*Cout is completed. The calculation unit waits for the second block of Feature Map matrix to be moved, and then uses the block and different Weight matrices to complete the complete output of the next block of M0*Cout... In this calculation process, the amount of Feature Map matrix moved is The amount of Weight data carried is The total transport volume is The amount of data moved is no different from that in case ③, but because ping-pong cannot be performed, the processor starts moving in the next input data after each calculation is completed, and can continue the calculation only after the next input data is moved in. Therefore, the bandwidth from DDR to shared memory will block the calculation and increase the calculation time of the operator.

[0188] ⑥ The shared memory is not enough to hold a Feature Map matrix and a Weight matrix of size M0*K0 and K0*N0.

[0189] When the shared memory is not enough to hold the input of size M0*K0 and K0*N0, it is necessary to continue to split in the K0 direction. As mentioned above, K0 = Cin*KH*KW. The last two parameters are the size of the convolution window. If it is not convenient to split, then Cin is split until the Feature Map matrix and Weight matrix input of size M0*K0 and K0*N0 can be stored in the shared memory. However, the splitting of Cin will additionally bring about repeated expansion of Feature Map. Feature Map can no longer be kept in the shared memory until it is multiplied with all Weight block matrices. In the next loop, it needs to switch to different Cin blocks. In such a calculation process, the amount of Feature Map matrix to be moved is The transport volume of the Weight matrix is The total transport volume is When the shared memory is not large enough to hold inputs of size M0*K0 and K0*N0, the input needs to be further segmented, which will lead to repeated transfer of the Feature Map matrix or the Weight matrix, greatly increasing the bandwidth overhead from DDR to shared memory.

[0190] After analyzing the above-mentioned situations, it can be found that the capacity of shared memory and the calculation logic directly affect the total amount of data transferred, and the bandwidth speed from DDR to shared memory is much slower than the transfer between shared memories. In particular, when the shared memory is insufficient to store the amount of data required for the calculation of a single cycle by the computing unit, the number of repeated data transfers is increased, the data transfer overhead is increased, the bandwidth overhead of data transfer inside and outside the chip is increased, and the operator operation time is increased. At the same time, it can be found that the img2col operation expands the original Feature Map data required for convolution calculation by KH*KW times, and the size of the Weight matrix remains unchanged. Therefore, in the embodiment of the present application, it is considered to transfer the original data or part of the original data to the shared memory, and use the remaining part of the shared memory for img2col and subsequent calculations.

[0191] Assume that after the coarse-grained reconfigurable processor performs matrix multiplication mapping, it calculates an output matrix or partial sum matrix of size M0*N0 per cycle, where M0 is the product of the dimensions of the output matrix in the H and W directions, and N0 is the C dimension of the output matrix. Similarly, an output matrix of size M0*N0 is obtained, and the required Feature Map original data size can be obtained as M0*K0 / (KH*kW)=M0*Cin, and the Weight data is K0*N0. Compared with the Feature Map matrix after moving img2col, the amount of Feature Map moved is reduced to the previous For the common 3*3 convolution, the amount of Feature Map carried is reduced to 1 / 9 of the original amount.

[0192] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0193] Based on the same inventive concept, the embodiment of the present application also provides a coarse-grained reconfigurable processor convolution operator bandwidth optimization design device for implementing the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more coarse-grained reconfigurable processor convolution operator bandwidth optimization design device embodiments provided below can refer to the limitations of the coarse-grained reconfigurable processor convolution operator bandwidth optimization design method above, and will not be repeated here.

[0194] In an exemplary embodiment, Fig.10 As shown, a coarse-grained reconfigurable processor convolution operator bandwidth optimization design device 1 is provided, comprising: a matrix conversion module 10 and a matrix multiplication calculation module 20, wherein:

[0195] The matrix conversion module 10 is used to perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0196] The matrix multiplication calculation module 20 is used to perform a matrix multiplication operation on the matrix data to obtain a convolution calculation result of the target original data.

[0197] In one embodiment, the matrix conversion module 10 is further used for:

[0198] According to the convolution kernel size and the convolution step size, the original data of the feature map is subjected to matrix transformation processing to obtain the feature matrix corresponding to the original data of the feature map; the weight data is subjected to matrix transformation processing to obtain the weight matrix corresponding to the weight data; the feature matrix corresponding to the original data of the feature map and the weight matrix corresponding to the weight data are determined as the matrix data of the target original data.

[0199] In one embodiment, the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design device 1 further includes:

[0200] A matrix dimension determination module, used to determine the single-cycle output matrix dimension of the computing unit according to the single-cycle computing capability of the computing unit;

[0201] The data handling module is used to handle the target original data from the dynamic random access memory according to the single-cycle output matrix dimension.

[0202] In one embodiment, the data handling module is further used for:

[0203] According to the single-cycle output matrix dimension, the feature map matrix size and the weight matrix size are determined; according to the feature map matrix size and the weight matrix size, the target original data is moved from the dynamic random access memory.

[0204] In one embodiment, the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design device 1 further includes:

[0205] A capability acquisition module is used to acquire the storage capability of the shared memory and the single-cycle computing capability of the computing unit;

[0206] The transport volume determination module is used to determine the feature map transport volume of the feature map original data and the weight transport volume of the weight data according to the storage capacity and the single-cycle computing capacity.

[0207] In one embodiment, the storage capacity is capable of storing complete feature map original data and complete weight data; the above-mentioned handling amount determination module is also used for:

[0208] The data size of the original data of the feature map is determined as the feature map transport volume, and the data size of the weight data is determined as the weight transport volume.

[0209] In one embodiment, the storage capacity is capable of storing feature map original data of a first preset data size and complete weight data; the above-mentioned handling amount determination module is also used for:

[0210] Determine the number of cycles of the matrix multiplication operation based on the complete output data size of the matrix multiplication operation and the single-cycle output matrix dimension; determine the feature map transport volume of the feature map original data based on the number of cycles and the first preset data size; determine the weight transport volume of the weight data based on the data size of the weight data.

[0211] In one embodiment, the transport volume determination module is further used to:

[0212] According to the number of cycles, characteristic graph cycle data related to the characteristic graph original data is obtained; according to the characteristic graph cycle data, the first preset data size and the matrix conversion expansion coefficient, the characteristic graph transport volume of the characteristic graph original data is determined.

[0213] In one embodiment, the storage capacity is capable of storing feature map original data and complete weight data of a second preset data size; the above-mentioned handling amount determination module is also used for:

[0214] Acquire feature map circulation data related to feature map original data; determine the product of the feature map circulation data and the second preset data size as the feature map carrying capacity of the feature map original data; and determine the data size of the weight data as the weight carrying capacity of the weight data.

[0215] In one embodiment, the storage capacity is capable of storing feature map original data of a second preset data size and weight data of a third preset data size; the above-mentioned handling amount determination module is also used for:

[0216] Obtain a preset loop order for the matrix multiplication operation; determine a feature map carrying amount of the feature map original data and a weight carrying amount of the weight data according to the preset loop order, the second preset data size, and the third preset data size.

[0217] In one embodiment, the preset loop order is that the original data of the feature map is fixed and the weight data is looped each time; the above-mentioned transport amount determination module is also used for:

[0218] Acquire feature map cycle data related to the feature map original data; determine the product of the feature map cycle data and the second preset data size as the feature map carrying capacity of the feature map original data; and determine the product of the number of cycles and the third preset data size as the weight carrying capacity of the weight data.

[0219] In one embodiment, the preset loop order is that the weight data is fixed and the original data of the feature map is looped each time; the above-mentioned transport amount determination module is also used for:

[0220] According to the number of cycles of the matrix multiplication operation, weight circulation data related to the weight data is obtained; the product of the number of cycles and the second preset data size is determined as the feature map carrying capacity of the feature map original data; the product of the weight circulation data and the third preset data size is determined as the weight carrying capacity of the weight data.

[0221] In one embodiment, the storage capacity is capable of storing data of the feature map original data of the second preset data size after segmentation on the channel dimension and data of the weight data of the third preset data size after segmentation on the channel dimension; the above-mentioned handling amount determination module is also used for:

[0222] Obtain the number of cycles of the matrix multiplication operation; determine the product of the number of cycles and the second preset data size as the feature map transport volume of the feature map original data, and determine the product of the number of cycles and the third preset data size as the weight transport volume of the weight data.

[0223] Each module in the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0224] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store bandwidth optimization design data of a coarse-grained reconfigurable processor convolution operator. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for optimizing the bandwidth of a coarse-grained reconfigurable processor convolution operator is implemented.

[0225] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0226] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0227] Perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0228] Perform matrix multiplication on the matrix data to obtain the convolution calculation result of the target original data.

[0229] The implementation principles and technical effects of each step implemented by the processor in the embodiment of the present application are similar to the principles of the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design method, and will not be repeated here.

[0230] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0231] Perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0232] Perform matrix multiplication on the matrix data to obtain the convolution calculation result of the target original data.

[0233] The implementation principles and technical effects of the steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design method, and will not be repeated here.

[0234] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0235] Perform matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory;

[0236] Perform matrix multiplication on the matrix data to obtain the convolution calculation result of the target original data.

[0237] The implementation principles and technical effects of the steps implemented when the computer program in the embodiment of the present application is executed by the processor are similar to the principles of the above-mentioned coarse-grained reconfigurable processor convolution operator bandwidth optimization design method, and will not be repeated here.

[0238] It should be noted that the personnel information (including but not limited to personnel equipment information, personnel personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the personnel or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0239] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0240] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0241] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for optimizing bandwidth design of convolution operators in a coarse-grained reconfigurable processor, characterized in that: Applied to a computing unit in a coarse-grained reconfigurable processor, the method comprises: Performing matrix conversion processing on the target original data to obtain matrix data of the target original data; the target original data is obtained by the computing unit from the shared memory; the target original data in the shared memory is moved from the dynamic random access memory; A matrix multiplication operation is performed on the matrix data to obtain a convolution calculation result of the target original data.

2. The method according to claim 1, characterized in that Before performing matrix conversion processing on the target original data to obtain matrix data of the target original data, the method further includes: Determining a single-cycle output matrix dimension of the computing unit according to the single-cycle computing capability of the computing unit; The target original data is moved from the dynamic random access memory according to the single-cycle output matrix dimension.

3. The method according to claim 2, characterized in that The step of transferring the target original data from the dynamic random access memory according to the single-cycle output matrix dimension includes: Determine the size of the feature map matrix and the size of the weight matrix according to the single-cycle output matrix dimension; The target original data is moved from the dynamic random access memory according to the size of the feature map matrix and the size of the weight matrix.

4. The method according to any one of claims 1 to 3, characterized in that: The target original data includes feature map original data and weight data; the method further includes: Acquire the storage capacity of the shared memory and the single-cycle computing capacity of the computing unit; The feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined according to the storage capacity and the single-cycle computing capacity.

5. The method according to claim 4, characterized in that The storage capacity is capable of storing feature map original data and complete weight data of a first preset data size; The determining, according to the storage capacity and the single-cycle computing capacity, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data comprises: Determining the number of cycles of the matrix multiplication operation according to the complete output data size of the matrix multiplication operation and the single-cycle output matrix dimension; Determining a feature map transfer amount of the feature map original data according to the number of cycles and the first preset data size; The weight carrying amount of the weight data is determined according to the data size of the weight data.

6. The method according to claim 5, characterized in that The step of determining the feature map transfer amount of the feature map original data according to the number of cycles and the first preset data size includes: According to the number of cycles, acquiring characteristic graph cycle data related to the characteristic graph original data; The feature map transport volume of the feature map original data is determined according to the feature map circulation data, the first preset data size and the matrix conversion expansion coefficient.

7. The method according to claim 4, characterized in that The storage capacity is capable of storing feature map original data and complete weight data of a second preset data size; The determining, according to the storage capacity and the single-cycle computing capacity, the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data comprises: Acquire characteristic graph cycle data related to the characteristic graph original data; Determine the product of the characteristic graph cycle data and the second preset data size as the characteristic graph carrying amount of the characteristic graph original data; The data size of the weight data is determined as the weight carrying amount of the weight data.

8. The method according to claim 4, characterized in that The storage capacity is capable of storing feature map original data of a second preset data size and weight data of a third preset data size; and determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the storage capacity and the single-cycle computing capacity, including: Obtaining a preset loop order for the matrix multiplication operation; The feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined according to the preset loop order, the second preset data size and the third preset data size.

9. The method according to claim 8, characterized in that The preset loop order is that the feature map original data is fixed and the weight data is looped each time; the determining the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data according to the preset loop order, the second preset data size and the third preset data size includes: Acquire characteristic graph cycle data related to the characteristic graph original data; Determine the product of the characteristic graph cycle data and the second preset data size as the characteristic graph carrying amount of the characteristic graph original data; The product of the number of cycles and the third preset data size is determined as the weighted carrying amount of the weighted data.

10. The method according to claim 8, characterized in that The preset loop order is that the weight data is fixed and the feature map original data is looped each time; the feature map carrying amount of the feature map original data and the weight carrying amount of the weight data are determined according to the preset loop order, the second preset data size and the third preset data size, including: According to the number of cycles of the matrix multiplication operation, obtaining weight cycle data related to the weight data; Determine the product of the number of cycles and the second preset data size as the feature map carrying amount of the feature map original data; The product of the weight circulation data and the third preset data size is determined as the weight carrying amount of the weight data.

Images