A convolution data stream scheduling method, device, medium and equipment
By splitting the input feature map and convolution kernel into multiple channel groups and convolution kernel groups, and outputting them to the multiplier array in a specific order, the problem of data congestion in convolution operations is solved, and the low-power multiplication is achieved Efficient convolution calculation in the machine.
Patent Information
- Application Number
- CN202311849647.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-29
AI Technical Summary
In low-power multipliers with limited bandwidth and tight computing power, convolution operations lead to data congestion when data is transmitted to the multiplier array, making it difficult to effectively perform convolutional calculations.
By dividing the input feature map and convolution kernel into multiple channel groups and convolution kernel groups according to the channel direction, and dividing them into multiple input blocks and convolution kernel blocks according to the calculated block width of the multiplier array, using first and then column The order of the channels is output to the multiplier array for convolution operations, and the large convolution task is converted into small tasks and gradually executed.
It greatly alleviates data congestion, reduces the impact on bandwidth, and supports the deployment of convolutional computing in low-power multipliers with limited bandwidth and tight computing power.
Smart Images

Figure CN117787365B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, medium and equipment for scheduling convolutional data streams. Background Art
[0002] In artificial neural networks, convolution operations can extract local features of input data and perform hierarchical feature learning by stacking multiple convolutional layers. Parameter sharing reduces the number of network parameters and improves computational efficiency.
[0003] Currently, convolution-related data is mainly transmitted to the multiplier array for multiplication and accumulation convolution calculation. This method requires instantaneous transmission of a large amount of data to the multiplier array, which easily causes data congestion, making it difficult to deploy and implement convolution calculations in low-power multipliers with limited bandwidth and tight computing power. Summary of the invention
[0004] The present invention provides a convolution data stream scheduling method, device, medium and equipment to solve or partially solve the technical problem of data congestion caused by excessive calculation data transmitted to a multiplier array.
[0005] In order to solve the above technical problems, the first aspect of the present invention discloses a method for scheduling a convolution data stream, the method comprising:
[0006] The input feature map is divided into C / M input channel groups in sequence according to the channel direction, and in each input channel group, each channel group is divided into a number of input blocks in sequence from left to right according to the calculation block width size required by the multiplier array; in a single input channel group, each input block participates in the convolution operation in sequence from left to right and from top to bottom, and the size of a single input block along the width direction is consistent with the required calculation block width size; wherein C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group;
[0007] The convolution kernels involved in the operation are divided into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is divided into C / M convolution kernel channel groups in the channel direction; where Z represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups;
[0008] The convolution kernel involved in the operation and the input feature map are output to the multiplier array to perform the convolution operation; wherein the convolution kernel involved in the operation is input in units of "channel group", and the input feature map is input in units of "input block".
[0009] Optionally, in a single input channel group, the number of input blocks depends on the output feature map size and the required computational block width size.
[0010] Optionally, the outputting the convolution kernel involved in the operation and the input feature map to the multiplier array to perform the convolution operation specifically includes:
[0011] Within a single input block, the convolution operation is performed by outputting the data to the multiplier array in the order of row, column and channel.
[0012] In a single convolution kernel channel group, the N convolution kernel blocks contained therein are output in columns to the multiplier array to perform convolution operations; wherein the single convolution kernel channel group contains N convolution kernel blocks arranged in a column.
[0013] Optionally, in a single convolution kernel channel group, outputting the N convolution kernel blocks contained therein to the multiplier array in columns to perform a convolution operation specifically includes:
[0014] For a single convolution kernel block, the weight data of each input feature corresponding position in the single convolution kernel block is output to the multiplier array to perform a convolution operation, and the output is traversed in sequence along the row, column, and channel order until the traversal of the single convolution kernel block is completed.
[0015] Optionally, the weight data corresponding to the convolution kernel involved in the operation and the input features corresponding to the input feature map are stored in a cache.
[0016] Optionally, before the input feature map is divided into C / M input channel groups in sequence according to the channel direction, the method further includes:
[0017] The output feature map is divided into Z / N output channel groups in sequence along the channel direction, and the calculation results in the multiplier array are filled in sequence in the order of rows first, columns second, and channels third.
[0018] Optionally, before dividing the output feature map into Z / N output channel groups in sequence along the channel direction, the method further includes:
[0019] Input a size calculation formula in advance according to the input feature map related parameters and the convolution kernel related parameters to calculate the size of the output feature map;
[0020] The size calculation formula is:
[0021]
[0022]
[0023] in, Indicates the left border padding size. represents the input feature map width, Indicates the right border padding size, represents the height of the convolution kernel, Indicates that the convolution kernel expands horizontally. represents the horizontal step size of the convolution, Indicates the top border padding size. represents the input feature map height, Indicates the bottom border padding size. represents the width of the convolution kernel, Indicates that the convolution kernel expands vertically. Represents the vertical stride of the convolution.
[0024] A second aspect of the present invention discloses a convolution data stream scheduling device, comprising:
[0025] The first segmentation module is used to divide the input feature map into C / M input channel groups in sequence according to the channel direction, and in each input channel group, each channel group is divided into a plurality of input blocks in sequence from left to right according to the calculation block width size required by the multiplier array; in a single input channel group, each input block participates in the convolution operation in sequence from left to right and from top to bottom, and the size of a single input block along the width direction is consistent with the required calculation block width size; wherein C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group;
[0026] The second segmentation module is used to divide the convolution kernels involved in the operation into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is sequentially segmented into C / M convolution kernel channel groups according to the channel direction; where Z represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups;
[0027] An output module is used to output the convolution kernel involved in the operation and the input feature map to the multiplier array to perform a convolution operation; wherein the convolution kernel involved in the operation is input in units of "channel group" and the input feature map is input in units of "block".
[0028] According to a third aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.
[0029] According to a fourth aspect of the present invention, a computer device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0030] Through one or more technical solutions of the present invention, the present invention has the following beneficial effects or advantages:
[0031] The present invention provides a scheduling method, device, medium and equipment for convolution data stream, which processes the input feature map and the convolution kernels involved in the operation in batches, converts large convolution tasks into small convolution tasks and gradually outputs them to the multiplier array to perform convolution operations, thereby greatly alleviating data congestion and reducing the impact on bandwidth, and can also support the deployment and implementation of convolution calculations in low-power multipliers with limited bandwidth and tight computing power.
[0032] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0034] Figure 1 A flow chart of a method for scheduling a convolutional data stream according to an embodiment of the present invention is shown;
[0035] Figure 2 A logical diagram of the division of the input feature map, the convolution kernel, and the output feature map in the i-th cycle according to one embodiment of the present invention is shown;
[0036] Figure 3-Figure 4 A schematic diagram showing the position correspondence relationship of input blocks according to an embodiment of the present invention is shown;
[0037] Figure 5 A schematic diagram showing a corresponding relationship of convolution calculation of an input block according to an embodiment of the present invention is shown;
[0038] Figure 6 A schematic diagram of a scheduling device for convolution data stream according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0039] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0040] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0041] In the first aspect, the embodiments of this specification provide a method for scheduling convolution data streams, which optimizes the underlying convolution operation and can be applied to various fields such as artificial intelligence, image recognition, and industry classification. Figure 1 As shown, a convolution data stream scheduling method provided in an embodiment of this specification includes at least the following steps:
[0042] S101, divide the input feature map into C / M input channel groups in sequence according to the channel direction, and in each input channel group, divide each channel group into a number of input blocks from left to right according to the calculation block width size required by the multiplier array.
[0043] Specifically, see Figure 2 , the size of the input feature map is [Wx, Hx, C]. Wx is the width, Hx is the height, and C is the number of channels. In order to reduce the output volume, the input feature map is first divided into C / M input channel groups in the channel direction; where C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group.
[0044] For example, if the input feature map has a size of 6×6 and 12 channels, it can be divided into 4 input channel groups with a size of 6×6 and 3 channels, and the number of channels in each input channel group is 3.
[0045] In a single input channel group, the single input channel group is divided into a plurality of input blocks. The input block division methods of other input channel groups are similar and will not be described in detail.
[0046] It is worth noting that the number of input block divisions for a single input channel group depends on the number of weights inside a single convolution kernel participating in the operation. Each weight data corresponds to one input block division in a single input channel group. For example, for a 3×3 convolution kernel, the number of weights is 9, and the number of input block divisions for a single input channel group is 9. The division method and the number of divided input blocks are the same each time, but the positions of the divided input blocks are different. The positions of the input blocks of a single input channel group are related to the positions of the weight data inside a single convolution kernel participating in the operation. The number of input block divisions, the number of cycles, and the number of weights inside a single convolution kernel participating in the operation for a single input channel group are the same. Figure 3-Figure 4For example, in the first cycle, the first weight data of a single convolution kernel is calculated. Then for the input feature map of S101, the position of the input block divided in a single input channel group is as follows: Figure 3 As shown (the number of channels is not shown). In the second cycle, the second weight data of a single convolution kernel is calculated. Then, for the input feature map of S101, the position of the input block divided from a single input channel group is as follows: Figure 4 (The number of channels is not shown). For a single input channel group, input block partitioning only needs to be performed once per cycle.
[0047] In a single input blocking operation for a single input channel group:
[0048] 1. The number of input block divisions depends on the output feature map size and the computational block width required by the multiplier array, which is: ;in, Indicates the size of the output feature map. For example, if the input feature map size is 6x6, the convolution kernel size is 3x3, and the step size is 1, the output feature map size is 4x4 (it is worth noting that the output feature map size can be calculated in advance. The calculation method of the output feature map size will be described later and will not be repeated here); L represents the calculation block width required by the multiplier array, which is related to the number of multiplier arrays and can also be customized. The value is known. See Figure 3 , when the output feature map size is 4x4, L=2, the number of input block divisions is: .
[0049] 2. The size of the input block is [width L, height 1, number of channels M]. The size L of the input block along the width direction is consistent with the required calculation block width size, the height is 1, and the number of channels M in the depth direction depends on the number of multiplier arrays. For example, if the number of multiplier arrays is 16, then M is 16. It can also take other values without limitation. A single input block participates in the convolution operation in the order of rows, columns, and channels. Each input block participates in the convolution operation in the order from left to right and from top to bottom.
[0050] 3. The position of the input block is related to the position of the single weight data. The corresponding position relationship between the two is determined by the size of the output feature map.
[0051] In the specific implementation process, follow the steps below:
[0052] (1) Calculate the number of elements in the output feature map based on the size of the output feature map. The number of elements indicates the number of related input features corresponding to a single weight data. Related means: having a convolution calculation relationship. For example, the size of the input feature map of a single channel is 6x6, the convolution kernel is 3x3, and the step size is 1, then the size of the output feature map is 4x4, which means that the number of elements in the output feature map is 16, that is, one weight data in the convolution kernel corresponds to 16 related input features. Figure 3 As shown in the figure, the first weight data of the convolution kernel has a convolution calculation relationship with the 16 input features in the shaded area. It is worth noting that the convolution operation of a single weight data and related input features is called a strip operation. The upper limit of the number of elements in the strip operation is determined by the number of multipliers. If there are 16 multipliers, the upper limit of the number of elements calculated by the strip operation is 16 and the lower limit is 8.
[0053] (2) According to the number of elements, determine the number of features of the relevant input features of a single weight data in a single channel. For example, when the number of elements in the output feature map is 16, it means that the number of features of the single weight data that has a convolution calculation relationship in the output feature map is 16.
[0054] (3) Determine the feature location area of the relevant input feature based on the number of elements and the location of the single weight data. For example, for the first weight data of the 3x3 convolution kernel, the location area of its relevant input feature can be found in Figure 3 The shaded areas are the locations of the 16 input features.
[0055] (4) Using the number of features of the relevant input features and the number of input blocks to be divided, the number of features contained in each input block is obtained. For example, based on the 16 input features determined and the 8 input blocks to be divided, it is determined that each input block contains 2 input features.
[0056] (5) According to the number of features contained in each input block, determine the division position of each input block from the feature position area from left to right and from top to bottom. When dividing the input blocks, since each input block needs to contain 2 input features, determine 2 input features from the feature position area from left to right and from top to bottom to form an input block, and you can get the division positions of the 8 input blocks.
[0057] For further explanation, see Figure 4 , it can be seen that the input block division position corresponding to the second weight data of the convolution kernel is also another input block division result in a single input channel group.
[0058] In addition to processing the input feature map into channel groups, the convolution operations involved are also split to further reduce the input amount of the input multiplier array.
[0059] S102, the convolution kernels involved in the operation are divided into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is divided into C / M convolution kernel channel groups in sequence along the channel direction.
[0060] Specifically, the size of the convolution kernel involved in the operation is [width Rx, height Sx, number of channels C, number of convolution kernels Z]. Among them, Z also represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups.
[0061] For further information, see Figure 2 , the Z convolution kernels involved in the operation are divided into Z / N columns of convolution kernel groups, and each column of convolution kernel group contains N convolution kernels. Furthermore, each column of convolution kernel group is divided into C / M convolution kernel channel groups in the channel direction. In a single convolution kernel channel group, N convolution kernel blocks are arranged in a column. Exemplarily, the size of the convolution kernel involved in the operation is [3, 3, 12, number 3], which is first divided into 3 columns of convolution kernels according to the number, with one convolution kernel in each column, and then divided into 4 convolution kernel channel groups in the channel direction, and each convolution kernel channel group contains 3 channels.
[0062] It is worth noting that when dividing the convolution kernel channel group, it is necessary to divide it according to the number of channels M in a single channel group when the input feature map is divided into the input channel group, so that each convolution kernel block in each convolution kernel channel group contains M channels to facilitate subsequent convolution calculations.
[0063] S103, output the convolution kernel and input feature map involved in the operation to the multiplier array to perform the convolution operation.
[0064] When executing the loop, the convolution kernel and input feature map involved in the operation are output to the multiplier array to perform the convolution operation of the i-th cycle. The concept of loop period will be introduced later and will not be repeated here.
[0065] The input feature map is input in units of "input blocks". For details, see Figure 2 , a single input channel group contains several input blocks. In a single input channel group, several input blocks are output to the multiplier array in order from left to right and from top to bottom to perform convolution operations. Within a single input block, the convolution operations are performed in the order of rows, columns, and channels.
[0066] For example, the input feature map is divided into channel groups along the channel direction, each input channel group contains M channels, and then each channel group is divided into several input blocks according to the width direction size L and the height direction size 1. The size of the input block is [width L, height 1, number of channels M].
[0067] When outputting the input feature map, the input feature value multiplier array is output in units of "input block" → "channel group" in sequence.
[0068] For the current input block of the output current input channel group, perform L traversals along the width direction, and then perform M traversals along the channel direction, and sequentially output M L-length data to the multiplier array, indicating that the output of the current input block is completed, and then transfer to the next input block from left to right and from top to bottom, and output and perform convolution operations in the same way. In the current input channel group, each input block is output to the multiplier array from left to right and from top to bottom.
[0069] After all input blocks in the current input channel group are output, the calculation of the current input channel group is completed, and the output of the next input channel group starts in the same output mode. The output of each input channel group is output in sequence according to the division direction of the input channel number until all C / M input channel groups are calculated.
[0070] The convolution kernels involved in the operation are output in units of "channel groups". For details, see Figure 2 , a single convolution kernel channel group contains N convolution kernel blocks arranged in a column. In a single convolution kernel channel group, the N convolution kernel blocks contained therein are output to the multiplier array in columns to perform convolution operations.
[0071] Furthermore, for a single convolution kernel block, the weight data at the corresponding position of each input feature in the single convolution kernel block is output to the multiplier array to perform a convolution operation, and the output is traversed in sequence in the order of row, column, and channel until the traversal of the single convolution kernel block is completed. Among them, the input feature and the weight data at the corresponding position have a convolution operation relationship. Moreover, the corresponding position relationship between the input feature and the weight data is determined by the size of the output feature map. The number of elements of the output feature map is calculated according to the size of the output feature map, and the number of features and the feature position area of the relevant input features are determined according to the number of elements and the location of the single weight data. For example, if the size of the input feature map is 6x6, the convolution kernel is 3x3, and the step size is 1, then the size of the output feature map is 4x4, indicating that the number of elements of the output feature map is 16, that is, one weight data in the convolution kernel corresponds to 16 input features. Figure 3 As shown, the first weight data of the convolution kernel has a convolution operation relationship with the 16 input features in the shaded area.
[0072] The output order of the convolution kernels involved in the calculation is described in detail below.
[0073] In a single convolution kernel channel group, N convolution kernel blocks are output to the multiplier array in columns. For the first convolution kernel block among the N convolution kernel blocks, in the first channel, the output is traversed in the order of rows first and columns later until the last row and the last column, indicating that the convolution kernel block scan output of the first channel is completed. After the convolution kernel block scan output of the first channel is completed, the next channel of the convolution kernel block is traversed in the same way in the channel direction until the convolution kernel block is traversed.
[0074] It is worth noting that although each position in the convolution kernel block will be traversed, only the weight data at the corresponding position of the input feature will be convolutionally operated.
[0075] When the convolution kernel block traversal output is completed, it is transferred to the next convolution kernel block in the N convolution kernel blocks for traversal output by column until the traversal of the N convolution kernel blocks is completed, indicating that the traversal of a single convolution kernel channel group is completed. The traversal method inside each convolution kernel block is similar to the above method, so it will not be repeated here.
[0076] After the traversal of a single convolution kernel channel group is completed, the next convolution kernel channel group is traversed in the channel direction until the traversal of C / M convolution kernel channel groups in a single column of convolution kernel groups is completed.
[0077] After a single column of convolution kernel groups is traversed, the remaining convolution kernel groups can be randomly traversed until the Z / N columns of convolution kernel groups are traversed; or the convolution kernel groups can be traversed one by one in the division direction until the Z / N columns of convolution kernel groups are traversed.
[0078] In an optional implementation, the weight data corresponding to the convolution kernel involved in the operation and the input features corresponding to the input feature map are stored in a cache. Of course, they can also be stored in other memories without limitation.
[0079] When extracting weight data, extract it in the order of the output of the convolution kernel to the multiplier array. Specifically, within the convolution kernel block, extract weight data in the order of rows first, columns second, and channels. Between each convolution kernel block, traverse the convolution kernel blocks one by one from top to bottom by column to extract weight data; between each convolution kernel channel group, traverse the convolution kernel channel group one by one in the switching order of the convolution kernel channel group to extract weight data; between each convolution kernel group, you can randomly traverse the remaining convolution kernel groups to extract weight data, or you can traverse the convolution kernel groups one by one in the division direction to extract weight data until the weight data corresponding to all convolution kernels are traversed and extracted.
[0080] When extracting the input feature map, the input features are extracted in the order in which the input feature map is output to the multiplier array. Specifically, within the input block, the input features are extracted in the order of rows, columns, and channels; between the input blocks, the input features are extracted by traversing the input blocks one by one in the order of columns from left to right; between the input channel groups, the input features are extracted by traversing the input channel groups one by one in the switching order of the input channel groups until the input features of the input feature map that is segmented this time are completely extracted.
[0081] It is worth noting that in a single cycle, the weight data can be used as the basis. After extracting the single weight data at the same position in the convolution kernel block, the corresponding input features can be extracted according to the convolution operation relationship. Similarly, the corresponding single weight data can be extracted according to the convolution operation relationship based on several input blocks in a single input channel group.
[0082] In the convolution calculation process of input feature map and convolution kernel, see Figure 5 , is a matrix multiplication operation between a single input block [width L, height 1, number of channels M] and a convolution kernel block [number of convolution kernels N, number of channels M] in a single convolution kernel channel group, to obtain partial output features corresponding to a single input block, with a size of [width L, height 1, number of channels N]. Fill the partial output features corresponding to a single input block into the output feature map.
[0083] The output feature map is divided into Z / N output channel groups along the channel direction, and the calculation results in the multiplier array are filled in the order of rows, columns, and channels. Figure 2 , each output channel group fills the calculation results of the multiplier array in the width direction and the channel number direction in sequence until all calculation blocks are filled, which means that the output features of this channel group are finished and the next output channel group is filled. Calculate in sequence until all Z / N output channel groups are filled. It is worth noting that the above logic is the logic of the output feature map when it is filled in the i-th cycle, and the output feature maps in the cycle are filled according to the above logic.
[0084] It is worth noting that S101~S103 performs the i-th loop output, and the loop S101~S103 operations are executed repeatedly until the final output feature map is obtained. The loop period is determined by the number of weight data in a single convolution kernel. For example, for a 3×3 convolution kernel with 9 weight data, 9 loop outputs are executed according to the operations of S101~S103. It is worth noting that in each loop, the positions of several input blocks divided in a single input channel group depend on the positions of weight data with a convolution relationship. Of course, all weights can be extracted at one time in S102. When all weight numbers are output, in order to save computing resources, they can be output only once. Of course, they can also be output in a loop together with S101 and S103.
[0085] by Figure 3-Figure 4 For example, in the first cycle, the first weight data of a single convolution kernel is involved in the calculation. Then in S101, when the input feature map is divided into input blocks, the positions of the input blocks divided from a single input channel group are as follows: Figure 3 (The number of channels is not shown). In the second cycle, the second weight data of a single convolution kernel is calculated. Then in S101, when the input feature map is divided into input blocks, the positions of the input blocks divided from a single input channel group are as follows: Figure 4 as shown (channel number not shown).
[0086] For the convenience of describing and explaining the present invention, an example is given below, where the input feature map size is [6, 6, 2], the number of convolution kernels is 2, and the size is [3, 3, 2], which are divided into 2 groups, each with 1 convolution kernel. 9 cycles are performed.
[0087] In the first cycle, the input feature map is divided into Figure 3 The 8 input blocks shown in the figure have a size of [2,1,2], and the data are a1, a2, b1, b2. The convolution kernel channel group contains 1 convolution kernel block with a size of [2,1,2]. After convolution calculation is performed on the data of the first input block and the first weight data A and B in the convolution kernel block, the partial output features corresponding to the first input block are: [2,1,1]. See Figure 5 Then, the partial output features corresponding to the first input block are filled into the corresponding output feature map, and the filling order is along the width direction and the channel direction. Figure 2 According to the above logic, the remaining 7 input blocks are convolved with the first weight data in the convolution kernel block, and the corresponding partial output features are filled into the output feature map in turn.
[0088] In the second cycle, the input feature map is divided into Figure 4 The 8 input blocks shown in the figure are convolved with the second weight data in the convolution kernel block in turn, and the corresponding partial output features are filled into the output feature map in turn, and the filling order is along the width direction and the channel direction.
[0089] When calculating the final output feature map, the data of the second cycle can be directly filled into the output feature map corresponding to the first cycle according to the progressive relationship, and the data of the third cycle can be directly filled into the output feature map corresponding to the second cycle, and so on. After the 9th cycle, the final output feature map can be obtained. It is also possible to fill one output feature map in each single cycle. After the 9th cycle is completed, the data in the output feature maps corresponding to the 9 cycles are accumulated according to the corresponding positions to obtain the final output feature map.
[0090] In an optional implementation, the calculation of the output feature map size is introduced. The size calculation formula is input in advance according to the input feature map related parameters and the convolution kernel related parameters to calculate the size of the output feature map; the size of the output feature map includes [width Ux, height Vx, number of channels Z].
[0091] The size calculation formula is:
[0092]
[0093]
[0094] in, Indicates the left border padding size of the input feature map, represents the input feature map width, Indicates the right border padding size of the input feature map, represents the height of the convolution kernel, Indicates that the convolution kernel expands horizontally. represents the horizontal step size of the convolution, Indicates the top border padding size of the input feature map, represents the input feature map height, Indicates the bottom border padding size of the input feature map, represents the width of the convolution kernel, Indicates that the convolution kernel expands vertically. Represents the vertical stride of the convolution.
[0095] The above is the specific scheduling logic of the scheduling method for the convolution data stream provided by the technical solution of the present invention. In order to more effectively map the convolution operation to the multiplier array, the loop order of the data stream (input feature map, convolution kernel) input to the multiplier array is re-optimized, and the data stream is grouped and processed in batches. By inputting the input feature map and the convolution kernel involved in the operation into the multiplier array in parallel in a pipeline manner, large convolution tasks are converted into small convolution tasks, which can greatly alleviate data congestion and reduce the impact on bandwidth, and can also support the deployment and implementation of convolution calculations in low-power multipliers with limited bandwidth and tight computing power.
[0096] In addition, since there is no data dependency between each loop in the convolution operation, the technical solution of the present invention adjusts the loop output order of the input feature map and the convolution kernel involved in the operation, and effectively divides the loop into multiple sub-loops to maximize the use of the multiplier array, improve the utilization efficiency of the multiplier array, reduce the pressure on the data cache, and reduce the impact on the bandwidth.
[0097] In the second aspect, based on the same inventive concept as the scheduling method of the convolution data stream provided in the embodiment of the first aspect, the embodiment of this specification also provides a scheduling device for the convolution data stream, see Figure 6 ,include:
[0098] The first segmentation module 601 is used to sequentially segment the input feature map into C / M input channel groups according to the channel direction, and in each input channel group, each channel group is sequentially divided into a number of input blocks according to the width size of the calculation block required by the multiplier array from left to right; in a single input channel group, each input block sequentially participates in the convolution operation in the order from left to right and from top to bottom, and the size of a single input block along the width direction is consistent with the required calculation block width size; wherein C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group;
[0099] The second segmentation module 602 is used to divide the convolution kernels involved in the operation into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is sequentially segmented into C / M convolution kernel channel groups according to the channel direction; wherein Z represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups;
[0100] The output module 603 is used to output the convolution kernel involved in the operation and the input feature map to the multiplier array to perform the convolution operation; wherein the convolution kernel involved in the operation is input in units of "channel group", and the input feature map is input in units of "block".
[0101] It should be noted that the scheduling device for the convolution data stream provided in the embodiment of this specification, wherein the specific manner in which each module performs operations has been described in detail in the method embodiment provided in the above-mentioned first aspect, and the specific implementation process can refer to the method embodiment provided in the above-mentioned first aspect, and will not be elaborated here.
[0102] In the third aspect, based on the same inventive concept as the scheduling method for convolution data stream provided in the embodiment of the first aspect mentioned above, an embodiment of the present invention further discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.
[0103] In the fourth aspect, based on the same inventive concept as the scheduling method for convolution data stream provided in the embodiment of the first aspect mentioned above, the embodiment of the present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the methods described above when executing the program.
[0104] Through one or more embodiments of the present invention, the present invention has the following beneficial effects or advantages:
[0105] The present invention provides a scheduling method, device, medium and equipment for convolution data stream, which processes the input feature map and the convolution kernels involved in the operation in batches, converts large convolution tasks into small convolution tasks and gradually outputs them to the multiplier array to perform convolution operations, thereby greatly alleviating data congestion and reducing the impact on bandwidth, and can also support the deployment and implementation of convolution calculations in low-power multipliers with limited bandwidth and tight computing power.
[0106] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the description of the above specific languages is for disclosing the best mode of the present invention.
[0107] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0108] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.
[0109] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.
[0110] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0111] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components of a gateway, proxy server, or system according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0112] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.
Claims
1. A method for scheduling a convolutional data stream, characterized in that: The method comprises: The input feature map is divided into C / M input channel groups in sequence according to the channel direction, and in each input channel group, each channel group is divided into a number of input blocks in sequence from left to right according to the calculation block width size required by the multiplier array; in a single input channel group, each input block participates in the convolution operation in sequence from left to right and from top to bottom, and the size of a single input block along the width direction is consistent with the required calculation block width size; wherein C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group; The output feature map is divided into Z / N output channel groups in sequence along the channel direction, and the calculation results in the multiplier array are filled in sequence in the manner of first row, then column, and then channel; a size calculation formula is input in advance according to the input feature map related parameters and the convolution kernel related parameters to calculate the size of the output feature map; The size calculation formula is: in, represents the width of the output feature map, represents the height of the output feature map, Indicates the left border padding size. represents the input feature map width, Indicates the right border padding size, represents the height of the convolution kernel, Indicates that the convolution kernel expands horizontally. represents the horizontal step size of the convolution, Indicates the top border padding size. represents the input feature map height, Indicates the bottom border padding size. represents the width of the convolution kernel, Indicates that the convolution kernel expands vertically. Represents the vertical step length of the convolution; The convolution kernels involved in the operation are divided into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is divided into C / M convolution kernel channel groups in the channel direction; where Z represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups; The convolution kernel involved in the operation and the input feature map are output to the multiplier array to perform the convolution operation; wherein, the convolution kernel involved in the operation is input in units of "channel group", and the input feature map is input in units of "input block".
2. The method according to claim 1, characterized in that In a single input channel group, the number of input blocks depends on the output feature map size and the required computational block width size.
3. The method according to claim 1, characterized in that Outputting the convolution kernel involved in the operation and the input feature map to the multiplier array to perform the convolution operation specifically includes: Within a single input block, the convolution operation is performed by outputting the data to the multiplier array in the order of row, column and channel. In a single convolution kernel channel group, the N convolution kernel blocks contained therein are output in columns to the multiplier array to perform convolution operations; wherein the single convolution kernel channel group contains N convolution kernel blocks arranged in a column.
4. The method according to claim 3, characterized in that The method of outputting the N convolution kernel blocks contained in a single convolution kernel channel group to the multiplier array in columns to perform convolution operation specifically includes: For a single convolution kernel block, the weight data of each input feature corresponding position in the single convolution kernel block is output to the multiplier array to perform a convolution operation, and the output is traversed in sequence along the row, column, and channel order until the traversal of the single convolution kernel block is completed.
5. The method according to claim 3 or 4, characterized in that The weight data corresponding to the convolution kernel involved in the operation and the input features corresponding to the input feature map are stored in a cache.
6. A convolution data stream scheduling device, characterized in that: include: The first segmentation module is used to sequentially segment the input feature map into C / M input channel groups according to the channel direction, and in each input channel group, each channel group is sequentially divided into a number of input blocks according to the width size of the calculation block required by the multiplier array from left to right; in a single input channel group, each input block sequentially participates in the convolution operation in the order from left to right and from top to bottom, and the size of a single input block along the width direction is consistent with the required calculation block width size; wherein C represents the number of channels of the input feature map, and M represents the number of channels in a single channel group; it is also used to sequentially segment the output feature map into Z / N output channel groups along the channel direction, and sequentially fill the calculation results in the multiplier array in the manner of first row, then column, and then channel; input the size calculation formula in advance according to the input feature map related parameters and the convolution kernel related parameters to calculate the size of the output feature map; The size calculation formula is: in, represents the width of the output feature map, represents the height of the output feature map, Indicates the left border padding size. represents the input feature map width, Indicates the right border padding size, represents the height of the convolution kernel, Indicates that the convolution kernel expands horizontally. represents the horizontal step size of the convolution, Indicates the top border padding size. represents the input feature map height, Indicates the bottom border padding size. represents the width of the convolution kernel, Indicates that the convolution kernel expands vertically. Represents the vertical step length of the convolution; The second segmentation module is used to divide the convolution kernels involved in the operation into Z / N columns of convolution kernel groups, and each column of convolution kernel groups is sequentially segmented into C / M convolution kernel channel groups according to the channel direction; where Z represents the number of channels of the output feature map, and N represents the number of convolution kernels in a single column of convolution kernel groups; An output module is used to output the convolution kernel involved in the operation and the input feature map to the multiplier array to perform a convolution operation; wherein the convolution kernel involved in the operation is input in units of "channel group" and the input feature map is input in units of "block".
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Convolution calculation data flow mapping method and device
CN112541565A
Data processing device, data processing method and related product
CN113850377A
Processing method and processing device for convolutional layer of three-dimensional convolutional neural network
CN115034360A
Systolic array unit and systolic array structure suitable for weight reuse neural network
CN116702851A
Computing device, method for implementing convolution operation by using computing device, and related product
WO2023045638A1
