Deconvolution calculation method, hardware accelerator, device and readable storage medium

Abstract

The present application discloses a method, hardware accelerator, device and readable storage medium for deconvolution calculation, which includes obtaining multiple input blocks, transforming the data in the deconvolution kernel according to a first pre-matrix, obtaining a deconvolution kernel matrix, transforming the data in the multiple input blocks according to a second pre-matrix, obtaining multiple input matrices, and then multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, and accumulating the data of all layer data matrices in any intermediate matrix by channel to obtain multiple accumulation matrices. According to the post-matrix, the data in the multiple accumulation matrices are transformed respectively to obtain multiple output blocks. The multiple output blocks are arranged in sequence into an output feature map to obtain the deconvolution calculation result. In the above calculation process, a large number of zeros are not inserted into the original input feature map, which effectively improves the calculation efficiency.

Description

Technical Field

[0001] The present application relates to the field of deep neural network technology, and in particular to a deconvolution calculation method, hardware accelerator, device, and readable storage medium. Background Art

[0002] When using a hardware accelerator to perform deconvolution calculations on the input feature map, a large number of zeros are usually inserted into the data of the original input feature map to expand it to obtain a new input feature map. Then, a window of the same size as the deconvolution kernel is used to translate and slide on the new input feature map according to a preset step size. Each time it slides, the data covered by the window is multiplied by the data of the deconvolution kernel and then summed, and the summed result is output. After the window slides over all the data on the new input feature map, the summed results obtained are the final deconvolution calculation results.

[0003] The calculation process is shown in Figure 1 A large number of zeros have been inserted into the data of the original input feature map. The blank grid represents the inserted zeros, the black grid represents the data of the original input feature map, and the gray grid represents the window. It can be seen that the window currently only contains the data in the two original input feature maps, and the rest are zero. When multiplying the data in the window with the data in the deconvolution kernel, these zeros will cause many multiplication operations to be invalid operations. In the entire deconvolution calculation process, there will be a large number of invalid operations, which will seriously affect the computing efficiency of the hardware accelerator. Summary of the Invention

[0004] In order to improve the deconvolution calculation efficiency of the hardware accelerator, the present application discloses a deconvolution calculation method, a hardware accelerator, a device and a readable storage medium through the following embodiments.

[0005] The first aspect of the present application discloses a deconvolution calculation method, comprising:

[0006] Acquire multiple input blocks, each of which is a data block covered by a preset sliding window after each sliding in the original input feature map, wherein the original input feature map includes multiple layers of feature maps, and correspondingly, each of the input blocks includes multiple layers of data, and the size of the sliding window is set according to the size of the deconvolution kernel and the deconvolution step size;

[0007] transforming the data in the deconvolution kernel according to a first pre-matrix to obtain a deconvolution kernel matrix, wherein the size of the first pre-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution;

[0008] transforming the data in the plurality of input blocks according to a second pre-matrix to obtain a plurality of input matrices, wherein each of the input matrices includes a multi-layer data matrix, and a size of the second pre-matrix is pre-set according to a size of the deconvolution kernel and a deconvolution step size;

[0009] Multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, wherein each of the intermediate matrices includes multiple layers of data matrices;

[0010] Acquire multiple accumulation matrices, wherein any accumulation matrix is a single-layer data matrix obtained by accumulating data of all layers of data matrices in the corresponding intermediate matrix according to channels;

[0011] According to a post-matrix, the data in the plurality of accumulation matrices are transformed respectively to obtain a plurality of output blocks, wherein the size of the post-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution;

[0012] Arrange the multiple output blocks in sequence into an output feature map to obtain a deconvolution calculation result.

[0013] Optionally, before obtaining the multiple input blocks, the method further includes:

[0014] The size of the sliding window is obtained by the following formula:

[0015]

[0016] D = I1 × I1;

[0017] Wherein, D represents the size of the sliding window, k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is a mathematical symbol used to obtain the smallest integer that is not less than the result of the calculation within the symbol.

[0018] The size of the first pre-matrix is obtained by the following formula:

[0019] m1=k+s-2;

[0020] H = m1 × k;

[0021] Wherein, H represents the size of the first pre-matrix;

[0022] The size of the second pre-matrix is obtained by the following formula:

[0023] P = I1 × m1;

[0024] Wherein, P represents the size of the second pre-matrix;

[0025] The size of the post-matrix is obtained by the following formula:

[0026] O1=1×s;

[0027] Q = m1 × O1;

[0028] Wherein, Q represents the size of the post-matrix.

[0029] Optionally, before obtaining the multiple input blocks, the method further includes:

[0030] The size of the sliding window is obtained by the following formula:

[0031]

[0032] D = I2 × I2;

[0033] Wherein, D represents the size of the sliding window, k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is a mathematical symbol used to obtain the smallest integer that is not less than the result of the calculation within the symbol.

[0034] The size of the first pre-matrix is obtained by the following formula:

[0035] m2=k+2×s-2;

[0036] H = m2 × k;

[0037] Wherein, H represents the size of the first pre-matrix;

[0038] The size of the second pre-matrix is obtained by the following formula:

[0039] P = I2 × m2;

[0040] Wherein, P represents the size of the second pre-matrix;

[0041] The size of the post-matrix is obtained by the following formula:

[0042] O2=2×s;

[0043] Q = m2 × O2;

[0044] Wherein, Q represents the size of the post-matrix.

[0045] Optionally, transforming the data in the deconvolution kernel according to the first pre-matrix to obtain the deconvolution kernel matrix includes:

[0046] Multiplying the first pre-matrix, the deconvolution kernel, and the transposed matrix of the first pre-matrix in sequence to obtain the deconvolution kernel matrix;

[0047] The step of transforming the data in the plurality of input blocks according to the second pre-matrix to obtain a plurality of input matrices includes:

[0048] Multiplying the transposed matrix of the second pre-matrix, any of the input blocks, and the second pre-matrix in sequence to obtain a corresponding input matrix;

[0049] The step of transforming the data in the plurality of accumulation matrices according to the post-matrix to obtain a plurality of output blocks comprises:

[0050] The transposed matrix of the post-matrix, any of the accumulation matrices, and the post-matrix are multiplied in sequence to obtain a corresponding output block.

[0051] A second aspect of the present application discloses a hardware accelerator for deconvolution calculation, wherein the hardware accelerator is used to execute the deconvolution calculation method described in the first aspect of the present application, and the hardware accelerator includes:

[0052] An input module, comprising a plurality of parallel input channels, for inputting the data in the deconvolution kernel matrix into the multiplication module in parallel, and for inputting the data of each layer in the input block into the pre-processing module in sequence, wherein only one input block is input in a single deconvolution calculation process;

[0053] A pre-processing module, comprising a deconvolution pre-processing unit, wherein the deconvolution pre-processing unit comprises a plurality of deconvolution pre-adders, configured to group and add data of any layer in the acquired input block to acquire an input matrix, and then sequentially input each layer of data matrix in the input matrix to the multiplication module;

[0054] A multiplication module, comprising a plurality of weight multipliers, for respectively multiplying the deconvolution kernel matrix with each layer of the data matrix in the input matrix to obtain an intermediate matrix, and inputting each layer of the data matrix of the intermediate matrix into the accumulation module in sequence;

[0055] An accumulation module, comprising an addition tree and a plurality of registers, for accumulating all layer data of the intermediate matrix by channel, obtaining an accumulation matrix, and then inputting the accumulation matrix into a post-processing module;

[0056] A post-processing module, comprising a deconvolution post-processing unit, wherein the deconvolution post-processing unit comprises a plurality of deconvolution post-adders, configured to group and add the data in the accumulation matrix to obtain an output block;

[0057] The output module includes a plurality of parallel output channels, and is used to output the data in the output block in parallel.

[0058] Optionally, the input module includes 36 parallel input channels;

[0059] The deconvolution pre-processing unit includes 13 deconvolution pre-adders;

[0060] The multiplication module includes 36 weight multipliers;

[0061] The deconvolution post-processing unit includes 60 deconvolution post-adders;

[0062] The output module includes 16 parallel output channels.

[0063] Optionally, the pre-processing module further includes: a convolution pre-processing unit and a pre-selector, the convolution pre-processing unit including a plurality of convolution pre-adders, and the pre-selector is used to select, according to current computing requirements, to input the output result of the input module into the deconvolution pre-processing unit or the convolution pre-processing unit;

[0064] The post-processing module also includes: a convolution post-processing unit and a post-selector, the convolution post-processing unit includes multiple convolution post-adders, and the post-selector is used to select the output result of the accumulation module to be input into the deconvolution post-processing unit or the convolution post-processing unit according to the current calculation requirements.

[0065] Optionally, the input module includes 36 parallel input channels;

[0066] The convolution pre-processing unit includes 180 convolution pre-adders;

[0067] The deconvolution pre-processing unit includes 13 deconvolution pre-adders;

[0068] The multiplication module includes 36 weight multipliers;

[0069] The convolution post-processing unit includes 100 convolution post-adders;

[0070] The deconvolution post-processing unit includes 60 deconvolution post-adders;

[0071] The output module includes 16 parallel output channels.

[0072] A third aspect of the present application discloses a computer device, comprising:

[0073] memory for storing computer programs;

[0074] A processor is used to implement the steps of the deconvolution calculation method as described in the first aspect of the present application when executing the computer program.

[0075] The fourth aspect of the present application discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is processed and executed, the steps of the deconvolution calculation method described in the first aspect of the present application are implemented.

[0076] The present application discloses a method, hardware accelerator, device and readable storage medium for deconvolution calculation, which includes obtaining multiple input blocks, transforming the data in the deconvolution kernel according to a first pre-matrix, obtaining a deconvolution kernel matrix, transforming the data in the multiple input blocks according to a second pre-matrix, obtaining multiple input matrices, and then multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, and accumulating the data of all layer data matrices in any intermediate matrix by channel to obtain multiple accumulation matrices. According to the post-matrix, the data in the multiple accumulation matrices are transformed respectively to obtain multiple output blocks. The multiple output blocks are arranged in sequence into an output feature map to obtain the deconvolution calculation result. Before the calculation, the above method does not need to insert a large number of zeros into the original input feature map, so there is no invalid operation, which effectively improves the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0078] Figure 1 This is a schematic diagram of the existing deconvolution calculation principle;

[0079] Figure 2 A schematic diagram of the workflow of a deconvolution calculation method disclosed in an embodiment of the present application;

[0080] Figure 3 A schematic diagram of the process of first-order deconvolution calculation disclosed in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of the flow of the second-order deconvolution calculation disclosed in the embodiments of this application;

[0082] Figure 5 A schematic diagram of the structure of a hardware accelerator for deconvolution calculation disclosed in an embodiment of the present application;

[0083] Figure 6 A schematic diagram of the structure of another hardware accelerator for deconvolution calculation disclosed in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of the structures of a convolution pre-processing unit and a deconvolution pre-processing unit in the hardware accelerator for deconvolution calculation disclosed in an embodiment of the present application;

[0085] Figure 8 This is a schematic structural diagram of a convolution post-processing unit and a deconvolution post-processing unit in the hardware accelerator for deconvolution calculation disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to improve the deconvolution calculation efficiency of the hardware accelerator, the present application discloses a deconvolution calculation method, a hardware accelerator, a device and a readable storage medium through the following embodiments.

[0087] The first embodiment of the present application discloses a method for deconvolution calculation, see Figure 2 The workflow diagram shown in FIG. 1 includes:

[0088] Step S11, obtaining multiple input blocks, wherein the multiple input blocks are data blocks covered by a preset sliding window after each sliding in the original input feature map, wherein the original input feature map includes multiple layers of feature maps, and correspondingly, each of the input blocks includes multiple layers of data, and the size of the sliding window is set according to the size of the deconvolution kernel and the step size of the deconvolution.

[0089] Step S12: transforming the data in the deconvolution kernel according to a first pre-matrix to obtain a deconvolution kernel matrix, wherein the size of the first pre-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution.

[0090] Specifically, the first pre-matrix, the deconvolution kernel, and the transposed matrix of the first pre-matrix are multiplied in sequence to transform the data in the deconvolution kernel and obtain the deconvolution kernel matrix.

[0091] Step S13: transform the data in the multiple input blocks according to the second pre-matrix to obtain multiple input matrices, wherein each of the input matrices includes a multi-layer data matrix, and the size of the second pre-matrix is pre-set according to the size of the deconvolution kernel and the deconvolution step size.

[0092] Specifically, the transposed matrix of the second pre-matrix, any of the input blocks, and the second pre-matrix are multiplied in sequence to transform the data in the input block and obtain the corresponding input matrix.

[0093] Step S14: multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, wherein each of the intermediate matrices includes multiple layers of data matrices.

[0094] Step S15 , obtaining a plurality of accumulation matrices, wherein any of the accumulation matrices is a single-layer data matrix obtained by accumulating the data of all layers of data matrices in the corresponding intermediate matrix by channel.

[0095] Step S16: transform the data in the multiple accumulation matrices respectively according to the post-matrix to obtain multiple output blocks. The size of the post-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution.

[0096] Specifically, the transposed matrix of the post-matrix, any of the accumulation matrices, and the post-matrix are multiplied in sequence to transform the data in the accumulation matrix and obtain the corresponding output block.

[0097] Step S17: Arrange the multiple output blocks in sequence into an output feature map to obtain a deconvolution calculation result.

[0098] In the actual calculation process, there may be multiple deconvolution kernels, each of which includes multiple layers of data. The deconvolution calculation method disclosed in the above embodiment is for the calculation between one deconvolution kernel and multiple input blocks, and the resulting output feature map is one layer. After the calculation is completed for multiple deconvolution kernels according to the above method, the resulting output feature map will have multiple layers.

[0099] The above embodiment discloses a method for deconvolution calculation, which includes obtaining multiple input blocks, transforming the data in the deconvolution kernel according to a first pre-matrix, obtaining a deconvolution kernel matrix, transforming the data in the multiple input blocks according to a second pre-matrix, obtaining multiple input matrices, and then multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, and accumulating the data of all layer data matrices in any intermediate matrix by channel to obtain multiple accumulation matrices. According to the post-matrix, the data in the multiple accumulation matrices are transformed respectively to obtain multiple output blocks. The multiple output blocks are arranged in sequence into an output feature map to obtain the deconvolution calculation result. Before the calculation, the above method does not insert a large number of zeros into the original input feature map, so there is no invalid operation, which effectively improves the calculation efficiency.

[0100] The specific operation process of the deconvolution calculation method disclosed in the first embodiment of the present application can be expressed by the following formula:

[0101] Y=Q T [(HgH T )⊙(P T .dP)]Q

[0102] Among them, Y is the final output result, that is, the output block, whose size is O1×O1, d is the input block, whose size is I1×I1, g is the deconvolution kernel, whose size is k×k, ⊙ represents the corresponding multiplication of matrix elements, and the sizes of the two multiplied matrices are both m1×m1, H is the first pre-matrix, H T is the transposed matrix of the first pre-matrix, the size of H is m1×k, HT The size is k×m1, P is the second pre-matrix, P T is the transposed matrix of the second pre-matrix, the size of P is I1×m1, P T The size is m1×I1, Q is the post-matrix, Q T is the transposed matrix of the post-matrix, the size of Q is m1×O1, Q T The size is O1×m1.

[0103] In response to different computational requirements, the deconvolution calculation method disclosed in the first embodiment of the present application includes a first-order deconvolution calculation method and a second-order deconvolution calculation method. The calculation steps of these two methods are the same, except that the parameters used in the calculation process (the size and sliding step of the sliding window, the first pre-matrix, the second pre-matrix, and the post-matrix) are different.

[0104] In one embodiment, the first-order deconvolution calculation method can be expressed as T[(1×s) 2 ,k 2 ], the sliding step of the sliding window is 1, s is the step size of the deconvolution, and k is the size of the deconvolution kernel, for example, T[(1×2) 2 ,3 2 ] indicates that during the calculation process, the deconvolution step size is 2 and the deconvolution kernel size is 3×3.

[0105] The first-order deconvolution calculation method also includes the following steps before obtaining multiple input blocks:

[0106] The size of the sliding window is obtained by the following formula:

[0107]

[0108] D=I1×I1.

[0109] Where D represents the size of the sliding window (consistent with the size of the input block), k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is a mathematical symbol used to obtain the smallest integer that is not less than the result of the calculation within the symbol.

[0110] The size of the first pre-matrix is obtained by the following formula:

[0111] m1=k+s-2.

[0112] H=m1×k.

[0113] Wherein, H represents the size of the first pre-matrix.

[0114] The size of the second pre-matrix is obtained by the following formula:

[0115] P=I1×m1.

[0116] Wherein, P represents the size of the second pre-matrix.

[0117] The size of the post-matrix is obtained by the following formula:

[0118] O1=1×s.

[0119] Q=m1×O1.

[0120] Wherein, Q represents the size of the post-matrix.

[0121] It should be noted that during the first-order deconvolution calculation process, those skilled in the art can use the first pre-matrix to transform the data in the deconvolution kernel according to a certain transformation rule. These transformation rules can be pre-set by those skilled in the art based on actual needs. Therefore, when the size of the first pre-matrix is fixed, those skilled in the art can set the values of each element in the first pre-matrix. Similarly, the values of each element in the second pre-matrix and the post-matrix can also be pre-set in advance.

[0122] As an example, if the deconvolution step size is 2 (s=2), the deconvolution kernel size is 3×3 (k=3), that is, for T[(1×2) 2 ,3 2 During the calculation process, the size of the first pre-matrix is 3×3, the size of the second pre-matrix is 2×3, and the size of the post-matrix is 3×2. The first pre-matrix, the second pre-matrix, and the post-matrix can be set to:

[0123]

[0124]

[0125]

[0126] If the deconvolution step size is 2 (s = 2), the deconvolution kernel size is 5 × 5 (k = 5), that is, for T[(1 × 2) 2 ,5 2 During the calculation process, the size of the first pre-matrix is 5×5, the size of the second pre-matrix is 3×5, and the size of the post-matrix is 5×2. The first pre-matrix, the second pre-matrix, and the post-matrix can be set to:

[0127]

[0128]

[0129]

[0130] As an example, Figure 3 Describes the process of first-order deconvolution calculation, which is T[(1×2) 2 ,3 2 ], where the sliding step of the sliding window is 1, that is, the moving step of the adjacent input block (S i1 ) is 1, the sliding window size is 2×2, and the size of the input block obtained by sliding once is also 2×2. During the entire deconvolution calculation process, multiple calculations need to be performed, each time only for one input block. The output blocks are arranged in sequence to obtain the output feature map.

[0131] In another embodiment, the second-order deconvolution calculation method can be expressed as T[(2×s) 2 ,k 2 ], the sliding step of the sliding window is 2, s is the step size of the deconvolution, and k is the size of the deconvolution kernel, for example, T[(2×2) 2 ,3 2 ] indicates that during the calculation process, the deconvolution step size is 2 and the deconvolution kernel size is 3×3.

[0132] The second-order deconvolution calculation method also includes the following steps before obtaining multiple input blocks:

[0133] The size of the sliding window is obtained by the following formula:

[0134]

[0135] D=I2×I2.

[0136] Where D represents the size of the sliding window (consistent with the size of the input block), k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is a mathematical symbol used to obtain the smallest integer that is not less than the result of the calculation within the symbol.

[0137] The size of the first pre-matrix is obtained by the following formula:

[0138] m2=k+2×s-2.

[0139] H=m2×k.

[0140] Wherein, H represents the size of the first pre-matrix.

[0141] The size of the second pre-matrix is obtained by the following formula:

[0142] P=I2×m2.

[0143] Wherein, P represents the size of the second pre-matrix.

[0144] The size of the post-matrix is obtained by the following formula:

[0145] O2=2×s.

[0146] Q = m2 × O2.

[0147] Wherein, Q represents the size of the post-matrix.

[0148] It should be noted that, similar to the first-order deconvolution calculation process, in the second-order deconvolution calculation process, those skilled in the art can use the first pre-matrix to transform the data in the deconvolution kernel according to certain transformation rules. These transformation rules can be pre-set by those skilled in the art based on actual needs. Therefore, when the size of the first pre-matrix is fixed, those skilled in the art can set the values of each element in the first pre-matrix. Similarly, the values of each element in the second pre-matrix and the post-matrix can be preset in advance.

[0149] As an example, if the deconvolution step size is 2 (s=2), the deconvolution kernel size is 3×3 (k=3), that is, for T[(2×2) 2 ,3 2 During the calculation process, the size of the first pre-matrix is 5×3, the size of the second pre-matrix is 3×5, and the size of the post-matrix is 5×4. The first pre-matrix, the second pre-matrix, and the post-matrix can be set to:

[0150]

[0151]

[0152]

[0153] As an example, Figure 4 Describes the process of second-order deconvolution calculation, which is T[(2×2) 2 ,3 2 ], where the sliding step of the sliding window is 2, that is, the moving step of the adjacent input blocks (S i2 ) is 2, the sliding window size is 3×3, and the size of the input block obtained by sliding once is also 3×3. During the entire deconvolution calculation process, multiple calculations are required, each time only for one input block. The obtained output blocks are arranged in sequence to obtain the output feature map.

[0154] Experiments show that T[(1×2) 2 ,3 2 ] During the calculation process, for the 4 data in an input block, 9 (3×3) multiplication operations are required to obtain 4 output values in an output block. 2,3 2 During the calculation process, 25 (5×5) multiplication operations are required for each of the 9 data in an input block to obtain 16 output values in an output block. Existing zero-insertion methods only produce one output value after performing 9 multiplication operations for each of the 9 data in an input block.

[0155] If you want to get 16 output values, T[(1×2) 2 ,3 2 ]The calculation method needs to perform 36 multiplication operations in total, T[(2×2) 2 ,3 2 The calculation method needs to perform 25 multiplication operations in total, while the existing zero insertion rule needs to perform 144 multiplication operations. Therefore, the deconvolution calculation method disclosed in this embodiment greatly improves the calculation efficiency.

[0156] The second embodiment of the present application discloses a hardware accelerator for deconvolution calculation, which is used to execute the deconvolution calculation method described in the first embodiment of the present application. Figure 5 The structural diagram, the hardware accelerator includes:

[0157] The input module includes multiple parallel input channels, which are used to input the data in the deconvolution kernel matrix into the multiplication module in parallel, and to input each layer of data in the input block into the pre-processing module in sequence, wherein only one input block is input in a single deconvolution calculation process.

[0158] In one implementation, the hardware accelerator is further provided with a deconvolution kernel processing module, which is used to transform the data in the deconvolution kernel, obtain the deconvolution kernel matrix, and then input the data in the deconvolution kernel matrix to the multiplication module through the input module.

[0159] In another implementation, the deconvolution kernel processing module is not set in the hardware accelerator. This module is set outside the hardware accelerator. It transforms the data in the deconvolution kernel in advance, and then stores the transformed data in the deconvolution kernel matrix in the multiplication module in advance through the input module. This implementation can maximize the savings of computing resources and storage resources of the hardware accelerator.

[0160] The pre-processing module includes a deconvolution pre-processing unit, which includes multiple deconvolution pre-adders for grouping and adding any layer of data in the acquired input block to obtain an input matrix, and then inputting each layer of data matrix in the input matrix into the multiplication module in sequence.

[0161] A multiplication module includes multiple weight multipliers, which are used to multiply the deconvolution kernel matrix with each layer of data matrix in the input matrix element by element, obtain an intermediate matrix, and input each layer of data matrix of the intermediate matrix into the accumulation module in sequence.

[0162] During the actual calculation process, the deconvolution kernel matrix includes multiple layers of data, and the multiplication module is used to multiply each layer of data of the deconvolution kernel matrix with each layer of data matrix in the input matrix one by one to obtain an intermediate matrix.

[0163] The accumulation module includes an addition tree and multiple registers, and is used to accumulate all layer data of the intermediate matrix channel by channel to obtain an accumulation matrix, and then input the accumulation matrix to the post-processing module.

[0164] The post-processing module includes a deconvolution post-processing unit, and the deconvolution post-processing unit includes a plurality of deconvolution post-adders, which are used to group and add the data in the accumulation matrix to obtain an output block.

[0165] The output module includes a plurality of parallel output channels, and is used to output the data in the output block in parallel.

[0166] Furthermore, the input module includes 36 parallel input channels for parallel input of data of 36 input blocks and data of 36 deconvolution kernel matrices.

[0167] The deconvolution pre-processing unit includes 13 deconvolution pre-adders, which are used to group 36 parallel input data according to certain rules and perform shifting or addition.

[0168] The multiplication module includes 36 weight multipliers, which are used to multiply the 36 output data of the pre-processing module and the data of 36 deconvolution kernel matrices in a certain order.

[0169] The deconvolution post-processing unit includes 60 deconvolution post-adders, which are used to group the obtained 36 data according to certain rules and perform shifting or addition.

[0170] The output module includes 16 parallel output channels for outputting 16 output data of the deconvolution post-processing unit in parallel.

[0171] In practical applications, those skilled in the art can set the number of adders or weight multipliers in each module and the connection relationship between them according to actual needs, which will not be repeated here.

[0172] In one embodiment, the present application discloses a reconfigurable hardware accelerator that, in addition to being able to perform deconvolution calculations, can also flexibly choose to perform convolution operations or deconvolution operations according to application scenarios.

[0173] See also Figure 6 The pre-processing module also includes: a convolution pre-processing unit and a pre-selector. The convolution pre-processing unit includes multiple convolution pre-adders. The pre-selector is used to select the output result of the input module to be input into the deconvolution pre-processing unit or the convolution pre-processing unit according to the current calculation requirements.

[0174] The post-processing module also includes: a convolution post-processing unit and a post-selector, the convolution post-processing unit includes multiple convolution post-adders, and the post-selector is used to select the output result of the accumulation module to be input into the deconvolution post-processing unit or the convolution post-processing unit according to the current calculation requirements.

[0175] Furthermore, the input module includes 36 parallel input channels, the output module includes 16 parallel output channels, and the multiplication module includes 36 weight multipliers.

[0176] See also Figure 7 The convolution pre-processing unit includes 180 convolution pre-adders. The deconvolution pre-processing unit includes 13 deconvolution pre-adders.

[0177] See also Figure 8 The convolution post-processing unit includes 100 convolution post-adders. The deconvolution post-processing unit includes 60 deconvolution post-adders.

[0178] The following describes the workflow of the hardware accelerator disclosed in this embodiment with the help of a simple example:

[0179] The input module feeds the pre-calculated 36 deconvolution kernel matrix data into the multiplication module. The pre-selector selects whether to perform convolution or deconvolution calculations based on the configuration. If convolution is performed, the input module feeds the 36 input data into the convolution pre-processing unit, while the deconvolution pre-processing unit receives an input of 0. If deconvolution is performed, the input module feeds the 36 input data into the deconvolution pre-processing unit, while the convolution pre-processing unit receives an input of 0.

[0180] In the pre-processing module, the convolution pre-processing unit divides the 36 input data into six groups of six data points each, then performs a certain amount of addition or shifting. The deconvolution pre-processing unit takes the first nine input data points and performs a certain amount of addition. After these two units calculate the results, the corresponding units send the output data to the multiplication module. If the calculation is a convolution, the output data of the convolution pre-processing unit is sent to the multiplication module; if the calculation is a deconvolution, the output data of the deconvolution pre-processing unit is sent to the multiplication module.

[0181] In the multiplication module, the data sent from the pre-processing module is multiplied one-to-one with the pre-calculated deconvolution kernel matrix data, and the result is then sent to the post-processing module. The post-processing selector selects whether to perform convolution or deconvolution calculations based on the configuration. If convolution is performed, the result is sent to the convolution post-processing unit, and the deconvolution post-processing unit inputs 0; if deconvolution is performed, the result is sent to the deconvolution post-processing unit, and the convolution post-processing unit inputs 0.

[0182] In the post-processing module, the convolution post-processing unit divides the data fed from the multiplication module into six groups of six, then performs a certain amount of addition or shifting. The deconvolution post-processing unit divides the data fed from the multiplication module into four groups of nine, then performs a certain amount of addition. After these two units calculate the final results, the corresponding unit outputs the calculation results. If the calculation is a convolution, the calculation result of the convolution post-processing unit is output; if the calculation is a deconvolution, the calculation result of the deconvolution post-processing unit is output.

[0183] In the output module, the output module outputs 16 output results in parallel at a time.

[0184] A third embodiment of the present application discloses a computer device, including:

[0185] Memory for storing computer programs.

[0186] A processor is used to implement the steps of the deconvolution calculation method as described in the first embodiment of the present application when executing the computer program.

[0187] The fourth embodiment of the present application discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is processed and executed, the steps of the deconvolution calculation method described in the first embodiment of the present application are implemented.

[0188] The present application has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present application. Those skilled in the art will appreciate that, without departing from the spirit and scope of the present application, various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present application, all of which fall within the scope of the present application. The scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for deconvolution calculation, characterized in that: include: Acquire multiple input blocks, each of which is a data block covered by a preset sliding window after each sliding in the original input feature map, wherein the original input feature map includes multiple layers of feature maps, and correspondingly, each of the input blocks includes multiple layers of data, and the size of the sliding window is set according to the size of the deconvolution kernel and the deconvolution step size; transforming the data in the deconvolution kernel according to a first pre-matrix to obtain a deconvolution kernel matrix, wherein the size of the first pre-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution; transforming the data in the plurality of input blocks according to a second pre-matrix to obtain a plurality of input matrices, wherein each of the input matrices includes a multi-layer data matrix, and a size of the second pre-matrix is pre-set according to a size of the deconvolution kernel and a deconvolution step size; Multiplying the deconvolution kernel matrix with the multiple input matrices respectively to obtain multiple intermediate matrices, wherein each of the intermediate matrices includes multiple layers of data matrices; Acquire multiple accumulation matrices, wherein any accumulation matrix is a single-layer data matrix obtained by accumulating data of all layers of data matrices in the corresponding intermediate matrix according to channels; According to a post-matrix, the data in the plurality of accumulation matrices are transformed respectively to obtain a plurality of output blocks, wherein the size of the post-matrix is pre-set according to the size of the deconvolution kernel and the step size of the deconvolution; Arrange the multiple output blocks in sequence into an output feature map to obtain a deconvolution calculation result.

2. The deconvolution calculation method according to claim 1, wherein: Before obtaining the plurality of input blocks, the method further includes: The size of the sliding window is obtained by the following formula: D = I1 × I1; Wherein, I1 represents the size of the input data matrix of the first-order efficient deconvolution, D represents the size of the sliding window, k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is the upper integer symbol in mathematical symbols, used to obtain a value not less than The minimum integer of the internal calculation result; The size of the first pre-matrix is obtained by the following formula: m1=k+s-2; H = m1 × k; Wherein, m1 represents the number of rows of the first-order convolution kernel pre-change matrix, and H represents the size of the first pre-matrix; The size of the second pre-matrix is obtained by the following formula: P = I1 × m1; Wherein, P represents the size of the second pre-matrix; The size of the post-matrix is obtained by the following formula: O1=1×s; Q = m1 × O1; Wherein, O1 represents the number of columns of the first-order post-transformation matrix, and Q represents the size of the post-transformation matrix.

3. The deconvolution calculation method according to claim 1, wherein: Before obtaining the plurality of input blocks, the method further includes: The size of the sliding window is obtained by the following formula: D = I2 × I2; Wherein, I2 represents the size of the input data matrix of the second-order efficient deconvolution, D represents the size of the sliding window, k represents the size of the deconvolution kernel, and s represents the step size of the deconvolution. It is a mathematical symbol used to obtain the smallest integer that is not less than the result of the calculation within the symbol. The size of the first pre-matrix is obtained by the following formula: m2=k+2×s-2; H = m2 × k; Wherein, m2 represents the number of rows of the second-order convolution kernel pre-change matrix, and H represents the size of the first pre-matrix; The size of the second pre-matrix is obtained by the following formula: P = I2 × m2; Wherein, P represents the size of the second pre-matrix; The size of the post-matrix is obtained by the following formula: O2=2×s; Q = m2 × O2; Wherein, O2 represents the number of columns of the second-order post-transformation matrix, and Q represents the size of the post-transformation matrix.

4. The deconvolution calculation method according to claim 2 or 3, characterized in that: The step of transforming the data in the deconvolution kernel according to the first pre-matrix to obtain the deconvolution kernel matrix includes: Multiplying the first pre-matrix, the deconvolution kernel, and the transposed matrix of the first pre-matrix in sequence to obtain the deconvolution kernel matrix; The step of transforming the data in the plurality of input blocks according to the second pre-matrix to obtain a plurality of input matrices includes: Multiplying the transposed matrix of the second pre-matrix, any of the input blocks, and the second pre-matrix in sequence to obtain a corresponding input matrix; The step of transforming the data in the plurality of accumulation matrices according to the post-matrix to obtain a plurality of output blocks comprises: The transposed matrix of the post-matrix, any of the accumulation matrices, and the post-matrix are multiplied in sequence to obtain a corresponding output block.

5. A hardware accelerator for deconvolution calculation, characterized in that: The hardware accelerator is used to perform the deconvolution calculation method according to any one of claims 1 to 3, and the hardware accelerator includes: An input module, comprising a plurality of parallel input channels, for inputting the data in the deconvolution kernel matrix into the multiplication module in parallel, and for inputting the data of each layer in the input block into the pre-processing module in sequence, wherein only one input block is input in a single deconvolution calculation process; A pre-processing module, comprising a deconvolution pre-processing unit, wherein the deconvolution pre-processing unit comprises a plurality of deconvolution pre-adders, configured to group and add data of any layer in the acquired input block to acquire an input matrix, and then sequentially input each layer of data matrix in the input matrix to the multiplication module; A multiplication module, comprising a plurality of weight multipliers, for respectively multiplying the deconvolution kernel matrix with each layer of the data matrix in the input matrix to obtain an intermediate matrix, and inputting each layer of the data matrix of the intermediate matrix into the accumulation module in sequence; An accumulation module, comprising an addition tree and a plurality of registers, for accumulating all layer data of the intermediate matrix by channel, obtaining an accumulation matrix, and then inputting the accumulation matrix into a post-processing module; A post-processing module, comprising a deconvolution post-processing unit, wherein the deconvolution post-processing unit comprises a plurality of deconvolution post-adders, configured to group and add the data in the accumulation matrix to obtain an output block; The output module includes a plurality of parallel output channels, and is used to output the data in the output block in parallel.

6. The hardware accelerator for deconvolution calculation according to claim 5, characterized in that: The input module includes 36 parallel input channels; The deconvolution pre-processing unit includes 13 deconvolution pre-adders; The multiplication module includes 36 weight multipliers; The deconvolution post-processing unit includes 60 deconvolution post-adders; The output module includes 16 parallel output channels.

7. The hardware accelerator for deconvolution calculation according to claim 5, characterized in that: The pre-processing module further includes: a convolution pre-processing unit and a pre-selector, wherein the convolution pre-processing unit includes a plurality of convolution pre-adders, and the pre-selector is used to select the output result of the input module to be input to the deconvolution pre-processing unit or the convolution pre-processing unit according to the current calculation requirements; The post-processing module also includes: a convolution post-processing unit and a post-selector, the convolution post-processing unit includes multiple convolution post-adders, and the post-selector is used to select the output result of the accumulation module to be input into the deconvolution post-processing unit or the convolution post-processing unit according to the current calculation requirements.

8. The hardware accelerator for deconvolution calculation according to claim 7, characterized in that: The input module includes 36 parallel input channels; The convolution pre-processing unit includes 180 convolution pre-adders; The deconvolution pre-processing unit includes 13 deconvolution pre-adders; The multiplication module includes 36 weight multipliers; The convolution post-processing unit includes 100 convolution post-adders; The deconvolution post-processing unit includes 60 deconvolution post-adders; The output module includes 16 parallel output channels.

9. A computer device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the deconvolution calculation method according to any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is processed and executed, the steps of the deconvolution calculation method according to any one of claims 1 to 4 are implemented.

Images