Deconvolution calculation method, calculation apparatus and calculation system, and storage medium

Abstract

A deconvolution calculation method, calculation apparatus (100) and calculation system (300), and a storage medium (400). The deconvolution calculation method comprises: by means of a systolic array (10), calculating each weight of each row of a convolution kernel with a first feature matrix, so as to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel; and by means of an adder tree (20), calculating a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel, so as to obtain a third feature matrix.

Description

Deconvolution computing method, computing device, computing system and storage medium

[0001] Priority information

[0002] This application claims priority to and the benefit of the filing date of Chinese Patent Application No. 2024113287395 filed on September 23, 2024, in the China National Intellectual Property Office, and is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, in particular to a deconvolution computing method, a computing device, a computing system and a computer readable storage medium. BACKGROUND

[0004] Convolutional neural networks are widely used in image recognition, object detection, semantic segmentation and other scenarios due to their high accuracy. The non-interpretable nature of the hidden layers of convolutional neural networks has become the primary safety concern for the promotion of deep learning. Deconvolution is an important means to solve this problem, and the current deconvolution computing method has low hardware computing efficiency. SUMMARY

[0005] The present application provides a deconvolution computing method, a computing device, a computing system and a computer readable storage medium to solve at least one of the above technical problems.

[0006] The deconvolution computing method of the present application is applied to a deconvolution computing device, which includes a systolic array and an adder tree, the systolic array is connected with the adder tree, and the deconvolution computing method includes:

[0007] inputting a convolution kernel and a first feature matrix into the systolic array;

[0008] calculating each weight of each row of the convolution kernel and the first feature matrix through the systolic array to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel;

[0009] calculating a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel through the adder tree to obtain a third feature matrix, the third feature matrix being a deconvolution result of the first feature matrix.

[0010] The deconvolution computing device of the present application includes a systolic array and an adder tree, the systolic array is connected with the adder tree;

[0011] the systolic array is used to receive an input convolution kernel and a first feature matrix;

[0012] The pulsating array is also used to calculate a second feature matrix corresponding to each weight of each row of the convolution kernel and the first feature matrix.

[0013] The adder tree is used to calculate multiple second feature matrices corresponding to multiple weights in multiple rows of the convolution kernel to obtain a third feature matrix, which is the result of the deconvolution of the first feature matrix.

[0014] The deconvolution calculation system of this application includes one or more processors and a memory. The memory stores a computer program, which, when executed by the processor, implements the above-described deconvolution calculation method.

[0015] The computer-readable storage medium of this application embodiment stores a computer program thereon, which, when executed by a processor, implements the above-described deconvolution calculation method.

[0016] In the deconvolution calculation method, computing device, computing system, and computer-readable storage medium of this application, during deconvolution calculation, the convolution kernel and the first feature matrix are directly input into the deconvolution calculation device. The third feature matrix is ​​obtained through the systolic array and adder tree of the deconvolution calculation device, and is the deconvolution calculation result of the first feature matrix, without any additional operations. Thus, the computing resources of the deconvolution calculation device are used for the calculation of effective data, and storage resources are not wasted on storing zero values ​​that do not contribute to the calculation, effectively improving the hardware computing efficiency of the deconvolution calculation device.

[0017] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0019] Figure 1 is a schematic diagram of positive convolution calculation in related technologies;

[0020] Figure 2 is a schematic diagram of deconvolution calculation in related technologies;

[0021] Figure 3 is a flowchart illustrating the deconvolution calculation method according to some embodiments of this application;

[0022] Figure 4 is a schematic diagram of the deconvolution computing device of some embodiments of this application operating at the first moment;

[0023] Figure 5 is a schematic diagram of a forward convolution calculation according to certain embodiments of the present application;

[0024] Figure 6 is a schematic diagram of a matrix multiplication operation corresponding to the forward convolution calculation of Figure 5;

[0025] Figure 7 is a schematic diagram of a matrix multiplication operation corresponding to an inverse convolution calculation of the forward convolution calculation of Figure 5;

[0026] Figure 8 is a schematic diagram of a breakdown of an inverse convolution calculation according to certain embodiments of the present application;

[0027] Figure 9 is a schematic diagram of the structure of a systolic array according to certain embodiments of the present application;

[0028] Figure 10 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0029] Figure 11 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0030] Figure 12 is a schematic diagram of an inverse convolution calculation device according to certain embodiments of the present application operating at a second time;

[0031] Figure 13 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0032] Figure 14 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0033] Figure 15 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0034] Figure 16 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0035] Figure 17 is a schematic diagram of a method of inverse convolution calculation according to certain embodiments of the present application;

[0036] Figure 18 is a schematic diagram of a second extended feature matrix according to certain embodiments of the present application;

[0037] Figure 19 is a schematic diagram of the modules of an inverse convolution calculation system according to certain embodiments of the present application;

[0038] Figure 20 is a schematic diagram of the connection state of a computer readable storage medium and a processor according to certain embodiments of the present application.

[0039] Legend: deconvolution computing device 100, systolic array 10, processing unit 11, multiplier 111, second register 112, third register 113, fourth register 114, adder 115, adder tree 20, first register 30, deconvolution computing system 300, processor 310, memory 320, computer-readable storage medium 400, computer program 410, processor 420. DETAILED DESCRIPTION

[0040] Embodiments of the present application are further described below with reference to the accompanying drawings. Like or similar elements in the drawings are denoted by the same or similar reference signs, and the description thereof will not be repeated. In addition, the embodiments of the present application described below with reference to the accompanying drawings are exemplary and are for the purpose of explaining the embodiments of the present application, and should not be understood as limiting the present application.

[0041] Convolutional neural networks are widely used in image recognition, object detection, semantic segmentation, and other scenarios due to their high accuracy. Deep learning, led by convolutional neural networks, has always been plagued by the black box problem. The unexplainability of the hidden layers of convolutional neural networks has become the primary security problem for promoting deep learning. Deconvolution is an important means to solve this problem. It is an algorithm that approximates the inverse operation of convolution and can restore the information of the hidden layers of convolutional neural networks to the greatest extent. Then, deconvolution is widely used in super-resolution deep neural networks as a high-precision upsampling method.

[0042] In the article “A guide to convolution arithmetic for deep learning”, a method of converting deconvolution operation into forward convolution calculation is proposed. In this article, the author points out that deconvolution can be regarded as a special forward convolution. When performing deconvolution calculation, the image size can be expanded by inserting zero values in the original image, and the convolution kernel can be rotated by 180 degrees. Then, the zero-inserted image and the rotated convolution kernel are convolved like forward convolution.

[0043] As shown in FIG. 1, the deconvolution calculation is performed on a 5x5 image with a 3x3 kernel size and a stride of 1, which is equivalent to the forward convolution calculation with a 3x3 kernel size, a stride of 1, and a padding of 1 on a 5x5 image. As shown in FIG. 2, the deconvolution calculation is performed on a 3x3 image with a 3x3 kernel size and a stride of 2. When the stride of the deconvolution calculation is not equal to 1, padding is needed between the pixels in the image. The stride of the deconvolution calculation is equal to 2, which means that one zero value is padded between each pixel in the image. Similarly, if the stride of the deconvolution calculation is equal to 3, two zero values are padded between each pixel in the image, and so on. The input feature map of the corresponding forward convolution calculation of the deconvolution calculation is the image after zero padding.

[0044] The computing device designed based on this scheme will result in a large loss of hardware computing efficiency. Because the convolution calculation is essentially a matrix multiplication operation, the above-mentioned deconvolution calculation scheme needs to introduce a large number of zero values during the deconvolution calculation process, especially when the stride is not equal to 1, in order to ensure the consistency of the deconvolution calculation and the forward convolution calculation. For example, in the deconvolution calculation with a stride of 2, the size of the original image needs to be expanded by 4 times, that is, the zero values equivalent to 3 times the size of the original image are inserted based on the original image. These zero values will not contribute to the final calculation result, and in order to process these zero values, the extremely limited computing resources and bandwidth on the chip will be occupied, resulting in a sharp decline in the computing efficiency of the hardware.

[0045] Therefore, in order to solve the above-mentioned problems, referring to FIG. 3 and FIG. 4, the present application provides a deconvolution calculation method applied to a deconvolution calculation device 100, the deconvolution calculation device 100 comprising a systolic array 10 and an adder tree 20, the systolic array 10 being connected with the adder tree 20, and the deconvolution calculation method comprising:

[0046] 010: inputting a convolution kernel and a first feature matrix into the systolic array 10;

[0047] 020: calculating each weight of each row of the convolution kernel and the first feature matrix by the systolic array 10 to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel;

[0048] 030: calculating a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel by the adder tree 20 to obtain a third feature matrix, the third feature matrix being a deconvolution calculation result of the first feature matrix.

[0049] In the deconvolution calculation method of the embodiments of the present application, when deconvolution calculation is performed, the convolution kernel and the first feature matrix are directly input into the deconvolution calculation device 100, and calculation is performed through the systolic array 10 and the adder tree 20 of the deconvolution calculation device 100 to obtain the third feature matrix, which is the deconvolution calculation result of the first feature matrix, without any additional operation. In this way, the calculation resources of the deconvolution calculation device 100 are all used for the calculation of effective data, and the storage resources are not wasted on storing zero values that do not contribute to the calculation, effectively improving the hardware calculation efficiency of the deconvolution calculation device 100.

[0050] Specifically, the process of forward convolution and the matrix multiplication operation corresponding to the forward convolution are analyzed first. As shown in FIGS. 5 and 6, the process of forward convolution and the matrix multiplication operation corresponding to the forward convolution are analyzed. When forward convolution is performed, the convolution kernel window continuously slides on the input feature map according to the convolution step size. After sliding to each position, the weight values in the convolution kernel are multiplied by the feature values at the corresponding positions in the convolution kernel window and then added to obtain a feature value in the output feature map. This process is repeated until each feature value in the input feature map is accessed. For example, as shown in FIG. 5, when the convolution kernel window is at position 1, the weights in the convolution kernel are multiplied by the feature values in the convolution kernel window and then added to obtain m1, which is the feature value at the corresponding position in the output feature map. When the convolution kernel window is at position 2, the weights in the convolution kernel are multiplied by the feature values in the convolution kernel window and then added to obtain m2, which is the feature value at the corresponding position in the output feature map.

[0051] The matrix multiplication operation corresponding to the deconvolution of the above forward convolution is analyzed again. As shown in FIG. 7, the matrix multiplication operation corresponding to the deconvolution calculation is analyzed. Matrix D is the expanded convolution kernel, matrix E is the input feature map of the deconvolution calculation, and matrix E corresponds to matrix C, that is, it is obtained by flattening the output feature map in FIG. 5. The number of rows of matrix D is the number of feature values contained in the input feature map in FIG. 5, and the number of columns of matrix D is the number of feature values contained in the output feature map in FIG. 5. Each column in matrix D corresponds to the position description when the convolution kernel window slides on each position of the input feature map in FIG. 5. The parts not covered by the convolution kernel window are filled with “0”. The calculated matrix F is the output feature map of the deconvolution calculation, which corresponds to the input feature map in FIG. 5.

[0052] It is found through research that the operation process of deconvolution can be disassembled. As shown in FIG. 8, the deconvolution with a convolution kernel size of 2x2 and a step size of 1 is disassembled. The deconvolution calculation process after disassembly is divided into two steps: the first step is the calculation of weights and feature values. The weight “a” of the convolution kernel is first multiplied by each feature value in the input feature map of the deconvolution to obtain a partial sum matrix with the same size as the input feature map of the deconvolution.

[0053] Then, the weights "b", "c" and "d" are also multiplied with the feature values in the deconvolution input feature map respectively, and three partial sum matrices are obtained. The second step is to accumulate the four partial sum matrices, but the accumulation of the four partial sum matrices is not one-to-one corresponding, and each partial sum matrix needs to be zero-extended first to obtain four 4x4 matrices. The final deconvolution output feature map is obtained by accumulating the four extended matrices. When the deconvolution output feature map in FIG. 8 is the output feature map in FIG. 5, the deconvolution output feature map in FIG. 8 is the input feature map in FIG. 5, and corresponds to the matrix F in FIG. 7.

[0054] The above deconvolution operation process can be mapped to the deconvolution calculation device 100, which includes the systolic array 10 and the adder tree 20, and the systolic array 10 is connected with the adder tree 20. The convolution kernel and the first feature matrix are input into the systolic array 10, and the first feature matrix can be a matrix corresponding to the feature map to be deconvolved. The systolic array 10 calculates each weight of each row of the convolution kernel with the first feature matrix respectively, and each weight corresponds to a second feature matrix, and the number of the second feature matrices is equal to the number of the weights in the convolution kernel, which corresponds to the first step of the above decomposed deconvolution calculation process.

[0055] The plurality of second feature matrices are output to the adder tree 20, and the adder tree 20 calculates the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain a third feature matrix, which is the deconvolution calculation result of the first feature matrix, corresponding to the second step of the above decomposed deconvolution calculation process.

[0056] In the embodiments of the present application, the adder tree 20 is added on the basis of the systolic array 10 to form the deconvolution calculation device 100, and no additional operation is required for the original feature map during deconvolution calculation. The calculation resources of the deconvolution calculation device 100 are all used for calculating effective data, and the storage resources are not wasted on storing zero values which do not contribute to the calculation, thereby effectively improving the hardware calculation efficiency of the deconvolution calculation device 100.

[0057] Please refer to FIG. 4 and FIG. 9, in some embodiments, the systolic array 10 includes a plurality of processing units 11 arranged in rows and columns, each processing unit 11 includes a multiplier 111, and each weight of each row of the convolution kernel is calculated with the first feature matrix through the systolic array 10 to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel (i.e. 020), including:

[0058] The multiple multipliers 111 corresponding to the multiple processing units 11 of each column respectively multiply each weight of each row of the convolution kernel with the multiple feature values of the first feature matrix to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel.

[0059] Specifically, the systolic array 10 includes multiple processing units 11 arranged in rows and columns, and the number of the processing units 11 is not limited. As shown in FIG. 5, the systolic array 10 includes 4x4 processing units 11 arranged in rows and columns. The multiple weights of the convolution kernel and the multiple feature values of the first feature matrix can be input into the multiple processing units 11 in the manner shown in FIG. 5, and the multiple processing units 11 respectively perform multiplication operations on the multiple weights and the multiple feature values.

[0060] Referring to FIG. 9, in some embodiments, each processing unit 11 includes a multiplier 111 and a second register 112. The multiplier 111 is configured to perform a multiplication operation on a weight and a feature value of the first feature matrix, and the second register 112 is configured to store the operation result.

[0061] Specifically, each processing unit 11 includes a multiplier 111 and a second register 112. The multiple weights of the convolution kernel and the multiple feature values of the first feature matrix are respectively input into the multipliers 111 of the multiple processing units 11, the multipliers 111 are configured to perform multiplication operations on the weights and the feature values of the first feature matrix, and the second registers 112 are configured to store the operation results.

[0062] Referring to FIG. 9, in some embodiments, each processing unit 11 further includes a third register 113 and a fourth register 114. The third register 113 is configured to perform data transmission between multiple processing units 11 in the horizontal direction, and the fourth register 114 is configured to perform data transmission between multiple processing units 11 in the vertical direction.

[0063] Specifically, each processing unit 11 further includes a third register 113 and a fourth register 114. The third register 113 is configured to perform data transmission between multiple processing units 11 in the horizontal direction, i.e., between multiple processing units 11 in the same row. The fourth register 114 is configured to perform data transmission between multiple processing units 11 in the vertical direction, i.e., between multiple processing units 11 in the same column.

[0064] Referring to FIG. 4, FIG. 9 and FIG. 10, in some embodiments, inputting the convolution kernel and the first feature matrix into the systolic array 10 (i.e., 010) includes:

[0065] 011: inputting multiple weights of each row of the convolution kernel into the systolic array 10 column by column;

[0066] 012: inputting the multiple rows of feature values of the first feature matrix into the systolic array 10 row by row.

[0067] Specifically, each row of the convolution kernel includes multiple weights, and the multiple weights of each row of the convolution kernel are input from different columns of the systolic array 10 respectively, and the multiple weights are first input into the multiple processing units 11 in different columns of the first row of the systolic array 10 respectively, and the multiple weights can be correspondingly transmitted to the multiple processing units 11 in different columns of the second row of the systolic array 10 through the fourth registers 114 of the multiple processing units 11 in different columns of the first row, and so on.

[0068] The first feature matrix includes MxN feature values, and the multiple rows of feature values of the first feature matrix are input from different rows of the systolic array 10 respectively, and the multiple rows of feature values are first input into the multiple processing units 11 in different rows of the first column of the systolic array 10 respectively, and the multiple rows of feature values can be correspondingly transmitted to the multiple processing units 11 in different rows of the second column of the systolic array 10 through the third registers 113 of the multiple processing units 11 in different rows of the first column, and so on. In this way, the processing units 11 in the systolic array 10 can receive feature values from the horizontal direction and weights from the vertical direction to calculate the feature values and the weights.

[0069] Referring to FIGS. 4 and 11, in some embodiments, each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array 10 to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel (i.e., 020), including:

[0070] 021: each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array 10 at different time points respectively to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel.

[0071] Specifically, the convolution kernel includes multiple rows of weights, and multiple weights in the same row are input into the systolic array 10 for calculation at the same time, as shown in FIGS. 4 and 12, and weights in different rows are input into the systolic array 10 for calculation at different time points, and the calculation between the weights in different rows of the convolution kernel and the first feature matrix is performed at different time points.

[0072] Referring to FIGS. 4 and 12, in some embodiments, each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array 10 to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel (i.e., 030), including:

[0073] At the first time point, each weight of the first row of the convolution kernel is calculated with the first feature matrix by the systolic array 10 to obtain a second feature matrix corresponding to each weight of the first row of the convolution kernel;

[0074] At the second time, each weight of the second row of the convolution kernel is calculated with the first feature matrix by the systolic array 10 to obtain a second feature matrix corresponding to each weight of the second row of the convolution kernel.

[0075] Specifically, at the first time, a plurality of weights of the first row of the convolution kernel are input into the systolic array 10 column by column. As shown in FIG. 4, taking a convolution kernel with a size of 2 as an example, that is, the convolution kernel includes 2x2 weights. Wa and Wb are two weights of the first row of the convolution kernel, and X is the first feature matrix. In the systolic array 10, a plurality of multipliers 111 corresponding to a plurality of processing units 11 in a column are used to calculate a weight of the first row of the convolution kernel with a plurality of feature values of a plurality of rows of the first feature matrix respectively.

[0076] For example, the first weight Wa of the first row of the convolution kernel is input into the first processing unit 11 in the first column of the plurality of processing units 11, and the first weight Wa is multiplied with the first row of feature values of the first feature matrix; the first weight Wa is transmitted to the second processing unit 11 in the first column through the fourth register 114 of the first processing unit 11 in the first column, so as to be multiplied with the second row of feature values of the first feature matrix. After the first weight Wa is multiplied with the plurality of rows of feature values of the first feature matrix, a corresponding second sub-feature matrix is obtained.

[0077] The second weight Wb of the first row of the convolution kernel is input into the first processing unit 11 in the second column of the plurality of processing units 11. The first row of feature values of the first feature matrix is transmitted to the first processing unit 11 in the second column through the third register 113 of the first processing unit 11 in the first column, so as to be multiplied with the second weight Wb. After the second weight Wb is multiplied with the plurality of rows of feature values of the first feature matrix, a corresponding second sub-feature matrix is obtained. For the plurality of weights of the first row, a plurality of second sub-feature matrices can be obtained.

[0078] At the second time, a plurality of weights of the second row of the convolution kernel are input into the systolic array 10 column by column. As shown in FIG. 12, Wc and Wd are two weights of the second row of the convolution kernel, and X is the first feature matrix. In the systolic array 10, a plurality of multipliers 111 corresponding to a plurality of processing units 11 in a column are used to calculate a weight of the second row of the convolution kernel with a plurality of feature values of a plurality of rows of the first feature matrix respectively.

[0079] For example, the first weight Wc of the second row of the convolution kernel is input into the first processing unit 11 of the plurality of processing units 11 of the first column, and is multiplied by the first feature value of the first row of the first feature matrix; the first weight Wc is transmitted to the second processing unit 11 of the first column through the fourth register 114 of the first processing unit 11 of the first column, so as to be multiplied by the second feature value of the second row of the first feature matrix. After the first weight Wc is multiplied by the plurality of feature values of the first feature matrix, a corresponding second sub-feature matrix is obtained.

[0080] The second weight Wd of the second row of the convolution kernel is input into the first processing unit 11 of the plurality of processing units 11 of the second column. The first feature value of the first row of the first feature matrix is transmitted to the first processing unit 11 of the second column through the third register 113 of the first processing unit 11 of the first column, so as to be multiplied by the second weight Wd. After the second weight Wd is multiplied by the plurality of feature values of the first feature matrix, a corresponding second sub-feature matrix is obtained, and a plurality of second sub-feature matrices can be obtained for the plurality of weights of the second row.

[0081] In some embodiments, the adder tree 20 includes a plurality of nodes, and the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are calculated through the adder tree 20 to obtain a third feature matrix (i.e., 030), including:

[0082] The plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are calculated through the plurality of nodes respectively to obtain the third feature matrix.

[0083] Specifically, the adder tree 20 can include a plurality of nodes, each node being configured to perform corresponding bit calculation on a row of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel, and the plurality of nodes can calculate to obtain the third feature matrix. The corresponding bit calculation includes corresponding bit splicing of the corresponding bit accumulation kernel. For example, as shown in FIG. 4 and FIG. 12, four second feature matrices corresponding to four weights of two rows of the convolution kernel, the adder tree 20 can include four nodes, the first node is configured to perform corresponding bit calculation on the first row of the four second feature matrices, the second node is configured to perform corresponding bit calculation on the second row of the four second feature matrices, the third node is configured to perform corresponding bit calculation on the third row of the four second feature matrices, and the fourth node is configured to perform corresponding bit calculation on the fourth row of the four second feature matrices.

[0084] Please refer to FIG. 4, FIG. 12 and FIG. 13, in some embodiments, the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are calculated through the adder tree 20 to obtain a third feature matrix (i.e., 030), including:

[0085] 031: The adder tree 20 is used to calculate the second feature matrix corresponding to each weight of each row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of each row of the convolution kernel is obtained;

[0086] 032: The adder tree 20 is used to calculate the third sub-feature matrix corresponding to each weight of each row of the convolution kernel, and a third feature matrix is obtained.

[0087] Specifically, the adder tree 20 can calculate the second feature matrix corresponding to each weight of each row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of each row of the convolution kernel is obtained. It can be understood that a third sub-feature matrix corresponding to each row of the convolution kernel is obtained. After obtaining the third sub-feature matrix corresponding to each row of the convolution kernel, the adder tree 20 is used to calculate the third sub-feature matrix, and a third feature matrix is obtained.

[0088] Referring to FIG. 4 and FIG. 14, in some embodiments, the deconvolution calculation device 100 further comprises a register 30 connected with the adder tree 20. The adder tree 20 is used to calculate the second feature matrix corresponding to each weight of each row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of each row of the convolution kernel is obtained (i.e. 031), which comprises:

[0089] 0311: The adder tree 20 is used to calculate the second feature matrix corresponding to each weight of the first row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of the first row of the convolution kernel is obtained;

[0090] At this time, the deconvolution calculation method further comprises:

[0091] 040: The register 30 is used to store the third sub-feature matrix corresponding to each weight of the first row of the convolution kernel.

[0092] Specifically, as shown in FIG. 4, the adder tree 20 can calculate the second feature matrix corresponding to each weight of the first row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of the first row of the convolution kernel is obtained. The register 30 is connected with the adder tree 20, so that after the third sub-feature matrix corresponding to each weight of the first row of the convolution kernel is calculated, the adder tree 20 can input the third sub-feature matrix into the register 30 for storage.

[0093] Referring to FIG. 4 and FIG. 14, in some embodiments, the adder tree 20 is used to calculate the second feature matrix corresponding to each weight of each row of the convolution kernel, and a third sub-feature matrix corresponding to each weight of each row of the convolution kernel is obtained (i.e. 031), which further comprises:

[0094] 0312: The adder tree 20 calculates the plurality of second feature matrices corresponding to the plurality of weights of the second row of the convolution kernel to obtain a third sub-feature matrix corresponding to the two weights of the second row of the convolution kernel.

[0095] Specifically, for example, as shown in FIG. 12, the adder tree 20 can calculate two second feature matrices corresponding to two weights of the second row of the convolution kernel to obtain a third sub-feature matrix corresponding to the two weights of the second row of the convolution kernel.

[0096] Referring to FIGS. 4 and 14, in some embodiments, the deconvolution calculation method further comprises:

[0097] 050: The register 30 transmits the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel to the adder tree 20.

[0098] At this time, the adder tree 20 calculates the plurality of third sub-feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain the third feature matrix, including:

[0099] 060: The adder tree 20 calculates the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel to obtain the third feature matrix.

[0100] Specifically, after calculating the third sub-feature matrix corresponding to the two weights of the second row of the convolution kernel, the register 30 can transmit the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel to the adder tree 20. The adder tree 20 calculates the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel to obtain the third feature matrix.

[0101] If the size of the convolution kernel is greater than 2, the adder tree 20 can calculate the plurality of second feature matrices corresponding to the third row, the fourth row,..., and the n-th row of the convolution kernel, respectively, to obtain a third sub-feature matrix for each row. The adder tree 20 calculates the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel to obtain a first cumulative feature matrix, and inputs the first cumulative feature matrix to the register 20 for storage.

[0102] When the adder tree 20 calculates a third sub-feature matrix corresponding to the plurality of weights of the third row of the convolution kernel, the first accumulated feature matrix is transmitted to the adder tree 20 through the register 30, and the adder tree 20 calculates the third sub-feature matrix corresponding to the plurality of weights of the third row of the convolution kernel and the first accumulated feature matrix to obtain a second accumulated feature matrix. In this way, the calculation of the third sub-feature matrix corresponding to the plurality of weights of the last row of the convolution kernel is completed. If the size of the convolution kernel is 3, the second accumulated feature matrix is the third feature matrix.

[0103] Referring to FIGS. 4 and 15, in some embodiments, the plurality of second feature matrices corresponding to the plurality of rows of the convolution kernel are calculated by the adder tree 20 to obtain the third feature matrix (i.e., 030), including:

[0104] 033: According to the flag bit of the plurality of second feature matrices corresponding to the plurality of rows of the convolution kernel, it is determined whether the plurality of second feature matrices corresponding to the plurality of rows of the convolution kernel need to be accumulated in the corresponding bit.

[0105] 034: When the plurality of second feature matrices corresponding to the plurality of rows of the convolution kernel need to be accumulated in the corresponding bit, the plurality of second feature matrices corresponding to the plurality of rows of the convolution kernel are accumulated in the corresponding bit by the adder tree 20 to obtain the third feature matrix.

[0106] Specifically, each second feature matrix further includes a flag bit, which can be a single input data of the systolic array 10 or input with the convolution kernel and / or the first feature matrix. The flag bit can be "0" or "1". According to the flag bit, it can be determined whether the plurality of second feature matrices corresponding to the first row need to be accumulated. It should be noted that the flag bits of the plurality of second feature matrices are consistent.

[0107] The flag bit is related to the size of the convolution kernel and the deconvolution step. It is found through research that, in the deconvolution calculation, when the size of the convolution kernel is equal to the deconvolution step, the plurality of second feature matrices corresponding to the first row do not need to be accumulated. When the size of the convolution kernel is not equal to the deconvolution step, the plurality of second feature matrices corresponding to the first row need to be accumulated. The corresponding deconvolution step of FIG. 9 is 1, and therefore, the plurality of partial sum matrices are accumulated.

[0108] In an example, when the size of the convolution kernel is equal to the deconvolution step, the flag bit is set to 0 to indicate that the plurality of second feature matrices corresponding to the first row do not need to be accumulated. When the size of the convolution kernel is not equal to the deconvolution step, the flag bit is set to 1 to indicate that the plurality of second feature matrices corresponding to the first row need to be accumulated. Of course, the opposite setting is also possible, which is not limited herein.

[0109] When the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel need to be added, the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are added by the adder tree 20 to obtain the third feature matrix. For example, as shown in FIG. 4 and FIG. 12, when the four second feature matrices corresponding to the first row and the second row of the convolution kernel need to be added, the four second feature matrices are added by the adder tree 20. After the addition is completed, one third feature matrix corresponding to the first row of the convolution kernel is obtained.

[0110] Referring to FIG. 4, FIG. 12 and FIG. 16, in some embodiments, the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are added by the adder tree 20 to obtain the third feature matrix (i.e., 034), including:

[0111] 0341: The second feature matrix corresponding to each weight of each row of the convolution kernel is zero-extended according to a first preset rule to obtain a first extended feature matrix corresponding to each weight of each row of the convolution kernel.

[0112] 0342: The plurality of first extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are added by the adder tree 20 to obtain the third feature matrix.

[0113] The specific process of zero-extending the second feature matrix corresponding to each weight of each row of the convolution kernel according to the first preset rule is as follows:

[0114] Referring to FIG. 4 and FIG. 12, in some embodiments, the second feature matrix corresponding to each weight of each row of the convolution kernel is zero-extended according to the first preset rule to obtain the first extended feature matrix corresponding to each weight of each row of the convolution kernel (i.e., 0341), including:

[0115] According to the position of each weight of each row of the convolution kernel in the convolution kernel, the second feature matrix corresponding to each weight of each row of the convolution kernel is zero-extended as a whole to obtain the first extended feature matrix corresponding to each weight of each row of the convolution kernel.

[0116] Specifically, the second feature matrix corresponding to each weight is zero-extended as a whole according to the position of each weight of each row of the convolution kernel in the convolution kernel. For example, in FIG. 4, the position of the weight Wa in the 2x2 convolution kernel is (1, 1), and the corresponding second feature matrix R1 is expanded to the first extended feature matrix S1:

[0117] Similarly, the plurality of second feature matrices are zero-value extended to obtain a plurality of first extended feature matrices.

[0118] The above is the case where the deconvolution step is 1. When the deconvolution step is not 1, the plurality of second feature matrices also need to be zero-value extended according to the deconvolution step. For example, when the deconvolution step is 2, 1 zero value is inserted between every two elements of the second feature matrix; when the deconvolution step is 3, 2 zero values are inserted between every two elements of the second feature matrix to obtain a fifth feature matrix. Similarly, if the size of the second feature matrix is AxB, at this time, the size of the first extended feature matrix can be represented as [A+(S-1)x(A-1)+N-1]x[B+(S-1)x(B-1)+N-1], where S is the deconvolution step, and N is the convolution kernel size.

[0119] After obtaining the first extended matrix, the plurality of first extended feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel are added by the adder tree 20 to obtain the third feature matrix. The corresponding bit addition is to add the feature values in the same position of the plurality of first extended feature matrices.

[0120] Referring to FIGS. 4 and 17, in some embodiments, the plurality of second feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel are calculated by the adder tree 20 to obtain the third feature matrix (i.e., 030), which further includes:

[0121] 035: When the plurality of second feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel do not need to be added by the corresponding bit, the plurality of second feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel are spliced by the adder tree 20 to obtain the third feature matrix.

[0122] Referring to FIG. 4, in some embodiments, the plurality of second feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel are spliced by the adder tree 20 to obtain the third feature matrix (i.e., 035), which includes:

[0123] According to the second preset rule, one second feature matrix corresponding to each weight of each row of the convolution kernel is zero-value extended to obtain one second extended feature matrix corresponding to each weight of each row of the convolution kernel.

[0124] The plurality of second extended feature matrices corresponding to the plurality of weights of the multiple rows of the convolution kernel are spliced by the adder tree 20 to obtain the third feature matrix.

[0125] Specifically, when the corresponding bit accumulation of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel is not required, according to a second preset rule, a zero value is expanded for one second feature matrix corresponding to each weight of each row of the convolution kernel to obtain one second extended feature matrix corresponding to each weight of each row of the convolution kernel, as shown in FIG. 18.

[0126] It can be understood that FIG. 18 is obtained by expanding the four second feature matrices in FIGS. 4 and 12 by zero values. In the four second extended feature matrices in FIG. 18, there is only one element at the corresponding position, and the elements in the four second extended feature matrices do not interfere with each other. Therefore, the plurality of second extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are spliced by the adder tree 20 to obtain one third feature matrix. The corresponding bit splicing is that the feature values of the plurality of second extended feature matrices are spliced into a new matrix according to the corresponding positions, and the position of each feature value in the second extended matrix is the same as the position in the new matrix.

[0127] Referring to FIG. 4, in some embodiments, the zero value is expanded for one second feature matrix corresponding to each weight of each row of the convolution kernel according to the second preset rule to obtain one second extended feature matrix corresponding to each weight of each row of the convolution kernel, including:

[0128] According to the position of each weight of each row of the convolution kernel in the convolution kernel, the zero value of the complementary position is expanded for each feature value in one second feature matrix corresponding to each weight of each row of the convolution kernel to obtain one second extended feature matrix corresponding to each weight of each row of the convolution kernel.

[0129] Specifically, according to the position of each weight of each row of the convolution kernel in the convolution kernel, the zero value of the complementary position is expanded for each feature value in one second feature matrix corresponding to each weight of each row of the convolution kernel to obtain one second extended feature matrix corresponding to each weight of each row of the convolution kernel. The number of expanded zero values is determined according to the deconvolution step. For example, in FIG. 4, the position of the weight Wa in the 2x2 convolution kernel is (1, 1), and for each feature value in the second feature matrix corresponding to the weight Wa, the zero value of the complementary position is expanded, and when the deconvolution step is 2, one zero value is expanded at the complementary position, and a 2x2 submatrix is expanded, for example, the feature value e can be expanded as follows:

[0130] The multiple feature values in the second feature matrix are respectively zero-value extended to obtain multiple sub-matrices, and the multiple sub-matrices are arranged according to the positions of the corresponding feature values, so that the second extended feature matrix T1 shown in FIG. 18 is obtained. In this way, the multiple second feature matrices are zero-value extended to obtain the corresponding multiple second extended feature matrices. The fifth feature matrices obtained after the zero-value extension of the multiple second feature matrices S1, S2, S3 and S4 in FIG. 3 and FIG. 16 are shown in FIG. 20, and correspond to the fifth feature matrices T1, T2, T3 and T4 respectively.

[0131] It can be understood that when the deconvolution step is 3, 2 zero values are respectively extended in the complementary positions, and the same is true for the deconvolution step S and N-1 zero values respectively extended in the complementary positions. If the size of the second feature matrix is AxB, the size of the second extended feature matrix can be represented as [A+(S-1)x(A-1)+N-1]x[B+(S-1)x(B-1)+N-1], where S is the deconvolution step and N is the convolution kernel size.

[0132] In some embodiments, the number of convolution kernels is multiple.

[0133] It can be understood that the systolic array 10 can be set to a larger size, the adder tree 20 can also perform multiple addition operations at the same time, and the first register 30 can also store multiple feature matrices at the same time. Therefore, the deconvolution calculation device 100 in the embodiments of the present application can perform the calculation of multiple convolution kernels at the same time. In this way, the hardware utilization rate of the deconvolution calculation device 100 is effectively improved.

[0134] Referring to FIG. 4, the embodiments of the present application further provide a deconvolution calculation device 100. The deconvolution calculation device 100 comprises a systolic array 10 and an adder tree 20, and the systolic array 10 is connected with the adder tree 20.

[0135] The systolic array 10 is configured to receive an input convolution kernel and a first feature matrix.

[0136] The systolic array 10 is further configured to calculate each weight of each row of the convolution kernel and the first feature matrix to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel.

[0137] The adder tree 20 is configured to calculate multiple second feature matrices corresponding to multiple weights of multiple rows of the convolution kernel to obtain a third feature matrix, and the third feature matrix is a deconvolution calculation result of the first feature matrix.

[0138] In some embodiments, the multiple weights of each row of the convolution kernel are input into the systolic array 10 by column.

[0139] The multiple feature values of the multiple rows of the first feature matrix are input into the systolic array 10 by row.

[0140] In some embodiments, the systolic array 10 is configured to calculate each weight of a different row of the convolution kernel with the first feature matrix at different time instants to obtain a second feature matrix corresponding to each weight of the different row of the convolution kernel.

[0141] In some embodiments, the adder tree 20 is configured to calculate a plurality of second feature matrices corresponding to a plurality of weights of each row of the convolution kernel to obtain a third sub-feature matrix corresponding to the plurality of weights of each row of the convolution kernel.

[0142] The adder tree 20 is configured to calculate a plurality of third sub-feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel to obtain the third feature matrix.

[0143] In some embodiments, at a first time instant, the systolic array 10 is configured to calculate each weight of a first row of the convolution kernel with the first feature matrix to obtain a second feature matrix corresponding to each weight of the first row of the convolution kernel.

[0144] At a second time instant, the systolic array 10 is configured to calculate each weight of a second row of the convolution kernel with the first feature matrix to obtain a second feature matrix corresponding to each weight of the second row of the convolution kernel.

[0145] In some embodiments, the deconvolution computing device 100 further comprises a register 30 connected to the adder tree 20.

[0146] The adder tree 20 is configured to calculate a plurality of second feature matrices corresponding to a plurality of weights of the first row of the convolution kernel to obtain a third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel.

[0147] The register 30 is configured to store the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel.

[0148] In some embodiments, the adder tree 20 is configured to calculate a plurality of second feature matrices corresponding to a plurality of weights of the second row of the convolution kernel to obtain a third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel.

[0149] In some embodiments, the register 30 is configured to transmit the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel to the adder tree 20.

[0150] The adder tree 20 is configured to calculate the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel to obtain the third feature matrix.

[0151] In some embodiments, when corresponding bit accumulation is needed for the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel, the adder tree 20 is configured to perform corresponding bit accumulation on the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain a third feature matrix.

[0152] In some embodiments, when corresponding bit accumulation is not needed for the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel, the adder tree 20 is configured to perform corresponding bit concatenation on the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain a third feature matrix.

[0153] In some embodiments, the adder tree 20 is configured to perform corresponding bit accumulation on the plurality of first extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain a third feature matrix.

[0154] In some embodiments, each of the first extended feature matrices is obtained by performing zero value extension on a second feature matrix corresponding to each of the weights of each of the rows of the convolution kernel according to a first preset rule.

[0155] In some embodiments, each of the second feature matrices corresponding to each of the weights of each of the rows of the convolution kernel is extended by zero value extension at complementary positions according to the position of each of the weights of each of the rows of the convolution kernel in the convolution kernel to obtain each of the first extended feature matrices corresponding to each of the weights of each of the rows of the convolution kernel.

[0156] In some embodiments, the adder tree 20 is configured to perform corresponding bit concatenation on the plurality of second extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain a third feature matrix.

[0157] In some embodiments, each of the second extended feature matrices is obtained by performing zero value extension on a second feature matrix corresponding to each of the weights of each of the rows of the convolution kernel according to a second preset rule.

[0158] In some embodiments, each of the feature values in each of the second feature matrices corresponding to each of the weights of each of the rows of the convolution kernel is extended by zero value extension at complementary positions according to the position of each of the weights of each of the rows of the convolution kernel in the convolution kernel to obtain each of the second extended feature matrices corresponding to each of the weights of each of the rows of the convolution kernel.

[0159] Referring to FIG. 9, in some embodiments, the systolic array 10 includes a plurality of processing units 11 arranged in rows and columns, each of the processing units 11 includes a multiplier 111, and the plurality of multipliers 111 corresponding to the plurality of processing units 11 in each column are configured to perform multiplication operation on each of the weights of each of the rows of the convolution kernel and the plurality of feature values of the plurality of rows of the first feature matrix, respectively, to obtain each of the second feature matrices corresponding to each of the weights of each of the rows of the convolution kernel.

[0160] In some embodiments, the adder tree 20 comprises a plurality of nodes for performing corresponding bit calculation on each row of the plurality of second feature matrices corresponding to the plurality of weights of each row of the convolution kernel, to obtain a third feature matrix.

[0161] It should be noted that the above-mentioned embodiments of the deconvolution calculation method are also applicable to the deconvolution calculation device 100 of the embodiments of the present application, and will not be described here.

[0162] Referring to FIG. 19, the embodiments of the present application further provide a deconvolution calculation system 300. The deconvolution calculation system 300 comprises one or more processors 310 and a memory 320, and the memory 320 stores a computer program. When the computer program is executed by the processor 310, the deconvolution calculation method of any of the above-mentioned embodiments is implemented.

[0163] For example, when the computer program is executed by the processor 310, the deconvolution calculation method as follows is implemented:

[0164] 010: obtaining a convolution kernel and a first feature matrix;

[0165] 020: inputting the convolution kernel and the first feature matrix into the deconvolution calculation device 100;

[0166] 030: performing deconvolution calculation on the convolution kernel and the first feature matrix based on the deconvolution calculation device 100, to obtain a third feature matrix.

[0167] For another example, when the computer program is executed by the processor 310, the deconvolution calculation method as follows is implemented:

[0168] 021: inputting the plurality of weights of each row of the convolution kernel into the systolic array 10 column by column;

[0169] 022: inputting the plurality of feature values of the first feature matrix into the systolic array 10 row by row.

[0170] It should be noted that the above-mentioned embodiments of the deconvolution calculation method and the deconvolution calculation device 100 are also applicable to the deconvolution calculation system 300 of the embodiments of the present application, and will not be described here.

[0171] Referring to FIG. 20, the embodiments of the present application further provide a computer readable storage medium 400, which stores a computer program 410. When the program is executed by a processor 420, the deconvolution calculation method of any of the above-mentioned embodiments is implemented.

[0172] For example, when the computer program 410 is executed by the processor 420, the deconvolution calculation method as follows is implemented:

[0173] 010: obtaining the convolution kernel and the first feature matrix;

[0174] 020: inputting the convolution kernel and the first feature matrix into the deconvolution computing device 100;

[0175] 030: based on the deconvolution computing device 100, performing deconvolution calculation on the convolution kernel and the first feature matrix to obtain a third feature matrix.

[0176] For another example, when the computer program 410 is executed by the processor 420, the deconvolution calculation method as follows is implemented:

[0177] 021: inputting the plurality of weights of each row of the convolution kernel into the systolic array 10 column by column;

[0178] 022: inputting the plurality of feature values of each row of the first feature matrix into the systolic array 10 row by row.

[0179] It should be noted that the above-mentioned embodiments of the deconvolution calculation method and the deconvolution computing device 100 are also applicable to the computer readable storage medium 400 of the embodiments of the present application, and will not be described here.

[0180] In summary, in the deconvolution calculation method, the deconvolution computing device 100, the computing system 300 and the computer readable storage medium 400 of the embodiments of the present application, when performing deconvolution calculation, the convolution kernel and the first feature matrix are directly input into the deconvolution computing device 100, and the deconvolution calculation is performed by the systolic array 10 and the adder tree 20 of the deconvolution computing device 100 to obtain the third feature matrix, which is the deconvolution calculation result of the first feature matrix, without any additional operation. In this way, the computing resources of the deconvolution computing device 100 are all used for the calculation of effective data, and the storage resources are not wasted on storing zero values that do not contribute to the calculation, effectively improving the hardware computing efficiency of the deconvolution computing device 100.

[0181] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0182] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0183] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, a computer-readable storage medium can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0184] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0185] Those skilled in the art can understand that all or part of the steps of the method carried out by the above-mentioned embodiments can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof. In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically alone, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of software function module. The integrated module, if realized in the form of software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0186] Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A deconvolution calculation method, wherein, The application is applied to a deconvolution calculation device (100), the deconvolution calculation device (100) comprises a systolic array (10) and an adder tree (20), the systolic array (10) is connected with the adder tree (20), and the deconvolution calculation method comprises: a convolution kernel and a first feature matrix are input into the systolic array (10); each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array (10), and a second feature matrix corresponding to each weight of each row of the convolution kernel is obtained; a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel are calculated by the adder tree (20), and a third feature matrix is obtained, wherein the third feature matrix is a deconvolution calculation result of the first feature matrix.

2. The deconvolution calculation method of claim 1, wherein, the convolution kernel and the first feature matrix are input into the systolic array (10), comprising: a plurality of weights of each row of the convolution kernel are input into the systolic array (10) by column; a plurality of feature values of a plurality of rows of the first feature matrix are input into the systolic array (10) by row.

3. The deconvolution calculation method of claim 1, wherein, each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array (10), and a second feature matrix corresponding to each weight of each row of the convolution kernel is obtained, comprising: each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array (10) at different time points, and a second feature matrix corresponding to each weight of each row of the convolution kernel is obtained.

4. The deconvolution calculation method of claim 1, wherein, a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel are calculated by the adder tree (20), and a third feature matrix is obtained, comprising: a plurality of second feature matrices corresponding to a plurality of weights of each row of the convolution kernel are calculated by the adder tree (20), and a third sub-feature matrix corresponding to a plurality of weights of each row of the convolution kernel is obtained; a plurality of third sub-feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel are calculated by the adder tree (20), and the third feature matrix is obtained.

5. The deconvolution calculation method of claim 4, wherein, each weight of each row of the convolution kernel is calculated with the first feature matrix by the systolic array (10), and a second feature matrix corresponding to each weight of each row of the convolution kernel is obtained, comprising: at a first time point, each weight of a first row of the convolution kernel is calculated with the first feature matrix by the systolic array (10), and a second feature matrix corresponding to each weight of the first row of the convolution kernel is obtained; at a second time point, each weight of a second row of the convolution kernel is calculated with the first feature matrix by the systolic array (10), and a second feature matrix corresponding to each weight of the second row of the convolution kernel is obtained.

6. The deconvolution calculation method of claim 5, wherein, The deconvolution calculation device (100) further comprises a register (30) connected with the adder tree (20), and the calculation of the plurality of second feature matrices corresponding to the plurality of weights of each row of the convolution kernel through the adder tree (20) obtains a third sub-feature matrix corresponding to the plurality of weights of each row of the convolution kernel, comprising: The calculation of the plurality of second feature matrices corresponding to the plurality of weights of the first row of the convolution kernel through the adder tree (20) obtains a third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel. The deconvolution calculation method further comprises: The third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel is stored in the register (30).

7. The deconvolution calculation method of claim 6, wherein, The calculation of the plurality of second feature matrices corresponding to the plurality of weights of each row of the convolution kernel through the adder tree (20) obtains a third sub-feature matrix corresponding to the plurality of weights of each row of the convolution kernel, further comprising: The calculation of the plurality of second feature matrices corresponding to the plurality of weights of the second row of the convolution kernel through the adder tree (20) obtains a third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel.

8. The deconvolution calculation method of claim 7, wherein, The deconvolution calculation method further comprises: The third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel is transmitted to the adder tree (20) through the register (30); The calculation of the plurality of third sub-feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix, comprising: The calculation of the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel through the adder tree (20) obtains the third feature matrix.

9. The deconvolution calculation method according to any one of claims 1 to 3, wherein, The calculation of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix, comprising: According to the flag bit of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel, it is judged whether the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel need to be accumulated in corresponding bits; When the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel need to be accumulated in corresponding bits, the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are accumulated in corresponding bits through the adder tree (20) to obtain the third feature matrix.

10. The deconvolution calculation method of claim 9, wherein, The calculation of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix, further comprising: When the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel do not need to be accumulated in corresponding bits, the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel are spliced in corresponding bits through the adder tree (20) to obtain the third feature matrix.

11. The deconvolution calculation method of claim 9, wherein, The corresponding bit accumulation of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix, including: According to a first preset rule, a second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended to obtain a first extended feature matrix corresponding to each weight of each row of the convolution kernel. The corresponding bit accumulation of the plurality of first extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix.

12. The deconvolution calculation method of claim 11, wherein, According to a first preset rule, a second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended to obtain a first extended feature matrix corresponding to each weight of each row of the convolution kernel. According to the position of each weight of each row of the convolution kernel in the convolution kernel, the second feature matrix corresponding to each weight of each row of the convolution kernel is complementarily zero value extended as a whole to obtain the first extended feature matrix corresponding to each weight of each row of the convolution kernel.

13. The deconvolution calculation method of claim 10, wherein, The corresponding bit accumulation of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix, including: According to a second preset rule, a second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended to obtain a second extended feature matrix corresponding to each weight of each row of the convolution kernel. The corresponding bit accumulation of the plurality of second extended feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel through the adder tree (20) obtains the third feature matrix.

14. The deconvolution calculation method of claim 13, wherein, According to a second preset rule, a second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended to obtain a second extended feature matrix corresponding to each weight of each row of the convolution kernel. According to the position of each weight of each row of the convolution kernel in the convolution kernel, each feature value in the second feature matrix corresponding to each weight of each row of the convolution kernel is complementarily zero value extended to obtain the second extended feature matrix corresponding to each weight of each row of the convolution kernel.

15. The deconvolution calculation method according to any one of claims 1 to 14, wherein, The systolic array (10) includes a plurality of processing units (11) arranged in rows and columns, each processing unit (11) including a multiplier (111), and the calculation of each weight of each row of the convolution kernel and the first feature matrix through the systolic array (10) to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel, including: Each column of the plurality of processing units (11) corresponding to the plurality of multipliers (111) respectively performs multiplication operation on each weight of each row of the convolution kernel and a plurality of rows of feature values of the first feature matrix to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel.

16. The deconvolution calculation method according to any one of claims 1 to 14, wherein, The adder tree (20) comprises a plurality of nodes, and the plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel are calculated through the adder tree (20) to obtain a third feature matrix, comprising: Each row of the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel is calculated by the plurality of nodes respectively to obtain the third feature matrix.

17. An apparatus for deconvolution computation (100), wherein The deconvolution calculation device (100) comprises a systolic array (10) and an adder tree (20), and the systolic array (10) is connected with the adder tree (20). The systolic array (10) is used for receiving an input convolution kernel and a first feature matrix. The systolic array (10) is further used for calculating each weight of each row of the convolution kernel with the first feature matrix to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel. The adder tree (20) is used for calculating a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel to obtain a third feature matrix, which is a deconvolution calculation result of the first feature matrix.

18. The deconvolution computing apparatus (100) of claim 17, wherein, The plurality of weights of each row of the convolution kernel is input into the systolic array (10) by column. The plurality of rows of feature values of the first feature matrix is input into the systolic array (10) by row.

19. The deconvolution computing apparatus (100) of claim 17, wherein, The systolic array (10) is used for calculating each weight of different rows of the convolution kernel with the first feature matrix at different time instants respectively to obtain a second feature matrix corresponding to each weight of different rows of the convolution kernel.

20. The deconvolution computing apparatus (100) of claim 17, wherein, The adder tree (20) is used for calculating a plurality of third sub-feature matrices corresponding to a plurality of weights of each row of the convolution kernel to obtain a third feature matrix corresponding to the plurality of weights of each row of the convolution kernel. The adder tree (20) is used for calculating a plurality of third sub-feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel to obtain the third feature matrix.

21. The deconvolution computing apparatus (100) of claim 20, wherein, At a first time instant, the systolic array (10) is used for calculating each weight of a first row of the convolution kernel with the first feature matrix to obtain a second feature matrix corresponding to each weight of the first row of the convolution kernel. At a second time instant, the systolic array (10) is used for calculating each weight of a second row of the convolution kernel with the first feature matrix to obtain a second feature matrix corresponding to each weight of the second row of the convolution kernel.

22. The deconvolution computing apparatus (100) of claim 21, wherein, The deconvolution calculation device (100) further comprises a register (30), and the register (30) is connected with the adder tree (20). The adder tree (20) is used for calculating a plurality of second feature matrices corresponding to a plurality of weights of the first row of the convolution kernel to obtain a third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel. The register (30) is used for storing the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel.

23. The deconvolution computing apparatus (100) of claim 22, wherein, The adder tree (20) is used for calculating a plurality of second feature matrices corresponding to a plurality of weights of the second row of the convolution kernel to obtain a third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel.

24. The deconvolution computing apparatus (100) of claim 23, wherein, The register (30) is used for transmitting the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel to the adder tree (20). The adder tree (20) is used for calculating the third sub-feature matrix corresponding to the plurality of weights of the first row of the convolution kernel and the third sub-feature matrix corresponding to the plurality of weights of the second row of the convolution kernel to obtain the third feature matrix.

25. The deconvolution computing apparatus (100) according to any one of claims 17 to 24, wherein, When corresponding bit accumulation is required for a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel, the adder tree (20) is used for performing corresponding bit accumulation on the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain the third feature matrix.

26. The deconvolution computing apparatus (100) of claim 25, wherein, When corresponding bit accumulation is not required for a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel, the adder tree (20) is used for performing corresponding bit splicing on the plurality of second feature matrices corresponding to the plurality of weights of the plurality of rows of the convolution kernel to obtain the third feature matrix.

27. The deconvolution computing apparatus (100) of claim 25, wherein, The adder tree (20) is used for performing corresponding bit accumulation on a plurality of first extended feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel to obtain the third feature matrix. Each weight of each row of the convolution kernel corresponds to a first extended feature matrix, and the first extended feature matrix is obtained by zero value extension of a second feature matrix corresponding to each weight of each row of the convolution kernel according to a first preset rule.

28. The deconvolution computing apparatus (100) of claim 27, wherein, The second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended in a complementary position according to the position of each weight of each row of the convolution kernel in the convolution kernel to obtain the first extended feature matrix corresponding to each weight of each row of the convolution kernel.

29. The deconvolution computing apparatus (100) of claim 26, wherein, The adder tree (20) is used for performing corresponding bit splicing on a plurality of second extended feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel to obtain the third feature matrix. Each weight of each row of the convolution kernel corresponds to a second extended feature matrix, and the second extended feature matrix is obtained by zero value extension of a second feature matrix corresponding to each weight of each row of the convolution kernel according to a second preset rule.

30. The deconvolution computing apparatus (100) of claim 29, wherein, Each feature value in the second feature matrix corresponding to each weight of each row of the convolution kernel is zero value extended in a complementary position according to the position of each weight of each row of the convolution kernel in the convolution kernel to obtain the second extended feature matrix corresponding to each weight of each row of the convolution kernel.

31. The deconvolution computing apparatus (100) according to any one of claims 17-30, wherein, The pulsating array (10) comprises a plurality of processing units (11) arranged in rows and columns, each processing unit (11) comprising a multiplier (111), and a plurality of multipliers (111) corresponding to a plurality of processing units (11) in each column are used to multiply each weight of each row of the convolution kernel with a plurality of feature values of a plurality of rows of the first feature matrix respectively to obtain a second feature matrix corresponding to each weight of each row of the convolution kernel.

32. The deconvolution computing apparatus (100) according to any one of claims 17-30, wherein, The adder tree (20) comprises a plurality of nodes, and the plurality of nodes are used to perform corresponding bit calculation on each row of a plurality of second feature matrices corresponding to a plurality of weights of a plurality of rows of the convolution kernel respectively to obtain the third feature matrix.

33. A deconvolution computing system (300), wherein The deconvolution computing system (300) comprises one or more processors (310) and a memory (320), and the memory (320) stores a computer program, and the computer program is executed by the processor (310) to implement the deconvolution computing method in any one of claims 1-16.

34. A computer readable storage medium (400) having stored thereon a computer program (410), wherein, The program is executed by the processor (420) to implement the deconvolution computing method in any one of claims 1-16.

Images