A convolution operator execution method, device, storage medium, and program product

Abstract

Embodiments of the present application provide a convolution operator execution method and device, a storage medium and a program product, relating to the technical field of artificial intelligence. The method comprises: for any adjacent first processing region and second processing region in an input tensor, first loading the first processing region from the video memory to perform convolution calculation and obtain a first calculation result. Then, the overlapping region of the first processing region and the second processing region is determined, and the remaining region of the second processing region except the overlapping region is loaded from the video memory. In this way, the second processing region can be spliced based on the overlapping region and the remaining region and convolution calculation is performed on the second processing region, thereby avoiding repeated loading of the overlapping region, effectively reducing the time consumption caused by loading data from the video memory, improving the efficiency of convolution calculation, and further improving the performance of the convolution operator.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, in particular to a convolution operator execution method, device, storage medium and program product in the field of artificial intelligence chips. BACKGROUND

[0002] An artificial intelligence model generally refers to a neural network model trained to perform inference and prediction, such as an image inference model, a speech inference model, etc. The calculation of an artificial intelligence model can be implemented by an operator in a computation graph. An operator refers to various calculations performed on tensors of each layer in an artificial intelligence model, such as convolution calculations performed by a convolution layer of an artificial intelligence model on input data of the artificial intelligence model, which are convolution operators.

[0003] In related technologies, in the execution process of a convolution operator, a region in a feature map is loaded from a video memory each time, and then a loaded convolution kernel is used to perform convolution calculation on the region. When performing convolution calculation on two adjacent regions, a situation of repeatedly loading an overlapping region of the two adjacent regions occurs, which causes a large amount of time to be spent on data loading in the convolution calculation process, thereby affecting the convolution calculation efficiency. SUMMARY

[0004] Embodiments of the present application provide a convolution operator execution method, device, storage medium and program product, which are used to improve the convolution calculation efficiency and thus improve the performance of the convolution operator.

[0005] In one aspect, the present application provides a convolution operator execution method, which comprises:

[0006] For a first processing region and a second processing region adjacent to each other in an input tensor, the following operations are performed:

[0007] Loading the first processing region from a video memory for convolution calculation to obtain a first calculation result;

[0008] Determining an overlapping region of the first processing region and the second processing region, and loading a remaining region of the second processing region except the overlapping region from the video memory;

[0009] Performing convolution calculation on the second processing region composed of the overlapping region and the remaining region to obtain a second calculation result.

[0010] In one aspect, the present application provides a convolution operator execution device, which comprises:

[0011] The loading module is configured to, for a first processing region and a second processing region adjacent to each other in the input tensor, load the first processing region from the video memory for convolution calculation to obtain a first calculation result, determine an overlapping region of the first processing region and the second processing region, and load a remaining region of the second processing region excluding the overlapping region from the video memory.

[0012] The calculation module is configured to perform convolution calculation on the second processing region composed of the overlapping region and the remaining region to obtain a second calculation result.

[0013] Optionally, the loading module is specifically configured to:

[0014] determine the overlapping region of the first processing region and the second processing region based on position information of the first processing region in the input tensor and position information of the second processing region in the input tensor.

[0015] Optionally, the loading module is specifically configured to:

[0016] determine a first overlapping range of the first processing region and the second processing region in a horizontal dimension based on position information of the first processing region and the second processing region in the horizontal dimension in the input tensor.

[0017] determine a second overlapping range of the first processing region and the second processing region in a vertical dimension based on position information of the first processing region and the second processing region in the vertical dimension in the input tensor.

[0018] determine the overlapping region of the first processing region and the second processing region based on the first overlapping range and the second overlapping range.

[0019] Optionally, when the first processing region and the second processing region are two regions adjacent to each other in the horizontal dimension, the overlapping region includes L columns of data of the first processing region and L columns of data of the second processing region that overlap each other, where L is a positive integer.

[0020] Optionally, the L is 2, the first processing region includes a first calculation region and a first padding region, and the second processing region includes a second calculation region and a second padding region.

[0021] the L columns of data of the first processing region include last column data of the first calculation region and last column data of the first padding region, and the L columns of data of the second processing region include first column data of the second padding region and first column data of the second calculation region.

[0022] the last column of data of the first calculation region overlaps with the first column of data of the second padding region; and the last column of data of the first padding region overlaps with the first column of data of the second calculation region.

[0023] Optionally, when the first processing region and the second processing region are two regions adjacent in the longitudinal dimension and calculated, the overlapping region comprises: M rows of data at the rear of the first processing region and M rows of data at the front of the second processing region which overlap, wherein M is a positive integer.

[0024] Optionally, the M = 2, the first processing region comprises: a first calculation region and a first padding region; and the second processing region comprises: a second calculation region and a second padding region.

[0025] M rows of data at the rear of the first processing region comprise: the last row of data of the first calculation region and the last row of data of the first padding region; and M rows of data at the front of the second processing region comprise: the first row of data of the second padding region and the first row of data of the second calculation region.

[0026] the last row of data of the first calculation region overlaps with the first row of data of the second padding region; and the last row of data of the first padding region overlaps with the first row of data of the second calculation region.

[0027] Optionally, the loading module is specifically configured to:

[0028] load the first processing region from the video memory to the on-chip cache;

[0029] read the first processing region from the on-chip cache to perform convolution calculation, and obtain a first calculation result.

[0030] Optionally, the loading module is specifically configured to:

[0031] move the overlapping region to a starting position of the on-chip cache;

[0032] load the remaining region of the second processing region except the overlapping region from the video memory to a remaining position of the on-chip cache except the starting position.

[0033] Optionally, the calculation module is specifically configured to:

[0034] read the second processing region from the on-chip cache to perform convolution calculation, and obtain a second calculation result.

[0035] In one aspect, the embodiment of the present application provides a computer device, comprising a memory, an artificial intelligence chip and a computer program stored in the memory and capable of running on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned convolution operator execution method when executing the computer program.

[0036] In one aspect, the embodiment of the present application provides a computer readable storage medium storing a computer program capable of being executed by a computer device, wherein when the computer program runs on the computer device, the computer device executes the steps of the above-mentioned convolution operator execution method.

[0037] In one aspect, the embodiment of the present application provides a computer program product, comprising a computer program stored in a computer readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned convolution operator execution method.

[0038] In the embodiment of the present application, for the first processing region and the second processing region of any adjacent calculation in the input tensor, the first processing region is loaded from the video memory to perform convolution calculation, and the first calculation result is obtained. Then, the overlapping region of the first processing region and the second processing region is determined, and the remaining region except the overlapping region in the second processing region is loaded from the video memory. In this way, the second processing region can be spliced based on the overlapping region and the remaining region, and convolution calculation is performed on the second processing region, which avoids repeated loading of the overlapping region, effectively reduces the time consumption caused by loading data from the video memory, thereby improving the efficiency of convolution calculation, and further improving the performance of the convolution operator. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 A structure schematic diagram of an artificial intelligence chip provided by the embodiment of the present application is provided.

[0041] Figure 2 A flowchart of a convolution operator execution method provided by the embodiment of the present application is provided.

[0042] Figure 3 A schematic diagram of an input tensor provided by the embodiment of the present application is provided.

[0043] Figure 4 A schematic diagram of a method for determining an overlapping region provided by the embodiment of the present application is provided.

[0044] Figure 5 A schematic diagram of a method for determining an overlapping region provided by an embodiment of the present application;

[0045] Figure 6 A schematic diagram of a method for determining an overlapping region provided by an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a method for determining an overlapping region provided by an embodiment of the present application;

[0047] Figure 8 A schematic diagram of a method for determining an overlapping region provided by an embodiment of the present application;

[0048] Figure 9 A structural schematic diagram of a convolution operator execution device provided by an embodiment of the present application;

[0049] Figure 10 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0051] Reference Figure 1 It is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application, which comprises at least a video memory 101, an on-chip cache 102 and a plurality of computing units 103.

[0052] The video memory 101 can be a high bandwidth memory (HBM) or other types of memory. The on-chip cache 102 is a temporary memory with a smaller capacity than the video memory 101, but faster data read-write speed than the video memory 101.

[0053] The computing unit 103 can be used to execute various types of operators, including but not limited to convolution operators, attention mechanism operators and matrix multiplication operators.

[0054] The computing unit 103 can read and write data to the video memory 101 or the on-chip cache 102. For example, the computing unit 103 can read data from the video memory 101 for calculation; or read data from the video memory 101, write the read data to the on-chip cache 102, and then read data from the on-chip cache 102 for calculation.

[0055] In addition to the above structure, the artificial intelligence chip 100 in the present application can also include other structures, which are not limited in the present application.

[0056] The artificial intelligence chip 100 can be a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc.

[0057] Under the related art, the execution process of the convolution operator includes performing convolution calculation on each region in the feature map. In the execution process of the convolution operator, each time a region in the feature map is loaded from the video memory, and then the loaded convolution kernel is used to perform convolution calculation on the region. Since in the convolution calculation process, a region not only contains a calculation region, but also includes a padding region; that is, each time in addition to calculating the calculation region of the region, the additional padding region in the region also needs to be calculated.

[0058] In order to ensure that all calculation regions in the feature map are subjected to convolution calculation, when performing convolution calculation on two adjacent regions, the case of repeatedly loading the overlapping regions of the two adjacent regions will occur, which causes a large amount of time to be spent on data loading in the convolution calculation process, thereby affecting the convolution calculation efficiency.

[0059] In view of this, the present application provides a convolution operator execution method based on Figure 1 The system architecture diagram is shown in FIG. 1, and a flow of a convolution operator execution method is provided, as shown in FIG. 2, which is executed by an artificial intelligence chip and includes the following steps: Figure 2

[0060] Step 201, for the first processing region and the second processing region adjacent to the input tensor, the following steps 2011 to 2013 are performed respectively:

[0061] Specifically, when performing convolution calculation on input data, the input data includes four dimensions of N, IC, H and W, wherein N represents batch, IC represents input channel number, and H and W represent the height and width of the data in each input channel.

[0062] Since the calculation method of the data in each input channel is the same when performing convolution calculation, the present application takes the data in one input channel in one batch as an example for expansion.

[0063] ​Specifically, data in one input channel is taken as a raw calculation region, and then other elements (such as numbers) are filled in the periphery of the raw calculation region according to a preset padding range, to obtain an input tensor.

[0064] For example, referring to Figure 3 , the raw calculation region is set as an 8-row 8-column matrix, and the preset padding range is 1. Fixed numbers are filled in the periphery of the raw calculation region according to the preset padding range, to obtain an input tensor (a 10-row 10-column matrix). The fixed numbers filled in the periphery of the raw calculation region constitute a padding region.

[0065] In this application, when performing convolution calculation on an input tensor, the input tensor is first divided into multiple processing regions, and then convolution calculation is performed on the multiple processing regions in sequence, where the traversal manner can be: first in the horizontal (W) dimension and then in the vertical (H) dimension, or first in the vertical (H) dimension and then in the horizontal (W) dimension, etc. The first processing region and the second processing region are any two adjacent regions in the input tensor that perform convolution calculation; that is, the first processing region in the input tensor is traversed first and convolution calculation is performed on the first region, and then the second processing region in the input tensor is traversed and convolution calculation is performed on the second region; or, the second processing region in the input tensor is traversed first and convolution calculation is performed on the second region, and then the first processing region in the input tensor is traversed and convolution calculation is performed on the first region.

[0066] The size of the first processing region and the second processing region can be the same or different.

[0067] It should be noted that "first" and "second" in this application are only used to distinguish different regions, and are not used to limit the execution order.

[0068] Step 2011, loading the first processing region from the video memory for convolution calculation to obtain a first calculation result.

[0069] Specifically, the first processing region is loaded from the video memory, and then convolution calculation is performed on the first processing region using a convolution kernel (also referred to as a weight), to obtain a first calculation result.

[0070] In some embodiments, the first processing region is loaded from the video memory to an on-chip cache. The first processing region is read from the on-chip cache for convolution calculation to obtain a first calculation result.

[0071] Specifically, an on-chip cache of a certain size is allocated for the convolution operator in advance, and the read-write speed of the on-chip cache is faster than that of the video memory. When the convolution operator is executed, the first processing region is first loaded from the video memory to the on-chip cache. Then the first processing region is read from the on-chip cache for convolution calculation to obtain a first calculation result.

[0072] At step 2012, an overlapping region of the first processing region and the second processing region is determined, and a remaining region of the second processing region except the overlapping region is loaded from the video memory.

[0073] In some embodiments, the overlapping region of the first processing region and the second processing region is determined based on position information of the first processing region in the input tensor and position information of the second processing region in the input tensor.

[0074] The position information of the first processing region in the input tensor includes position information of the first processing region in a width (W) dimension of the input tensor and position information of the first processing region in a height (H) dimension of the input tensor. In specific implementations, the position information of the first processing region in the width dimension of the input tensor includes a start position and an end position of the first processing region in the width dimension of the input tensor. The position information of the first processing region in the height dimension of the input tensor includes a start position and an end position of the first processing region in the height dimension of the input tensor.

[0075] The position information of the second processing region in the input tensor includes position information of the second processing region in the width dimension of the input tensor and position information of the second processing region in the height dimension of the input tensor. In specific implementations, the position information of the second processing region in the width dimension of the input tensor includes a start position and an end position of the second processing region in the width dimension of the input tensor. The position information of the second processing region in the height dimension of the input tensor includes a start position and an end position of the second processing region in the height dimension of the input tensor.

[0076] A first overlapping range of the first processing region and the second processing region in the width dimension is determined based on the position information of the first processing region and the second processing region in the width dimension of the input tensor.

[0077] A second overlapping range of the first processing region and the second processing region in the height dimension is determined based on the position information of the first processing region and the second processing region in the height dimension of the input tensor.

[0078] The overlapping region of the first processing region and the second processing region is determined based on the first overlapping range and the second overlapping range.

[0079] For example, referring to Figure 4 , an input tensor is set as a 12x12 matrix.

[0080] The first processing region 401 is a 4-row 7-column matrix. The start position and the end position of the first processing region 401 in the horizontal (W) dimension of the input tensor are the first column and the seventh column respectively. The start position and the end position of the first processing region 401 in the vertical (H) dimension of the input tensor are the first row and the fourth row respectively.

[0081] The second processing region 402 is a 4-row 7-column matrix. The start position and the end position of the second processing region 402 in the horizontal (W) dimension of the input tensor are the sixth column and the twelfth column respectively. The start position and the end position of the second processing region 402 in the vertical (H) dimension of the input tensor are the first row and the fourth row respectively.

[0082] Based on the above position information, it can be known that the overlapping range of the first processing region 401 and the second processing region 402 in the horizontal (W) dimension is the sixth column to the seventh column; the overlapping range of the first processing region 401 and the second processing region 402 in the vertical (H) dimension is the first row to the fourth row; therefore, the position information of the overlapping region 403 of the first processing region 401 and the second processing region 402 in the input tensor includes that the start position and the end position in the horizontal (W) dimension are the sixth column and the seventh column respectively; the start position and the end position in the vertical (H) dimension are the first row and the fourth row respectively.

[0083] It should be noted that when the overlapping region of the first processing region and the second processing region is empty, the remaining region of the second processing region except the overlapping region is loaded from the video memory.

[0084] In some embodiments, the overlapping region is moved to the start position of the on-chip cache. The remaining region of the second processing region except the overlapping region is loaded from the video memory to the remaining position except the start position of the on-chip cache.

[0085] Specifically, the overlapping region of the first processing region and the second processing region is generally the tail region of the first processing region and the start region of the second processing region. When the first processing region is loaded from the video memory to the on-chip cache for convolution calculation, the tail region of the first processing region is often saved in the tail position of the on-chip cache.

[0086] During the convolution calculation of the first processing region, the overlapping region of the first processing region and the second processing region has been loaded from the video memory to the on-chip cache. Since the overlapping region is also needed when the convolution calculation of the second processing region is performed, the overlapping region that has been loaded to the on-chip cache is kept in the on-chip cache.

[0087] Since the overlapping region is generally the start region of the second processing region, before the convolution calculation on the second processing region, the overlapping region saved in the on-chip cache is moved from the end position of the on-chip cache to the start position of the on-chip cache. In this way, after the remaining region of the second processing region except the overlapping region is loaded from the display memory, the loaded remaining region can be saved in the remaining position except the start position of the on-chip cache, so as to ensure the continuity and correctness of the second processing region saved in the on-chip cache.

[0088] In addition, after the convolution calculation on the first processing region, the overlapping region loaded into the on-chip cache is reserved in the on-chip cache, so that when the convolution calculation is performed on the second processing region, only the remaining region of the second processing region except the overlapping region needs to be loaded from the display memory to the on-chip cache, avoiding repeated loading of the overlapping region from the display memory to the on-chip cache, which effectively reduces the time consumption and resource overhead caused by repeated loading of data from the display memory to the on-chip cache, thereby improving the efficiency of the convolution calculation and further improving the performance of the convolution operator.

[0089] In step 2013, the convolution calculation is performed on the second processing region composed of the overlapping region and the remaining region to obtain a second calculation result.

[0090] In some embodiments, the second processing region is read from the on-chip cache for the convolution calculation to obtain the second calculation result.

[0091] Specifically, the overlapping region reserved in the on-chip cache is spliced with the remaining region read from the display memory to the on-chip cache to obtain the second processing region. Then the second processing region is read from the on-chip cache for the convolution calculation to obtain the second calculation result.

[0092] In the embodiments of the present application, for any adjacent calculation of the first processing region and the second processing region in the input tensor, the first processing region is loaded from the display memory for the convolution calculation to obtain a first calculation result. Then the overlapping region of the first processing region and the second processing region is determined, and the remaining region of the second processing region except the overlapping region is loaded from the display memory. In this way, the overlapping region and the remaining region can be spliced to form the second processing region and the convolution calculation is performed on the second processing region, which avoids repeated loading of the overlapping region and effectively reduces the time consumption caused by loading data from the display memory, thereby improving the efficiency of the convolution calculation and further improving the performance of the convolution operator.

[0093] In some embodiments, when the first processing region and the second processing region are two regions adjacent in the horizontal dimension, the overlapping region includes L columns of data of the first processing region and L columns of data of the second processing region which overlap, where L is a positive integer.

[0094] Specifically, when the first processing region and the second processing region are two regions adjacent in the transverse dimension, the height of the first processing region and the second processing region is the same, and the width of the first processing region and the second processing region can be the same or different.

[0095] The last L columns of data of the first processing region refer to L columns of data (including the last column of data in the first processing region) continuously selected from the first processing region in a direction from the last column of data in the first processing region to the first column of data in the first processing region. The first L columns of data of the second processing region refer to L columns of data (including the first column of data in the second processing region) continuously selected from the second processing region in a direction from the first column of data in the second processing region to the last column of data in the second processing region.

[0096] The last L columns of data of the first processing region overlap with the first L columns of data of the second processing region, to obtain an overlap region of the first processing region and the second processing region.

[0097] It should be noted that in the present application, the width of each column of data is associated with the hardware attribute of the artificial intelligence chip, and for different hardware attributes, the width of each column of data can be different.

[0098] In some embodiments, L = 2, the first processing region includes a first calculation region and a first padding region, and the second processing region includes a second calculation region and a second padding region. The first padding region surrounds the first calculation region, and the second padding region surrounds the second calculation region.

[0099] The last L columns of data of the first processing region include the last column of data of the first calculation region and the last column of data of the first padding region. Since the first padding region surrounds the first calculation region, the last L columns of data of the first processing region also include padding elements in the first padding region located in the column of the last column of data of the first calculation region.

[0100] The first L columns of data of the second processing region include the first column of data of the second padding region and the first column of data of the second calculation region. Since the second padding region surrounds the second calculation region, the first L columns of data of the second processing region also include padding elements in the second padding region located in the column of the first column of data of the second calculation region.

[0101] The last column of data of the first calculation region overlaps with the first column of data of the second padding region; and the last column of data of the first padding region overlaps with the first column of data of the second calculation region.

[0102] For example, referring to Figure 5, the input tensor is set as: a 12x12 matrix, L=2, the width and height of the first processing region and the second processing region are the same.

[0103] The first processing region is a 4x7 matrix, and the first processing region includes a first calculation region and a first padding region. The first processing region has a start position and an end position in the horizontal (W) dimension of the input tensor, which are the 1st column and the 7th column respectively. The first processing region has a start position and an end position in the vertical (H) dimension of the input tensor, which are the 1st row and the 4th row respectively. The first calculation region is a 2x5 matrix, and the first padding region surrounds the first calculation region.

[0104] The second processing region is a 4x7 matrix. The second processing region includes a second calculation region and a second padding region. The second processing region has a start position and an end position in the horizontal (W) dimension of the input tensor, which are the 6th column and the 12th column respectively. The second processing region has a start position and an end position in the vertical (H) dimension of the input tensor, which are the 1st row and the 4th row respectively. The second calculation region is a 2x5 matrix, and the second padding region surrounds the second calculation region.

[0105] The 6th column data of the first processing region (i.e., the last column data of the first calculation region and the padding elements located in the column of the last column data) and the 1st column data of the second processing region (i.e., the first column data of the second padding region) overlap; and the 7th column data of the first processing region (i.e., the last column data of the first padding region) and the 2nd column data of the second processing region (i.e., the first column data of the second calculation region and the padding elements located in the column of the first column data) overlap, to obtain an overlap region of the first processing region and the second processing region.

[0106] For example, referring to Figure 6 , the input tensor is set as: a 10x10 matrix, L=2, the height of the first processing region and the second processing region is the same, but the width is different.

[0107] The first processing region is a 4x7 matrix, and the first processing region includes a first calculation region and a first padding region. The first processing region has a start position and an end position in the horizontal (W) dimension of the input tensor, which are the 1st column and the 7th column respectively. The first processing region has a start position and an end position in the vertical (H) dimension of the input tensor, which are the 1st row and the 4th row respectively. The first calculation region is a 2x5 matrix, and the first padding region surrounds the first calculation region.

[0108] The second processing region is a matrix of 4 rows and 5 columns. The second processing region includes a second calculation region and a second padding region, wherein the starting position and the ending position of the second processing region in the horizontal (W) dimension of the input tensor are the 6th column and the 10th column respectively. The starting position and the ending position of the second processing region in the vertical (H) dimension of the input tensor are the 1st row and the 4th row respectively. The second calculation region is a matrix of 2 rows and 3 columns, and the second padding region surrounds the second calculation region.

[0109] The 6th column data of the first processing region (i.e., the last column data of the first calculation region and the padding elements located in the column of the last column data) overlaps with the 1st column data of the second processing region (i.e., the first column data of the second padding region); and the 7th column data of the first processing region (i.e., the last column data of the first padding region) overlaps with the 2nd column data of the second processing region (i.e., the first column data of the second calculation region and the padding elements located in the column of the first column data), to obtain an overlap region of the first processing region and the second processing region.

[0110] In some embodiments, when the first processing region and the second processing region are two regions adjacent in the vertical dimension for calculation, the overlap region includes the last M rows of data of the first processing region and the first M rows of data of the second processing region that overlap, wherein M is a positive integer.

[0111] Specifically, when the first processing region and the second processing region are two regions adjacent in the vertical dimension for calculation, the widths of the first processing region and the second processing region are the same, and the heights of the first processing region and the second processing region can be the same or different.

[0112] The last M rows of data of the first processing region refer to L rows of data (including the last row of data of the first processing region) continuously selected from the first processing region starting from the last row of data of the first processing region and towards the first row of data of the first processing region. The first M rows of data of the second processing region refer to M rows of data (including the first row of data of the second processing region) continuously selected from the second processing region starting from the first row of data of the second processing region and towards the last row of data of the second processing region.

[0113] The last M rows of data of the first processing region overlap with the first M rows of data of the second processing region, to obtain an overlap region of the first processing region and the second processing region.

[0114] It should be noted that in this application, the height of each row of data is associated with the hardware properties of the artificial intelligence chip, and for different hardware properties, the height of each row of data can be different.

[0115] In some embodiments, M=2, the first processing region comprises: a first calculation region and a first padding region; and the second processing region comprises: a second calculation region and a second padding region.

[0116] The last M rows of data of the first processing region comprise: the last row of data of the first calculation region and the last row of data of the first padding region. Since the first padding region surrounds the first calculation region, the last M rows of data of the first processing region further comprise: padding elements in the first padding region which are located in the row where the last row of data of the first calculation region is located.

[0117] The first M rows of data of the second processing region comprise: the first row of data of the second padding region and the first row of data of the second calculation region. Since the second padding region surrounds the second calculation region, the last M rows of data of the second processing region further comprise: padding elements in the second padding region which are located in the row where the first row of data of the second calculation region is located.

[0118] The last row of data of the first calculation region overlaps with the first row of data of the second padding region; and the last row of data of the first padding region overlaps with the first row of data of the second calculation region.

[0119] For example, referring to Figure 7 , the input tensor is set as: a 12x12 matrix, M=2, and the width and height of the first processing region and the second processing region are the same.

[0120] The first processing region is a 7x4 matrix. The first processing region comprises: a first calculation region and a first padding region. The first processing region has a starting position and an ending position in the horizontal (W) dimension of the input tensor of: the 1st column and the 4th column, respectively. The first processing region has a starting position and an ending position in the vertical (H) dimension of the input tensor of: the 1st row and the 7th row, respectively. The first calculation region is a 5x2 matrix, and the first padding region surrounds the first calculation region.

[0121] The second processing region is a 7x4 matrix. The second processing region comprises: a second calculation region and a second padding region. The second processing region has a starting position and an ending position in the horizontal (W) dimension of the input tensor of: the 1st column and the 4th column, respectively. The second processing region has a starting position and an ending position in the vertical (H) dimension of the input tensor of: the 6th row and the 12th row, respectively. The second calculation region is a 5x2 matrix, and the second padding region surrounds the second calculation region.

[0122] The 6th row of data of the first processing region (i.e., the last row of data of the first calculation region and the padding elements located in the row of the last row of data) overlaps with the 1st row of data of the second processing region (i.e., the first row of data of the second padding region); and the 7th row of data of the first processing region (i.e., the last row of data of the first padding region) overlaps with the 2nd row of data of the second processing region (i.e., the first row of data of the second calculation region and the padding elements located in the row of the first row of data), to obtain an overlapping region of the first processing region and the second processing region.

[0123] For example, referring to Figure 8 , the input tensor is set as: a 10-row 10-column matrix, M = 2, the widths of the first processing region and the second processing region are the same, and the heights of the first processing region and the second processing region are different.

[0124] The first processing region is a 7-row 4-column matrix, and the first processing region includes a first calculation region and a first padding region. The starting position and the end position of the first processing region in the horizontal (W) dimension of the input tensor are the 1st column and the 4th column respectively. The starting position and the end position of the first processing region in the vertical (H) dimension of the input tensor are the 1st row and the 7th row respectively. The first calculation region is a 5-row 2-column matrix, and the first padding region surrounds the first calculation region.

[0125] The second processing region is a 5-row 4-column matrix. The second processing region includes a second calculation region and a second padding region. The starting position and the end position of the second processing region in the horizontal (W) dimension of the input tensor are the 1st column and the 4th column respectively. The starting position and the end position of the second processing region in the vertical (H) dimension of the input tensor are the 6th row and the 10th row respectively. The second calculation region is a 3-row 2-column matrix, and the second padding region surrounds the second calculation region.

[0126] The 6th row of data of the first processing region (i.e., the last row of data of the first calculation region and the padding elements located in the row of the last row of data) overlaps with the 1st row of data of the second processing region (i.e., the first row of data of the second padding region); and the 7th row of data of the first processing region (i.e., the last row of data of the first padding region) overlaps with the 2nd row of data of the second processing region (i.e., the first row of data of the second calculation region and the padding elements located in the row of the first row of data), to obtain an overlapping region of the first processing region and the second processing region.

[0127] In the embodiments of the present application, for two regions calculated adjacent in the horizontal dimension and two regions calculated adjacent in the longitudinal dimension, the overlap region between the two regions is determined according to the corresponding calculation rule, respectively. Therefore, in different calculation scenarios, repeated loading of the overlap region from the video memory to the on-chip cache can be avoided. In this way, the time consumption and resource consumption caused by data loading are reduced, thereby improving the efficiency of convolution calculation, and the generality of the convolution operator is improved.

[0128] In addition, in order to verify the performance of the convolution operator execution method in the present application, the following method is designed to calculate the saved data transfer amount from the video memory to the on-chip cache:

[0129] Taking the convolution calculation of input data [N, IC, H, W] * convolution kernel [OC, IC, KH, KW] as an example, wherein N represents batch, IC represents input channel number, H and W represent the height and width of data in each input channel; OC represents output channel number, KH represents the height of the convolution kernel, and KW represents the width of the convolution kernel.

[0130] The outer tiling parameter (i.e., the parameter of the data loaded from the video memory each time) is set to [outPartitionN, outPartitionIC, outPartitionH, outPartitionW], wherein outPartitionN represents the batch loaded each time, outPartitionIC represents the input channel number loaded each time, and outPartitionH and outPartitionW represent the height and width of the data (including the calculation region and the padding region) for which the convolution calculation is performed in each input channel.

[0131] For the case that there is an overlap region between two processing regions calculated adjacent in the horizontal dimension, the width paddingW of the overlap region is determined according to the hardware attribute. Then, the following formula (1) can be used to calculate the saved data transfer amount DS when loading data from the video memory to the on-chip cache each time:

[0132]

[0133] demSize represents the number of bytes occupied by the data type of the input data; the data type of the input data includes int32_t (i.e., 32-bit integer, occupying 4 bytes, corresponding to demSize = 4), FP32 (i.e., 32-bit floating point number, occupying 4 bytes, corresponding to demSize = 4), BF16 (i.e., 16-bit floating point number, occupying 2 bytes, corresponding to demSize = 2), etc.

[0134] ceil denotes upward rounding; hReloadTimes denotes the number of times of loading in the vertical (H) dimension; when outPartitionH is greater than or equal to H, the value of hReloadTimes is 1; when outPartitionH is less than or equal to H, the value of hReloadTimes is: H / (outPartitionH-paddingW).

[0135] For example, the parameters of the input data are set as [16, 256, 256, 256], the parameters of the convolution kernel are set as [256, 256, 3, 3], the data type of the input data is BF16, and the layout type of the input data is an Activation type.

[0136] The parameters of the activation data loaded from the video memory to the on-chip cache each time are [outPartitionN, outPartitionIC, outPartitionH, outPartitionW] = [1, 256, 256, 32].

[0137] According to the hardware attribute, the width of the overlap region is determined as paddingW = 16, and the data transfer amount DS = 480 MB that can be saved when the activation data is loaded from the video memory to the on-chip cache each time can be calculated by using the above formula (1), which greatly reduces the time consumption and resource consumption of loading data from the video memory to the on-chip cache.

[0138] It should be noted that, for the case that the two processing regions adjacent in the vertical dimension have an overlap region, the manner used for calculating the saved data transfer amount is the same as that described above for the case that the two processing regions adjacent in the horizontal dimension have an overlap region, which will not be described herein again.

[0139] Based on the same technical concept, an embodiment of the present application provides a structural schematic diagram of a convolution operator execution device, as shown in Figure 9 The convolution operator execution device 900 includes:

[0140] The loading module 901 is configured to perform the following operations for the first processing region and the second processing region adjacent in the input tensor: load the first processing region from the video memory for convolution calculation to obtain a first calculation result; determine an overlap region of the first processing region and the second processing region, and load a remaining region except the overlap region in the second processing region from the video memory.

[0141] The calculation module 902 is configured to perform convolution calculation on the second processing region composed of the overlap region and the remaining region to obtain a second calculation result.

[0142] Optionally, the loading module 901 is specifically configured to:

[0143] determine an overlapping region of the first processing region and the second processing region based on the position information of the first processing region in the input tensor and the position information of the second processing region in the input tensor.

[0144] Optionally, the loading module 901 is specifically configured to:

[0145] determine a first overlapping range of the first processing region and the second processing region in a horizontal dimension based on the position information of the first processing region and the second processing region in the horizontal dimension in the input tensor;

[0146] determine a second overlapping range of the first processing region and the second processing region in a vertical dimension based on the position information of the first processing region and the second processing region in the vertical dimension in the input tensor;

[0147] determine an overlapping region of the first processing region and the second processing region based on the first overlapping range and the second overlapping range.

[0148] Optionally, when the first processing region and the second processing region are two regions calculated adjacent in the horizontal dimension, the overlapping region includes L columns of data of the first processing region and L columns of data of the second processing region that overlap, where L is a positive integer.

[0149] Optionally, the L = 2, the first processing region includes a first calculation region and a first padding region; and the second processing region includes a second calculation region and a second padding region.

[0150] the L columns of data of the first processing region include last column data of the first calculation region and last column data of the first padding region; and the L columns of data of the second processing region include first column data of the second padding region and first column data of the second calculation region.

[0151] the last column data of the first calculation region overlaps with the first column data of the second padding region; and the last column data of the first padding region overlaps with the first column data of the second calculation region.

[0152] Optionally, when the first processing region and the second processing region are two regions calculated adjacent in the vertical dimension, the overlapping region includes M rows of data of the first processing region and M rows of data of the second processing region that overlap, where M is a positive integer.

[0153] Optionally, the M=2, the first processing region comprises a first calculation region and a first padding region; and the second processing region comprises a second calculation region and a second padding region.

[0154] The last M rows of data of the first processing region comprise the last row of data of the first calculation region and the last row of data of the first padding region; and the first M rows of data of the second processing region comprise the first row of data of the second padding region and the first row of data of the second calculation region.

[0155] The last row of data of the first calculation region overlaps the first row of data of the second padding region; and the last row of data of the first padding region overlaps the first row of data of the second calculation region.

[0156] Optionally, the loading module 901 is specifically used for:

[0157] loading the first processing region from the video memory to the on-chip cache;

[0158] reading the first processing region from the on-chip cache to perform convolution calculation and obtaining a first calculation result.

[0159] Optionally, the loading module 901 is specifically used for:

[0160] moving the overlapping region to a starting position of the on-chip cache;

[0161] loading the remaining region of the second processing region except the overlapping region from the video memory to a remaining position of the on-chip cache except the starting position.

[0162] Optionally, the calculation module 902 is specifically used for:

[0163] reading the second processing region from the on-chip cache to perform convolution calculation and obtaining a second calculation result.

[0164] In the embodiments of the present application, for any adjacent first processing region and second processing region in the input tensor, the first processing region is first loaded from the video memory to perform convolution calculation and obtain a first calculation result. Then the overlapping region of the first processing region and the second processing region is determined, and the remaining region of the second processing region except the overlapping region is loaded from the video memory. In this way, the second processing region can be spliced based on the overlapping region and the remaining region and convolution calculation is performed on the second processing region, which avoids repeated loading of the overlapping region, effectively reduces the time consumption caused by loading data from the video memory, and improves the efficiency of convolution calculation, thereby improving the performance of the convolution operator.

[0165] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 10 As shown, it includes at least one artificial intelligence chip 100 and a memory 1001 connected to the at least one artificial intelligence chip. The specific connection medium between the artificial intelligence chip 100 and the memory 1001 is not limited in the embodiment of the present application. Figure 10 For example, the artificial intelligence chip 100 and the memory 1001 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0166] In an embodiment of the present application, the memory 1001 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned convolution operator execution method by executing the instructions stored in the memory 1001.

[0167] Among them, the artificial intelligence chip 100 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device, and realize convolution calculation by running or executing instructions stored in the memory 1001 and calling data stored in the memory 1001. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 1001 can be implemented on the same chip. In some embodiments, they can also be implemented separately on independent chips.

[0168] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0169] The memory 1001 can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1001 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, and the like. The memory 1001 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer device, but is not limited thereto. The memory 1001 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0170] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computer device, which, when executed on the computer device, causes the computer device to perform the steps of the above-mentioned method for performing a convolution operator.

[0171] Based on the same inventive concept, the embodiments of the present application provide a computer program product, which comprises a computer program stored on a computer readable storage medium, the computer program comprising program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-mentioned method for performing a convolution operator.

[0172] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The present application is described in reference to the flowchart illustrations and / or block diagrams according to the methods, apparatus (systems) and computer program products of embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0174] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0176] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to embrace all such variations and modifications as fall within the scope of the present application.

[0177] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of performing a convolution operation, the method comprising: The method comprises: performing the following operations on a first processing region and a second processing region adjacent to each other in an input tensor: loading the first processing region from a video memory to a set-size on-chip cache previously allocated for the convolution operator; reading the first processing region from the on-chip cache for convolution calculation to obtain a first calculation result; determining an overlapping region of the first processing region and the second processing region, moving the overlapping region to a starting position of the on-chip cache, and loading the remaining region of the second processing region except the overlapping region from the video memory to the remaining position of the on-chip cache other than the starting position; the overlapping region is the end region of the first processing region and the start region of the second processing region that overlap each other; the first processing region comprises a first calculation region and a first padding region; and the second processing region comprises a second calculation region and a second padding region; when the first processing region and the second processing region are two regions adjacent to each other in a horizontal dimension, the end region comprises the last column of data of the first calculation region and the last column of data of the first padding region; and the start region comprises the first column of data of the second padding region and the first column of data of the second calculation region; the last column of data of the first calculation region overlaps the first column of data of the second padding region; and the last column of data of the first padding region overlaps the first column of data of the second calculation region; performing convolution calculation on the second processing region composed of the overlapping region and the remaining region to obtain a second calculation result.

2. The method of claim 1, wherein, The determination of the overlapping region of the first processing region and the second processing region comprises: determining the overlapping region of the first processing region and the second processing region based on the position information of the first processing region in the input tensor and the position information of the second processing region in the input tensor.

3. The method of claim 2, wherein, The determination of the overlapping region of the first processing region and the second processing region based on the position information of the first processing region in the input tensor and the position information of the second processing region in the input tensor comprises: determining a first overlapping range of the first processing region and the second processing region in a horizontal dimension based on the position information of the first processing region and the second processing region in the horizontal dimension in the input tensor; determining a second overlapping range of the first processing region and the second processing region in a vertical dimension based on the position information of the first processing region and the second processing region in the vertical dimension in the input tensor; determining the overlapping region of the first processing region and the second processing region based on the first overlapping range and the second overlapping range.

4. The method of claim 1, wherein, The method further comprises: When the first processing region and the second processing region are two regions adjacent in a longitudinal dimension, the overlapping region comprises: M last rows of data of the first processing region and M first rows of data of the second processing region, where M is a positive integer.

5. The method of claim 4, wherein, The M is 2, the first processing region comprises: a first calculation region and a first padding region; and the second processing region comprises: a second calculation region and a second padding region. M last rows of data of the first processing region comprise: last row data of the first calculation region and last row data of the first padding region; and M first rows of data of the second processing region comprise: first row data of the second padding region and first row data of the second calculation region. The last row data of the first calculation region overlaps with the first row data of the second padding region; and the last row data of the first padding region overlaps with the first row data of the second calculation region.

6. The method of any one of claims 1 to 5, wherein, The step of performing convolution calculation on the second processing region composed of the overlapping region and the remaining region to obtain a second calculation result comprises: reading the second processing region from the on-chip cache to perform convolution calculation and obtain a second calculation result.

7. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored on the memory and executable on the artificial intelligence chip, characterized in that, The artificial intelligence chip, when executing the computer program, implements the steps of the method in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer device stores a computer program executable by the computer device, and when the computer program runs on the computer device, the computer device executes the steps of the method in any one of claims 1-6.

9. A computer program product, characterised in that, The computer program product comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method in any one of claims 1-6.

Images