Convolution operation method, convolution operation device, electronic device and storage medium

Abstract

A convolution operation method, convolution operation device, electronic device, and storage medium. The convolution operation method includes: determining a computational convolution kernel, where the computational convolution kernel is obtained based on an initial convolution kernel, where the initial convolution kernel is represented as [R, S, C, K], and the computational convolution kernel is represented as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0; adjusting the arrangement of input data based on the number of channels of the computational convolution kernel to obtain target data, where the size and number of channels of the target data are different from those of the input data, and the number of channels of the target data is equal to the number of channels of the computational convolution kernel; and performing a convolution operation on the target data and the computational convolution kernel to obtain a convolution operation result. The convolution operation result of the target data and the computational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel. This convolution operation method can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, and shorten the convolution operation time.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a convolution operation method, a convolution operation device, an electronic device, and a storage medium. Background Art

[0002] With the development of technology, artificial intelligence (AI) has been widely applied in various fields. Deep learning is one of the key technologies in AI. Deep learning technology based on artificial neural networks has made great progress in fields such as object classification, text processing, image search, and human-computer interaction. Convolutional neural networks (CNNs) are a widely used deep learning technology that can directly input image data without complex preprocessing, offering significant advantages in image processing and other areas. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides a convolution operation method, including: determining an operational convolution kernel, wherein the operational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], and the operational convolution kernel is expressed as [1, 1, (C×R×S), K], where R, S, C, and K are all integers greater than 0; adjusting an arrangement of input data based on the number of channels of the operational convolution kernel to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; performing a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0004] At least one embodiment of the present disclosure further provides a convolution operation device, including: a determination unit, configured to determine an operational convolution kernel, wherein the operational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], the operational convolution kernel is expressed as [1, 1, (C×R×S), K], and R, S, C, and K are all integers greater than 0; an adjustment unit, configured to adjust the arrangement of input data based on the number of channels of the operational convolution kernel to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; a calculation unit, configured to perform a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0005] At least one embodiment of the present disclosure further provides an electronic device, comprising the convolution operation device provided by any embodiment of the present disclosure.

[0006] At least one embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory, comprising at least one computer program module; wherein the at least one computer program module is stored in the memory and configured to be executed by the processor, and the at least one computer program module is used to implement the convolution operation method provided by any embodiment of the present disclosure.

[0007] At least one embodiment of the present disclosure further provides a storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implements the convolution operation method provided by any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0009] Figure 1 A data flow diagram of a convolution operation provided in some embodiments of the present disclosure;

[0010] Figure 2 A flowchart of a convolution operation method provided in some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram of the principle of convolution operation;

[0012] Figure 4 for Figure 2 Schematic flow chart of step S20;

[0013] Figure 5 for Figure 4 Schematic flow chart of step S21;

[0014] Figure 6 A schematic diagram of a storage method of input data in a memory in a convolution operation method provided in some embodiments of the present disclosure;

[0015] Figure 7 A schematic diagram of a storage method of input data in a static memory in a convolution operation method provided in some embodiments of the present disclosure;

[0016] Figure 8 for Figure 4 Schematic flow chart of step S22;

[0017] Figure 9 for Figure 4 Schematic flow chart of step S23;

[0018] Figure 10 for Figure 9 Schematic flow chart of step S232;

[0019] Figure 11 This is one of the schematic diagrams of performing data arrangement transformation in the convolution operation method provided in some embodiments of the present disclosure;

[0020] Figure 12 A second schematic diagram of data arrangement transformation in the convolution operation method provided in some embodiments of the present disclosure;

[0021] Figure 13 A schematic block diagram of a convolution operation device provided in some embodiments of the present disclosure;

[0022] Figure 14 A schematic block diagram of an electronic device provided for some embodiments of the present disclosure;

[0023] Figure 15 A schematic block diagram of another electronic device provided for some embodiments of the present disclosure;

[0024] Figure 16 A schematic block diagram of another electronic device provided for some embodiments of the present disclosure; and

[0025] Figure 17 A schematic diagram of a storage medium provided for some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] The input data for a convolutional neural network is typically a three-channel image. For example, the input image for the first convolution layer of the residual network ResNet50 is [1, 224, 224, 3]. This means the input image has three channels, each with a size of 224×224. The convolution kernel used in the first convolution layer of the residual network ResNet50 is [7, 7, 3, 64]. Common neural network accelerators are equipped with a matrix operation unit (Matrix), which is primarily responsible for accelerating matrix operations and convolution operations in neural networks. To accelerate matrix operations, the Matrix operation unit typically increases computational parallelism by increasing the computational scale, such as 64×64 or 128×128. However, because the input data for the first convolution layer of a convolutional neural network has fewer channels (for example, 3 channels), the computing power utilization of the Matrix operation unit on the hardware accelerator is low. Furthermore, the computation time for the first convolution layer is relatively long, resulting in a less significant acceleration effect. In addition, if the data layout method (Channel Align Tensor Layout) is strictly followed according to the number of channels, the data storage space will be significantly increased and the data transmission time will be increased.

[0033] like Figure 1 As shown, the hardware accelerator is usually mounted on the host's PCIe (Peripheral Component Interconnect Express) node as a slave device of the host. PCIe is a high-speed serial computer expansion bus standard that can achieve high-speed data transmission. Relative to the host side, the hardware accelerator is the device side. When performing convolution operations, the data input to the first layer of convolution needs to be sent from the host to the hardware accelerator via PCIe. This process is called Host2Device. For example, the central processing unit (CPU) reads data from the memory, transmits it to the hardware accelerator on the device side via PCIe, and stores it in the memory on the hardware accelerator (for example, in DDR). Afterwards, the hardware accelerator can use this data to perform convolution operations.

[0034] Taking the first convolution layer of the ResNet50 residual network as an example, the first convolution layer requires the operation of the input data and the convolution kernel, which can be expressed as: [1,224,224,3]×[7,7,3,64]=[1,112,112,64]. Here, the input data is represented as [1,224,224,3], which means 3 channels of data with a size of 224×224; the convolution kernel is represented as [7,7,3,64], which means 64 groups of 3 convolution kernels each with a size of 7×7. The result is 64 channels of data with a size of 112×112.

[0035] Assuming the matrix operation unit on the hardware accelerator is 64×64, due to the limitations of the channel-aligned data layout, the first-layer convolution input data must be expanded on the host from [1,224,224,3] to [1,224,224,64], with redundant channel data filled with zeros. This requires a 21.33-fold increase in storage space; similarly, the data transfer time from the host to the hardware accelerator also increases by 21.33 times. In this scenario, the matrix operation unit's computing power utilization is only 4.68%. Regarding convolution operation time, the matrix operation unit requires 614,656 cycles to complete the first-layer convolution operation.

[0036] Because the number of input data channels for the first-layer convolution calculation of the convolutional neural network is small and the matrix operation unit of the hardware accelerator is large, the computing requirements do not match the hardware characteristics, which in turn causes the following problems in the first-layer convolution calculation of the convolutional neural network. First, the input data needs to be rearranged using the host's CPU, which increases the storage space and consumes CPU time. Second, the amount of input data increases after rearrangement, and the PCIe transmission time of Host2Device increases. Third, the matrix operation unit of the hardware accelerator has low utilization and cannot exert its full computing power, resulting in a waste of hardware resources. Fourth, the matrix operation unit of the hardware accelerator takes a long time to execute the first-layer convolution calculation, which cannot achieve the purpose of hardware acceleration.

[0037] At least one embodiment of the present disclosure provides a convolution operation method, a convolution operation device, an electronic device, and a storage medium. The convolution operation method can improve the utilization rate of a matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the convolution operation time, improve the computing efficiency, and save data transmission time.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements described.

[0039] At least one embodiment of the present disclosure provides a convolution operation method. The convolution operation method includes: determining an operational convolution kernel, where the operational convolution kernel is obtained based on an initial convolution kernel, where the initial convolution kernel is expressed as [R, S, C, K], and the operational convolution kernel is expressed as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0; adjusting the arrangement of input data based on the number of channels of the operational convolution kernel to obtain target data, where the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; and performing a convolution operation on the target data and the operational convolution kernel to obtain a convolution operation result. The convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0040] Figure 2 A flow chart of a convolution operation method provided in some embodiments of the present disclosure. Figure 2 As shown, in some embodiments, the convolution operation method includes steps S10 to S30.

[0041] Step S10: Determine a computational convolution kernel, where the computational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], and the computational convolution kernel is expressed as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0.

[0042] Step S20: Based on the number of channels of the operational convolution kernel, adjusting the arrangement of the input data to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel;

[0043] Step S30: performing a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0044] For example, the convolution operation method can be used for the first-layer convolution operation of a convolutional neural network. Of course, the embodiments of the present disclosure are not limited to this. The convolution operation method can be used not only for convolutional neural networks, but also for convolution operations of other types of networks. It can be used not only for the first-layer convolution operation (first-layer convolution operation), but also for convolution operations of other layers. This can be determined according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0045] For example, in step S10, the initial convolution kernel is the convolution kernel required for the first-layer convolution operation, and the initial convolution kernel is expressed as [R, S, C, K]. Taking the first-layer convolution of the residual network ResNet50 as an example, the operation required for the first-layer convolution is expressed as: [1,224,224,3]×[7,7,3,64]=[1,112,112,64], then the initial convolution kernel [R, S, C, K] is [7,7,3,64], that is, in this example, R=7, S=7, C=3, K=64. The parameters of the initial convolution kernel are transformed to obtain the operational convolution kernel [1,1,(C×R×S),K]. In the above example, the operational convolution kernel can be obtained based on the initial convolution kernel, and the operational convolution kernel is [1,1,147,64]. The following is combined with Figure 3 Briefly explain the transformation principle of the convolution kernel.

[0046] Figure 3 This is a schematic diagram of the principle of convolution operation. Figure 3 As shown in , the input data size is [1,3,3,5], the convolution kernel size is [2,2,5,4], and the output data size is [1,2,2,4]. For example, for point M, the calculation method is as follows Figure 3 As shown, since the convolution kernel is 2×2 and has 5 channels, the M points are the result of multiplying the input data and the convolution kernel by 20 points, followed by the accumulation. By leveraging the characteristics of convolution, the convolution kernel can be transformed from R×S×C×K to 1×1×(C×R×S)×K, and the input data can be adjusted accordingly, so that the overall convolution result remains unchanged. This transformation can increase the number of channels. For the first layer of the convolutional neural network, the convolution kernel is adjusted from [7,7,3,64] to [1,1,147,64], and the number of channels is expanded from 3 to 3×7×7=147, thereby fully utilizing the computing power of the matrix operation unit (Matrix). Therefore, in step S10, the initial convolution kernel [R,S,C,K] can be transformed to obtain the operational convolution kernel [1,1,(C×R×S),K], thereby changing the convolution kernel arrangement.

[0047] For example, the arrangement of the convolution kernel can be changed in an offline manner. This is because the convolution kernel used by the neural network model in the deployment phase is fixed and will not change with changes in the input. Therefore, the convolution kernel can be pre-processed into the required arrangement. In an embodiment of the present disclosure, the convolution kernel [R, S, C, K] to be used can be set to [1, 1, (C × R × S), K] during the deployment phase of the neural network model, and used as the convolution kernel for subsequent use. For example, the code corresponding to the initial convolution kernel [R, S, C, K] can be modified in a high-level language (such as Python) during the model compilation phase, so that the operational convolution kernel [1, 1, (C × R × S), K] can be adjusted. Of course, the embodiments of the present disclosure are not limited to this. The initial convolution kernel [R, S, C, K] to be used can also be adjusted before each convolution operation to obtain the operational convolution kernel [1, 1, (C × R × S), K] actually used in this operation.

[0048] Back to Figure 2 In step S20, the arrangement of the input data is adjusted based on the number of channels of the convolution kernel to obtain the target data. For example, if the size and number of channels of the target data are different from those of the input data, the number of channels of the target data is equal to the number of channels of the convolution kernel. Taking the first convolution layer of the residual network ResNet50 as an example, the calculation to be performed is [1,224,224,3]×[7,7,3,64]=[1,112,112,64]. Since the convolution kernel is adjusted to [1,1,147,64], in order to ensure that the calculation result does not change, the arrangement of the input data needs to be adjusted so that the calculation becomes [1,112,112,147]×[1,1,147,64]=[1,112,112,64]. Therefore, the data obtained after adjusting the input data is [1,112,112,147]. For example, the data obtained after adjusting the arrangement of the input data is called the target data. The target data is the data that is ultimately convolved with the computational convolution kernel. For example, due to the adjustment of the arrangement, the size and number of channels of the target data are different from the size and number of channels of the input data. As can be seen from the above calculation formula, the number of channels of the target data is equal to the number of channels of the computational convolution kernel (for example, 147 in the above example), thereby facilitating the convolution operation between the two.

[0049] For example, the number of channels of the target data is greater than the number of channels of the input data, and the number of channels of the operational convolution kernel is greater than the number of channels of the initial convolution kernel, thereby increasing the number of channels and making full use of the computing power of the matrix operation unit. For example, in the above example, the number of channels of the input data and the number of channels of the initial convolution kernel are both 3, and the number of channels of the target data and the number of channels of the operational convolution kernel are both 147, thereby increasing the number of channels. For example, the conversion of the arrangement of the input data needs to be completed online, that is, in the neural network inference stage, the arrangement of the input data needs to be adjusted each time.

[0050] Figure 4 for Figure 2 For example, in some examples, such as Figure 4 As shown, step S20 may further include steps S21 to S23.

[0051] Step S21: storing the input data in a static memory in units of rows, wherein each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0;

[0052] Step S22: performing a padding operation on the input data stored in the static memory to obtain expanded data;

[0053] Step S23: adjusting the arrangement of the expanded data to change the size and number of channels of the expanded data to obtain target data.

[0054] For example, in step S21, the input data is first stored in a static memory, which is set in the hardware accelerator. The static memory is, for example, a static random access memory (SRAM). The input data is stored in the static memory in units of rows, that is, each row of the input data is stored in N corresponding storage rows in the static memory, where N is an integer greater than 0. For example, Figure 1 The data flow shown transfers the input data into static memory.

[0055] like Figure 5 As shown, step S21 may further include steps S211 to S212.

[0056] Step S211: storing the input data in a memory in a compact arrangement, wherein the input data includes multiple channels, and the compact arrangement means that multiple channels of the same data point are sequentially stored adjacently in the memory;

[0057] Step S212: The input data in the memory is transferred to the static memory of the hardware accelerator by direct storage access, and the first data point of each row of the input data is stored in the first column of different rows of the static memory, so that each row of the input data is stored in the corresponding N storage rows in the static memory.

[0058] For example, in step S211, the close arrangement method is, for example, a channel-aligned data arrangement method (Channel Align Tensor Layout). The input data includes multiple channels, and the close arrangement method means that multiple channels of the same data point are stored sequentially adjacent to each other in the memory. Figure 6 As shown in the figure, in some examples, the first-layer convolution input data is [1, 224, 224, 3], where the number 1 represents Batch Size = 1, one of the numbers 224 represents Height = 224, the other number 224 represents Width = 224, and the number 3 represents Channel = 3. That is, the input data is a 3-channel image of size 224 × 224. In order to reduce the storage space of the data on the host, a storage method that arranges the data tightly is adopted. For example, for the pixel points in the first row and first column of the input data, the values ​​of its three channels are stored in sequence in the memory space; then, the pixel points in the first row and second column are stored, and the values ​​of its three channels are stored in sequence in the memory space, and so on.

[0059] For example, the value of each channel of each pixel is represented by the FP16 data format, which occupies 2 bytes of address space. The first-layer input image occupies a total of 224×224×3×2Byte=301056Byte, or 294KB. The data format used here and the address space occupied are exemplary and do not constitute a limitation to the embodiments of the present disclosure.

[0060] For example, in step S212, the input data in the memory is transferred to the static memory of the hardware accelerator by using direct memory access (DMA). Figure 1 In the method shown, the input data of the first convolution layer is transferred from the host side to the DDR on the device side via PCIe. The storage method of the input data in DDR is consistent with that in host memory, thus implementing the Host2Device process of a one-dimensional tensor (1D Tensor). For example, the input data is stored continuously in DDR, occupying, for example, 294KB. Then, the input data needs to be transferred to static memory, that is, the 294KB data needs to be transferred from DDR to SRAM in the processing engine (PE, also known as hardware accelerator).

[0061] For example, when storing, the first data point of each row of input data is stored in the first column of different rows of the static memory, so that each row of input data is stored in corresponding N storage rows in the static memory.

[0062] like Figure 7 As shown, in some examples, the organization of SRAM can be abstractly considered as a table with M rows and N columns, with each table storing one data point. Since the input data size is 224×224, the input data is logically divided into 224 rows, with each row starting at the first column of a row of SRAM. Due to the limited number of SRAM columns, it is difficult to store an entire row of input data in one row of SRAM. Therefore, a row of input data is spread across multiple rows of SRAM, that is, different SRAM addresses. For the input data of [1,224,224,3], considering the data filled in the subsequent padding operation, each row has 229 points, and each point has 3 channels. For an SRAM row with 1024 bits of storage space per row, the number of SRAM rows required to store a row of data is: ceil(229*3 / 64) = 11, where ceil represents rounding up. That is, one row of data points of the input data is stored in 11 storage rows in the SRAM. In this example, N=11, and the entire input data occupies 224×11=2464 rows of SRAM.

[0063] like Figure 7 As shown in the figure, the left side shows that the input data is stored continuously in the DDR, without the concepts of H, W, and C. The right side shows that after being transferred to SRAM via DMA, the data is split into rows in the SRAM, with each row of data occupying a certain amount of SRAM space (for example, 11 storage rows). This enables data transfer from the DDR to the SRAM in the PE, completing the conversion from a one-dimensional tensor (1D Tensor) to a two-dimensional tensor (2D Tensor).

[0064] The DMA handling process is briefly described as follows.

[0065] Assuming the input data is stored in a contiguous DDR space starting with the source address (source_address), the first row of data (224 × 3 × 2 bytes = 1344 bytes) must be moved to a contiguous SRAM space starting with the destination address (destiny_address). Since each SRAM row is 128 bytes, these 1344 bytes must be stored in 11 SRAM rows (ceil(1344 / 128) = 1344 bytes), meaning the DMA must continuously send 11 × 128 bytes of data. After completing the first row of data transfer, the DMA must jump the read address from source_address to source_address + 1344 bytes, the DDR address at the beginning of the second row of input data. It then continuously transfers 11 × 128 bytes to the SRAM space starting with destiny_address + 11. This continues in this manner. After 224 transfers, all the input data has been transferred from the DDR to the SRAM within the processing engine, completing the conversion from a one-dimensional tensor to a two-dimensional tensor.

[0066] It should be noted that the amount of 11×128Byte data sent each time is greater than the actual amount of data in each row, that is, it also includes the data of the next row. However, since the starting address sent each time is accurate, even if the data is sent repeatedly, it will not affect the data itself, and these redundant data will not affect subsequent processing.

[0067] Back to Figure 4 For example, in step S22, a padding operation is performed on the input data stored in the static memory to obtain expanded data. Here, the expanded data refers to the data obtained after performing the padding operation. For example, in some examples, assuming that the convolution calculation that actually needs to be completed is: [1,224,224,3]×[7,7,3,64]=[1,112,112,64], it is necessary to perform padding operations (padding) on ​​the input data in all directions of up, down, left and right. When padding, 3 points need to be filled on the left and top of the input data (3 columns on the left and 3 rows on the top), and 2 points need to be filled on the right and bottom of the input data (2 columns on the right and 2 rows on the bottom). The size of the expanded data obtained after the padding operation is [1,229,229,3].

[0068] Figure 8 for Figure 4 Schematic flow chart of step S22 in FIG. Figure 8 As shown, step S22 may further include steps S221 to S223.

[0069] Step S221: In a static memory, filling a first preset value in a storage row before and after a storage location corresponding to the input data to obtain first intermediate data, wherein the first intermediate data includes the input data and the filled first preset value;

[0070] Step S222: transmitting the first intermediate data to the vector calculation unit, and using the shift instruction and fill instruction of the vector calculation unit to fill the two ends of each row corresponding to the first intermediate data with the second preset value to obtain second intermediate data, wherein the second intermediate data includes the first intermediate data and the filled second preset value;

[0071] Step S223: transferring the second intermediate data to a corresponding storage location in the static memory to obtain extended data, wherein the extended data has the same content as the second intermediate data.

[0072] For example, in step S221, the storage rows before and after the storage location corresponding to the input data in the static memory are filled with a first preset value, thereby obtaining first intermediate data, and the first intermediate data includes the input data and the first preset value of the filling. This step, for example, performs a padding operation on the upper and lower sides of the input data. For example, in some examples, near the target address of the SRAM, it is necessary to reserve the SRAM space required for the upper side padding, that is, several rows of data must be inserted before the first row of actual input data. The vector calculation unit (Vector) in the hardware accelerator is used to perform the padding operation (padding) on ​​the upper and lower sides. For example, the first preset value of the padding is usually 0, so it is necessary to write all 0 values ​​on several addresses before and after the storage space of the input data in the SRAM, thereby obtaining the first intermediate data. The first intermediate data is the data that has been padded on the upper and lower sides, and the first intermediate data has not yet been padded on the left and right sides.

[0073] For example, in step S222, the first intermediate data is transmitted to the vector computing unit, and the shift instructions (e.g., vshiftri instructions) and padding instructions (e.g., SI2V instructions) of the vector computing unit are used to pad the ends of each row corresponding to the first intermediate data with a second preset value to obtain second intermediate data. The second intermediate data includes the first intermediate data and the padded second preset value. This step, for example, performs padding operations on the left and right sides of the first intermediate data.

[0074] For example, in some examples, the data on the 2464 address spaces in the SRAM are grouped into 11 rows each, sent to the vector computing unit in sequence, and stored in the storage space vmem in the vector computing unit. The vector computing unit then uses the vshiftri instruction to shift the data right as a whole to leave space for the left padding, and then uses the SI2V instruction to write these locations to the corresponding second preset value (for example, usually set to 0). For right-side padding, after the data is shifted right as a whole, the corresponding second preset value is written after the last column of the first row of the input data. In the case where the amount of data to be filled is too large, additional vmem space needs to be added as needed. For example, a pipeline can be used to perform left and right padding operations on multiple groups of 11 rows of data to improve processing efficiency.

[0075] For example, in step S223, the second intermediate data is transferred to a corresponding storage location in the static memory to obtain extended data. The extended data has the same content as the second intermediate data. In other words, the second intermediate data in the vmem that has completed the fill operation is written back to the corresponding address space in the SRAM. The data stored in the SRAM that has completed the fill operation is referred to as the extended data.

[0076] It should be noted that, in the case where no padding operation is required for the input data, step S22 can be omitted. Furthermore, in the embodiments of the present disclosure, in the case where a padding operation is required, top and bottom padding can be performed first, followed by left and right padding, or left and right padding can be performed first, followed by top and bottom padding. The specific padding order is not limited. The instructions used when performing the padding operation are not limited to the vshiftri instruction and the SI2V instruction. Other applicable instructions can also be used, as long as the padding operation can be implemented. The embodiments of the present disclosure do not limit this.

[0077] Back to Figure 4 For example, in step S23, the arrangement of the expanded data is adjusted to change the size and number of channels of the expanded data, thereby obtaining the target data. That is, in order to match the computational convolution kernel and ensure that the computational result remains unchanged, it is necessary to adjust the arrangement of the expanded data and change its size and number of channels. For example, the number of channels of the target data obtained after adjustment is equal to the number of channels of the computational convolution kernel, and the target data is represented as [1, ht, wt, (C×R×S)], where ht and wt are both integers greater than 0.

[0078] Figure 9 for Figure 4 Schematic flow chart of step S23 in FIG. Figure 9 As shown, step S23 may further include steps S231 to S232.

[0079] Step S231: reading data in R*N storage rows in the static memory one by one and transmitting the data to the vector calculation unit;

[0080] Step S232: the vector calculation unit converts the data in the R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain target data.

[0081] For example, in step S231, data from R*N storage rows in the static memory is read each time and transmitted to the vector calculation unit, so that the vector calculation unit converts the data from the R*N storage rows received each time. For example, the starting address of each read is shifted by str*N storage rows according to a preset skip step str. The preset skip step is the skip step in the row and column directions of the sliding window required to convolve the input data with the initial convolution kernel. The total number of times data is read from the static memory is equal to ht.

[0082] For example, in some examples, still taking the first convolution layer of the residual network ResNet50 as an example, the initial convolution kernel [R, S, C, K] = [7, 7, 3, 64], so R = 7. The input data is [1, 224, 224, 3]. One row of input data is stored in N storage rows of SRAM. If the space of an SRAM row is 128 bytes, then N = 11. Therefore, each time the data in R*N = 77 storage rows in the static memory is read and transmitted to the vector calculation unit, the data stored in these 77 storage rows corresponds to one row of the 224×224 input data. For example, the sliding window required to perform a convolution operation on the input data [1, 224, 224, 3] with the initial convolution kernel [7, 7, 3, 64] has a skip step of 2 in the row and column directions, so the preset skip step str is 2. The starting address of each read is shifted by str*N (i.e., 2×11=22) memory rows according to the preset skip step str, so that the read data is consistent with the data contained in the sliding window during the original convolution operation. According to the formula [1,224,224,3]×[7,7,3,64]=[1,112,112,147]×[1,1,147,64]=[1,112,112,64], the convolution kernel is transformed to [1,1,147,64], and the target data [1,ht,wt,(C×R×S)] to be transformed is [1,112,112,147], so ht=112 and wt=112. For example, the total number of data reads from the static memory is equal to ht (e.g., 112), and each read data read corresponds to a row of the target data after conversion.

[0083] For example, in step S232, the vector calculation unit converts the data received in R*N storage rows each time into data in wt*ceil((C×R×S) / L) storage rows, and the converted data is the target data. That is, the data arrangement is adjusted to change the data size and number of channels. For example, L in the calculation formula represents the number of data points that can be stored in each storage row of the static memory, and ceil((C×R×S) / L) represents rounding up (C×R×S) / L. For example, in some examples, the vector calculation unit receives data in 7×11 storage rows each time; if the SRAM row space is 128 bytes, the number of data points that can be stored in each storage row L is 64; for the initial convolution kernel [7,7,3,64], R=7, S=7, C=3; for the target data [1,ht,wt,(C×R×S)]=[1,112,112,147], wt=112. Therefore, wt*ceil((C×R×S) / L)=112×3, that is, the vector calculation unit converts the data in 7×11 storage lines received each time into data in 112×3 storage lines.

[0084] Figure 10 for Figure 9 For example, in some examples, the above step S232 further includes steps S2321 to S2323.

[0085] Step S2321: Divide the data in the R*N storage rows into multiple groups of data according to a preset skipping step;

[0086] Step S2322: For each set of data, determine the initial position information parameters and target position information parameters of each row of data in the sliding window corresponding to the set of data;

[0087] Step S2323: The vector calculation unit stores each set of data in a corresponding position of the target memory in a converted arrangement according to the initial position information parameters and the target position information parameters to obtain target data.

[0088] For example, in step S2321, the data in the R*N storage rows is divided into multiple groups of data according to a preset skip step, each group of data corresponds to a sliding window in the row direction, and the number of groups of data is equal to wt. For example, in some examples, the data in the 7×11 storage rows is divided into 112 groups of data according to a preset skip step str=2, and wt=112. The data in the 7×11 storage rows corresponds to a row of data in the 224×224 input data, and the 112 groups of data correspond to 112 sliding windows of the 224 rows of data with a skip step of 2.

[0089] For example, in step S2322, for each set of data, the initial location information parameters and target location information parameters of each row of data in the sliding window corresponding to the set of data are determined. The initial location information parameters are used to determine the source address of the row of data in the sliding window, and the target location information parameters are used to determine the target address for transporting the data.

[0090] The following example illustrates the operation of steps S2321-S2322.

[0091] For example, after padding the input data [1,224,224,3], the entire input data size becomes [1,229,229,3], occupying 229×11 address spaces in the SRAM. Next, the shape of the input data needs to be converted to [1,112,112,147]. Essentially, for convolution operations, each sliding window (Feature Window) that the convolution kernel slides to needs to be completed. Figure 11 The transformation from [7×7×3] to [1,1,147] is shown.

[0092] Because the sliding window to which each convolution kernel slides corresponds to the 7 rows and 7 columns of the original data (input data or input image), the sliding windows swept by the convolution kernel as it slides from the upper left to the lower right overlap. For overlap in the column direction, to avoid repeated reading of data from the SRAM, each read of data from the SRAM in 7×11=77 address spaces is handed over to the vector calculation unit for processing. For overlap in the row direction, sliding the sliding window from left to right will repeatedly read the overlapping row data. Overall, the data in the SRAM is divided into 112 groups of data, corresponding to the 112 rows after conversion. Each group of data occupies 7×11 address spaces before conversion. After the vector calculation unit reads a group of data, it processes it and outputs a data size of 112×3 address spaces, where 112 corresponds to the width of the converted data and 3 corresponds to the space occupied by 147 channels (147 channels need to occupy 3 SRAM storage rows, that is, 3 SRAM address spaces).

[0093] After the vector computing unit receives a set of data, the data in these 7×11 SRAM storage rows (entries) is temporarily stored in the vector computing unit's internal vmem, with the data layout unchanged. The vector computing unit's instructions then convert this data into 112×3 vmem storage rows. The results are then written back to the SRAM.

[0094] The converted data width is 112 points in the row direction, and the number of channels for each point is 147, distributed on 3 vmem storage rows. For each point in the row direction, the original 7×7×3 sliding window needs to be converted to 1×1×147. To do this, it is necessary to find the 7 rows of data corresponding to each sliding window and then reorganize them into a new data arrangement. Figure 12 As shown in FIG, for the first sliding window, the data width of the sliding window is 7×3=21 channels, with a total of 7 rows of data (corresponding to storage in 7×11 storage rows). It is necessary to determine the storage addresses of these 7 rows of data and then convert their arrangement. Figure 12 On the right side of the image, the seven rows of data are rearranged into three rows. The original rows 0, 1, and 2 form the new row 1, the original rows 3 and 4 form the new row 2, and the original rows 5 and 6 form the new row 3. These three new rows store a total of 147 data points covered by the original 7×7×3 sliding window. Similarly, for the next sliding window in the row direction, the data arrangement is transformed in a similar manner until the row corresponding to the seven rows of data is completely transformed. Then, the next set of 7×11 stored rows of data is read.

[0095] In order to determine the initial position and target position of each row of data in the sliding window, it is necessary to define initial position information parameters and target position information parameters for each row of data in the sliding window.

[0096] The initial position information parameters include a first starting boundary coordinate, a first ending boundary coordinate, a first starting address, a first ending address, a first starting sequence number, and a first ending sequence number.

[0097] The first starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the row direction of the expanded data, and the first ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the row direction of the expanded data. The starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the expanded data. The data obtained after performing the filling operation is the expanded data, so these coordinates and parameters are defined for the expanded data. For other situations where the filling operation does not need to be performed, these coordinates and parameters can be defined directly for the input data. Figure 12 As shown, the starting boundary is, for example, the left boundary of the sliding window, and the first starting boundary coordinate is the relative coordinate of the left boundary of the sliding window in the row direction of the 229×3 expanded data; the ending boundary is, for example, the right boundary of the sliding window, and the first ending boundary coordinate is the relative coordinate of the right boundary of the sliding window in the row direction of the 229×3 expanded data.

[0098] The calculation formula for the first starting boundary coordinate is: src_row_start_index = i*str*ch. src_row_start_index represents the first starting boundary coordinate, i represents the sequence number of the corresponding sliding window data point in the target data size wt (for example, it represents the number of the sliding window in a row of 112 sliding windows, that is, the number of the output data width wt = 112), str represents the row jump of the sliding window (for example, 2), and ch represents the number of channels of the input data (for example, 3).

[0099] The calculation formula for the first end boundary coordinate is: src_row_end_index = src_row_start_index + (kernel_w*ch-1). src_row_end_index represents the first end boundary coordinate, kernel_w represents the width of the sliding window (for example, 7), and the size of the sliding window is equal to the size of the initial convolution kernel (for example, both are 7×7).

[0100] The first starting address represents the address of the first starting boundary coordinate in the memory (e.g., vmem) of the vector calculation unit, and the first ending address represents the address of the first ending boundary coordinate in the memory (e.g., vmem) of the vector calculation unit. The first starting sequence number represents the sequence number of the data point corresponding to the first starting boundary coordinate at the first starting address, and the first ending sequence number represents the sequence number of the data point corresponding to the first ending boundary coordinate at the first ending address. Since vmem is stored by row, a storage row in vmem can be located based on the first starting address or the first ending address, and the first starting sequence number or the first ending sequence number represents the number of the corresponding data in the storage row.

[0101] The first starting address is calculated as follows: src_row_start_address = src_row_start_index / vmem_lane + j * N. src_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, and j represents the row number of the corresponding data in the sliding window (e.g., a value from 1 to 7).

[0102] The calculation formula of the first end address is: src_row_end_address = src_row_end_index / vmem_lane + j*N. src_row_end_address represents the first end address.

[0103] The calculation formula of the first starting sequence number is: src_row_start_lane=src_row_start_index%vmem_lane. src_row_start_lane represents the first starting sequence number. For example, % represents a modulo operation.

[0104] The calculation formula of the first end sequence number is: src_row_end_lane=src_row_end_index%vmem_lane. src_row_end_lane represents the first end sequence number.

[0105] After determining the above parameters, we can determine the location of the source data in vmem required for converting a 7×7×3. In order to transfer this source data to the destination address in vmem, we also need to determine the corresponding destination address and related parameters, that is, we also need to determine the target location information parameters.

[0106] The target position information parameters include a second starting boundary coordinate, a second ending boundary coordinate, a second starting address, a second ending address, a second starting sequence number, and a second ending sequence number.

[0107] The second starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)], and the second ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)]. The starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the expanded data. For example, the target data is represented as [1,ht,wt,(C×R×S)]. In some examples, the target data is [1,112,112,147], and the data size corresponding to each sliding window needs to be converted from [7,7,3] to [1,1,147]. Figure 12 As shown, the starting boundary is, for example, the left boundary of the sliding window, and the second starting boundary coordinates are the relative coordinates of the left boundary of the sliding window in the data size of [1,1,147]; the ending boundary is, for example, the right boundary of the sliding window, and the second ending boundary coordinates are the relative coordinates of the right boundary of the sliding window in the data size of [1,1,147].

[0108] The calculation formula for the second starting boundary coordinate is: dst_row_start_index = j * kernel_w * ch. dst_row_start_index represents the second starting boundary coordinate, j represents the row number of the corresponding data in the sliding window (for example, a value from 1 to 7), kernel_w represents the width of the sliding window (for example, 7), the size of the sliding window is equal to the size of the initial convolution kernel (for example, 7×7), and ch represents the number of channels of the input data (for example, 3).

[0109] The calculation formula of the second end boundary coordinate is: dst_row_end_index=dst_row_start_index+(kernel_w*ch-1). dst_row_end_index represents the second end boundary coordinate.

[0110] The second starting address represents the address of the second starting boundary coordinate in the memory (e.g., vmem) of the vector calculation unit, and the second ending address represents the address of the second ending boundary coordinate in the memory (e.g., vmem) of the vector calculation unit. The second starting sequence number represents the sequence number of the data point corresponding to the second starting boundary coordinate at the second starting address, and the second ending sequence number represents the sequence number of the data point corresponding to the second ending boundary coordinate at the second ending address. Since vmem is stored by row, a storage row in vmem can be located based on the second starting address or the second ending address, and the second starting sequence number or the second ending sequence number represents the number of the corresponding data in the storage row.

[0111] The calculation formula of the second starting address is: dst_row_start_address=dst_row_start_index / vmem_lane. dst_row_start_address represents the second starting address, and vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit.

[0112] The calculation formula of the second end address is: dst_row_end_address=dst_row_end_index / vmem_lane. dst_row_end_address represents the second end address.

[0113] The calculation formula of the second starting sequence number is: dst_row_start_lane=dst_row_start_index%vmem_lane. dst_row_start_lane represents the second starting sequence number.

[0114] The calculation formula of the second end sequence number is: dst_row_end_lane=dst_row_end_index%vmem_lane. dst_row_end_lane represents the second end sequence number.

[0115] After determining the initial location information parameters and the target location information parameters, the source address and target address required for data transfer can be determined, and then the source data is moved to the target address according to these parameters.

[0116] For example, in step S2323, after determining the initial position information parameters and the target position information parameters, the vector calculation unit stores each set of data in the corresponding position of the target memory in the converted arrangement, and the target address indicated by the target position information parameter is the address in the target memory, thereby obtaining the target data. For example, the target memory is stored in units of rows, and the data transmitted to the target memory and stored on the target memory is the target data. For example, the target memory can be the static memory described above (in this case, the data before conversion and the data after conversion are stored in different addresses of the static memory), or it can be other storage devices different from the static memory described above, and the embodiments of the present disclosure are not limited to this.

[0117] For example, step S2323 may further include: based on the initial position information parameters and the target position information parameters, the vector calculation unit uses a circular shift instruction and according to the preset enable signal in the predicate register, each group of data is spliced ​​in the converted arrangement and stored in the corresponding position of the target memory to obtain the target data. For example, in some examples, the vshiftri instruction in the instruction set architecture (Vector ISA) of the vector calculation unit can be used to circularly shift the data of the source address to the right by several positions and then write it to the destination address according to the write enable signal in the predicate register (Vector Predicate Register, VPR, also known as the VP register). The aforementioned preset enable signal is, for example, a write enable signal. In the process of converting the data corresponding to a 7×7×3 sliding window into 1×1×147 data, it is necessary to determine the VP register to be used based on the second starting sequence number dst_row_start_lane and the second ending sequence number dst_row_end_lane. Regarding the use of the vshiftri instruction and the VP register, reference can be made to the conventional design and will not be described in detail here.

[0118] In the above manner, the conversion from a two-dimensional tensor (2D Tensor) to a three-dimensional tensor (3DTensor) is completed using the vector computing unit.

[0119] After processing in each step, the input data [1,224,224,3] is converted to the target data [1,112,112,147]. The operational convolution kernel determined according to the initial convolution kernel [7,7,3,64] is [1,1,147,64], which increases the number of channels from 3 to 147.

[0120] Back to Figure 2In step S30, a convolution operation is performed on the target data and the operational convolution kernel to obtain a convolution result. The convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel. For example, step S30 may further include: performing a convolution operation on the target data and the operational convolution kernel using a matrix operation unit (Matrix).

[0121] For example, in some examples, taking the first convolution layer of the residual network ResNet50 as an example, the operation of the input data and the initial convolution kernel that needs to be implemented is: [1,224,224,3]×[7,7,3,64]=[1,112,112,64]. Since there are only 3 channels, the computing power of the matrix operation unit is not fully utilized. By adopting the convolution operation method provided by the embodiment of the present disclosure, the operational convolution kernel obtained based on the initial convolution kernel [7,7,3,64] is [1,1,147,64]. After adjusting the arrangement of the input data [1,224,224,3], the target data obtained is [1,112,112,147]. Therefore, the actual operation of the target data and the operational convolution kernel is: [1,112,112,147]×[1,1,147,64]=[1,112,112,64]. The convolution operation result is consistent with the original convolution operation result. Since the number of channels is increased to 147, the computing power of the matrix operation unit can be fully utilized, the utilization rate of the matrix operation unit can be improved, the time of the convolution operation is shortened, and the operation efficiency is improved. Furthermore, since the input data does not need to be rearranged by the host CPU and the number of channels does not need to be expanded on the host, the occupied data space will not increase significantly, and the amount of data transmitted from the host to the device will not increase. Therefore, the PCIe transmission time of Host2Device will not be increased, thereby saving data transmission time.

[0122] The convolution operation method provided by the embodiment of the present disclosure is helpful to achieve the purpose of hardware acceleration, and can realize the acceleration of the first-layer convolution calculation of the convolutional neural network (CNN), and has the characteristics of small storage space, short transmission time, high hardware module utilization, and short calculation time. For example, the time required to perform the first-layer convolution of the residual network ResNet50 using the conventional convolution operation method is 614656 cycles, while the theoretical time required to perform the first-layer convolution of the residual network ResNet50 using the convolution operation method provided by the embodiment of the present disclosure is 37632 cycles, which is reduced to 6.1% of the previous time. This greatly shortens the first-layer convolution calculation time of the convolutional neural network (CNN).

[0123] It should be noted that, in the embodiments of the present disclosure, the convolution operation method provided in the above-mentioned embodiments of the present disclosure may include more or fewer operations, and these operations may be performed sequentially or in parallel. Although the process of the convolution operation method described above includes multiple operations that appear in a specific order, it should be clearly understood that the order of the multiple operations is not limited. The convolution operation method described above can be performed once or multiple times according to predetermined conditions.

[0124] It should be noted that the above description uses the first convolution layer of the residual network ResNet50 as an example, but this does not constitute a limitation on the embodiments of the present disclosure. The convolution operation method provided in the embodiments of the present disclosure can be applied to any applicable convolution operation. The size and number of channels of various types of data, and the size and number of channels of various types of convolution kernels can be determined according to actual needs and are not limited to the specific values ​​described above.

[0125] At least one embodiment of the present disclosure further provides a convolution operation device that can improve the utilization rate of a matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the convolution operation time, improve computing efficiency, and save data transmission time.

[0126] Figure 13 This is a schematic block diagram of a convolution operation device provided in some embodiments of the present disclosure. Figure 13 As shown, in some embodiments, the convolution operation device 100 includes a determination unit 110, an adjustment unit 120, and a calculation unit 130.

[0127] The determining unit 110 is configured to determine the operational convolution kernel. For example, the operational convolution kernel is obtained based on the initial convolution kernel, the initial convolution kernel is represented as [R, S, C, K], and the operational convolution kernel is represented as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0. For example, the determining unit 110 may perform Figure 2 Step S10 of the convolution operation method shown.

[0128] The adjustment unit 120 is configured to adjust the arrangement of the input data based on the number of channels of the operational convolution kernel to obtain the target data. For example, the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel. For example, the adjustment unit 120 may perform Figure 2 Step S20 of the convolution operation method shown.

[0129] The computing unit 130 is configured to perform a convolution operation based on the target data and the computational convolution kernel to obtain a convolution operation result. For example, the convolution operation result of the target data and the computational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel. For example, the computing unit 130 may perform Figure 2Step S30 of the convolution operation method shown.

[0130] For example, the determining unit 110, the adjusting unit 120, and the calculating unit 130 may be hardware, software, firmware, or any feasible combination thereof. For example, the determining unit 110, the adjusting unit 120, and the calculating unit 130 may be dedicated or general-purpose circuits, chips, or devices, or may be a combination of a processor and memory. The embodiments of the present disclosure do not limit the specific implementation of the determining unit 110, the adjusting unit 120, and the calculating unit 130.

[0131] It should be noted that in the embodiment of the present disclosure, the various units of the convolution operation device 100 correspond to the various steps of the aforementioned convolution operation method. For the specific functions of the convolution operation device 100, please refer to the relevant description of the convolution operation method above, which will not be repeated here. Figure 13 The components and structures of the convolution operation device 100 shown are merely exemplary and non-limiting. The convolution operation device 100 may further include other components and structures as needed.

[0132] At least one embodiment of the present disclosure further provides an electronic device that can improve the utilization rate of a matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the convolution operation time, improve computing efficiency, and save data transmission time.

[0133] Figure 14 This is a schematic block diagram of an electronic device provided in some embodiments of the present disclosure. Figure 14 As shown, the electronic device 200 includes a convolution operation device 210. The convolution operation device 210 can be a convolution operation device provided by any embodiment of the present disclosure, for example, the aforementioned convolution operation device 100. The electronic device 200 can be any device with computing functions, such as a server, terminal device, personal computer, etc., and the embodiments of the present disclosure are not limited to this.

[0134] Figure 15 This is a schematic block diagram of another electronic device provided in some embodiments of the present disclosure. Figure 15 As shown, the electronic device 300 includes a processor 310 and a memory 320, which can be used to implement a client or server. The memory 320 is used to non-transiently store computer-executable instructions (e.g., at least one (or more) computer program modules). The processor 310 is used to run the computer-executable instructions. When the computer-executable instructions are run by the processor 310, one or more steps in the convolution operation method described above can be executed, thereby implementing the convolution operation method described above. The memory 320 and the processor 310 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0135] For example, the processor 310 may be a central processing unit (CPU), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture. The processor 310 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 300 to perform desired functions.

[0136] For example, the memory 320 may include any combination of at least one (e.g., one or more) computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc. At least one (e.g., one or more) computer program modules may be stored on the computer-readable storage medium, and the processor 310 may execute the at least one (e.g., one or more) computer program modules to implement various functions of the electronic device 300. The computer-readable storage medium may also store various applications and various data, as well as various data used and / or generated by the applications.

[0137] It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 300 can be referred to the above description of the convolution operation method, which will not be repeated here.

[0138] Figure 16 A schematic block diagram of another electronic device provided for some embodiments of the present disclosure. The electronic device 400 is, for example, suitable for implementing the convolution operation method provided in the embodiments of the present disclosure. The electronic device 400 may be a terminal device, etc., and may be used to implement a client or a server. The electronic device 400 may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that, Figure 16 The electronic device 400 shown is merely an example and does not limit the functions and scope of use of the embodiments of the present disclosure.

[0139] like Figure 16As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 410, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 420 or a program loaded from a storage device 480 into a random access memory (RAM) 430. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 430. The processing device 410, the ROM 420, and the RAM 430 are connected to each other via a bus 440. An input / output (I / O) interface 450 is also connected to the bus 440.

[0140] Typically, the following devices may be connected to the I / O interface 450: an input device 460 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 470 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 480 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 490. The communication device 490 may allow the electronic device 400 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 16 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 400 may instead implement or possess more or fewer devices.

[0141] For example, according to an embodiment of the present disclosure, the above-mentioned convolution operation method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for performing the above-mentioned convolution operation method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 490, or installed from the storage device 480, or installed from the ROM 420. When the computer program is executed by the processing device 410, the functions defined in the convolution operation method provided in the embodiment of the present disclosure can be implemented.

[0142] At least one embodiment of the present disclosure further provides a storage medium that can improve the utilization rate of a matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the convolution operation time, improve computing efficiency, and save data transmission time.

[0143] Figure 17 A schematic diagram of a storage medium provided in some embodiments of the present disclosure. Figure 17As shown, the storage medium 500 may be a non-transitory computer-readable storage medium storing non-transitory computer-readable instructions 510. When the non-transitory computer-readable instructions 510 are executed by a processor, the convolution operation method described in the embodiment of the present disclosure may be implemented. For example, when the non-transitory computer-readable instructions 510 are executed by a processor, one or more steps in the convolution operation method described above may be performed.

[0144] For example, the storage medium 500 may be applied to the above-mentioned electronic device. For example, the storage medium 500 may include the memory 320 in the electronic device 300 .

[0145] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0146] For example, the description of the storage medium 500 can refer to the description of the memory in the embodiment of the electronic device, and the repeated parts are not repeated. The specific functions and technical effects of the storage medium 500 can refer to the description of the convolution operation method above, and are not repeated here.

[0147] In the above, combined with Figures 1 to 17 The convolution operation method, convolution operation device, electronic device, and storage medium provided by the embodiments of the present disclosure are described. The convolution operation method provided by the embodiments of the present disclosure can be used for the first-layer convolution operation of a convolutional neural network. By adjusting the arrangement of data to increase the number of channels, the target data with more channels can be convolved with the convolution kernel with more channels. This can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the convolution operation time, improve computing efficiency, and save data transmission time.

[0148] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0149] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0150] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0151] The computer program code for performing the operations of the present disclosure can be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0154] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0155] According to one or more embodiments of the present disclosure, a convolution operation method includes: determining an operational convolution kernel, wherein the operational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], and the operational convolution kernel is expressed as [1, 1, (C×R×S), K], where R, S, C, and K are all integers greater than 0; adjusting the arrangement of input data based on the number of channels of the operational convolution kernel to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; performing a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0156] According to one or more embodiments of the present disclosure, the number of channels of the target data is greater than the number of channels of the input data, and the number of channels of the operational convolution kernel is greater than the number of channels of the initial convolution kernel.

[0157] According to one or more embodiments of the present disclosure, based on the number of channels of the operational convolution kernel, the arrangement of the input data is adjusted to obtain target data, including: storing the input data in a static memory in rows, wherein each row of the input data is stored in N corresponding storage rows in the static memory, where N is an integer greater than 0; performing a padding operation on the input data stored in the static memory to obtain expanded data; and adjusting the arrangement of the expanded data to change the size and number of channels of the expanded data to obtain the target data.

[0158] According to one or more embodiments of the present disclosure, the input data is stored in the static memory in units of rows, including: storing the input data in the memory in a tightly packed manner, wherein the input data includes multiple channels, and the tightly packed manner means that multiple channels of the same data point are stored sequentially and adjacently in the memory; transferring the input data in the memory to the static memory of the hardware accelerator in a direct storage access manner, and storing the first data point of each row of the input data in the first column of different rows of the static memory, so that each row of the input data is stored in corresponding N storage rows in the static memory.

[0159] According to one or more embodiments of the present disclosure, the filling operation is performed on the input data stored in the static memory to obtain the extended data, including: in the static memory, filling the storage rows before and after the storage position corresponding to the input data with a first preset value to obtain first intermediate data, wherein the first intermediate data includes the input data and the filled first preset value; transferring the first intermediate data to a vector calculation unit, and using the shift instruction and fill instruction of the vector calculation unit to fill the two ends of each row corresponding to the first intermediate data with a second preset value to obtain second intermediate data, wherein the second intermediate data includes the first intermediate data and the filled second preset value; transferring the second intermediate data to the corresponding storage position in the static memory to obtain the extended data, wherein the content of the extended data is the same as that of the second intermediate data.

[0160] According to one or more embodiments of the present disclosure, the target data is represented as [1, ht, wt, (C×R×S)], where ht and wt are both integers greater than 0; adjusting the arrangement of the extended data to change the size and number of channels of the extended data to obtain the target data, including: successively reading the data in R*N storage rows in the static memory and transmitting them to the vector calculation unit, wherein the starting address of each read is moved str*N storage rows according to a preset jump step str, and the preset jump step is to separate the input data from the initial convolution kernel. The sliding window jumps in the row and column directions required for the convolution operation, and the total number of times data is read from the static memory is equal to ht; the vector calculation unit converts the data in the R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain the target data, where L represents the number of data points that can be stored in each storage row in the static memory, ceil((C×R×S) / L) represents rounding up (C×R×S) / L, and the converted data is the target data.

[0161] According to one or more embodiments of the present disclosure, the vector calculation unit converts the data in the R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain the target data, including: dividing the data in the R*N storage rows into multiple groups of data according to the preset jump step, wherein each group of data corresponds to a sliding window in a row direction, and the number of groups of the multiple groups of data is equal to wt; for each group of data, determining the initial position information parameters and target position information parameters of each row of data of the sliding window corresponding to the group of data; the vector calculation unit stores each group of data in the corresponding position of the target memory in a converted arrangement according to the initial position information parameters and the target position information parameters to obtain the target data, wherein the target memory is stored in units of rows, and the data transmitted to the target memory and stored on the target memory is the target data.

[0162] According to one or more embodiments of the present disclosure, the initial position information parameters include a first starting boundary coordinate, a first ending boundary coordinate, a first starting address, a first ending address, a first starting sequence number, and a first ending sequence number; the first starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the row direction of the expanded data, and the first ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the row direction of the expanded data, and the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the expanded data; the first starting address represents the address of the first starting boundary coordinate in the memory of the vector calculation unit, and the first ending address represents the address of the first ending boundary coordinate in the memory of the vector calculation unit; the first starting sequence number represents the sequence number of the data point corresponding to the first starting boundary coordinate at the first starting address, and the first ending sequence number represents the sequence number of the data point corresponding to the first ending boundary coordinate at the first ending address.

[0163] According to one or more embodiments of the present disclosure, the calculation formula of the first starting boundary coordinate is: src_row_start_index = i*str*ch, wherein src_row_start_index represents the first starting boundary coordinate, i represents the serial number of the data point corresponding to the corresponding sliding window in the size wt of the target data, str represents the jump step of the sliding window in the row direction, and ch represents the number of channels of the input data; the calculation formula of the first ending boundary coordinate is: src_row_end_index = src_row_start_index + (kernel_w*ch-1), wherein src_row_end_index represents the first ending boundary coordinate, kernel_w represents the width of the sliding window, and the size of the sliding window is equal to the size of the initial convolution kernel; the calculation formula of the first starting address is: src_row_start_address = src_row_start_index / vmem_lane+j*N, wherein sr c_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, and j represents the row number of the corresponding data in the sliding window; the calculation formula of the first end address is: src_row_end_address = src_row_end_index / vmem_lane + j*N, where src_row_end_address represents the first end address; the calculation formula of the first starting sequence number is: src_row_start_lane = src_row_start_index%vmem_lane, where src_row_start_lane represents the first starting sequence number, and % represents the modulo operation; the calculation formula of the first end sequence number is: src_row_end_lane = src_row_end_index%vmem_lane, where src_row_end_lane represents the first end sequence number.

[0164] According to one or more embodiments of the present disclosure, the target position information parameters include a second starting boundary coordinate, a second ending boundary coordinate, a second starting address, a second ending address, a second starting sequence number, and a second ending sequence number; the second starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)], and the second ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)], and the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the extended data; the second starting address represents the address of the second starting boundary coordinate in the memory of the vector calculation unit, and the second ending address represents the address of the second ending boundary coordinate in the memory of the vector calculation unit; the second starting sequence number represents the sequence number of the data point corresponding to the second starting boundary coordinate at the second starting address, and the second ending sequence number represents the sequence number of the data point corresponding to the second ending boundary coordinate at the second ending address.

[0165] According to one or more embodiments of the present disclosure, the calculation formula of the second starting boundary coordinate is: dst_row_start_index = j*kernel_w*ch, wherein dst_row_start_index represents the second starting boundary coordinate, j represents the row number of the corresponding data in the sliding window, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels of the input data; the calculation formula of the second ending boundary coordinate is: dst_row_end_index = dst_row_start_index + (kernel_w*ch-1), wherein dst_row_end_index represents the second ending boundary coordinate; the calculation formula of the second starting address is: dst_row_start_address = dst_row_start_index / vmem_lane, wherein dst_ro w_start_address represents the second starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit; the calculation formula of the second end address is: dst_row_end_address = dst_row_end_index / vmem_lane, wherein dst_row_end_address represents the second end address; the calculation formula of the second starting sequence number is: dst_row_start_lane = dst_row_start_index% vmem_lane, wherein dst_row_start_lane represents the second starting sequence number, and % represents a modulo operation; the calculation formula of the second end sequence number is: dst_row_end_lane = dst_row_end_index% vmem_lane, wherein dst_row_end_lane represents the second end sequence number.

[0166] According to one or more embodiments of the present disclosure, the vector calculation unit stores each group of data in a converted arrangement to a corresponding position of the target memory according to the initial position information parameters and the target position information parameters to obtain the target data, including: according to the initial position information parameters and the target position information parameters, the vector calculation unit uses a circular shift instruction and according to a preset enable signal in a predicate register to splice each group of data in a converted arrangement and store them in a corresponding position of the target memory to obtain the target data.

[0167] According to one or more embodiments of the present disclosure, performing a convolution operation based on the target data and the operational convolution kernel includes: performing a convolution operation on the target data and the operational convolution kernel using a matrix operation unit.

[0168] According to one or more embodiments of the present disclosure, the convolution operation method is used for the first-layer convolution operation of a convolutional neural network.

[0169] According to one or more embodiments of the present disclosure, a convolution operation device includes: a determination unit, configured to determine an operational convolution kernel, wherein the operational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], the operational convolution kernel is expressed as [1, 1, (C×R×S), K], and R, S, C, and K are all integers greater than 0; an adjustment unit, configured to adjust the arrangement of input data based on the number of channels of the operational convolution kernel to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; a calculation unit, configured to perform a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel.

[0170] According to one or more embodiments of the present disclosure, the number of channels of the target data is greater than the number of channels of the input data, and the number of channels of the operational convolution kernel is greater than the number of channels of the initial convolution kernel.

[0171] According to one or more embodiments of the present disclosure, the adjustment unit includes a first adjustment subunit, a second adjustment subunit, and a third adjustment subunit. The first adjustment subunit is configured to store the input data in a static memory in rows, wherein each row of the input data is stored in N corresponding storage rows in the static memory, where N is an integer greater than 0. The second adjustment subunit is configured to perform a padding operation on the input data stored in the static memory to obtain expanded data. The third adjustment subunit is configured to adjust the arrangement of the expanded data to change the size and number of channels of the expanded data to obtain the target data.

[0172] According to one or more embodiments of the present disclosure, the first adjustment subunit includes a first storage unit and a second storage unit. The first storage unit is configured to store the input data in a memory in a tightly packed manner, wherein the input data includes multiple channels, and the tightly packed manner means that multiple channels of the same data point are stored sequentially and adjacently in the memory. The second storage unit is configured to transfer the input data in the memory to the static memory of the hardware accelerator in a direct storage access manner, and store the first data point of each row of the input data in the first column of different rows of the static memory, so that each row of the input data is stored in the corresponding N storage rows in the static memory.

[0173] According to one or more embodiments of the present disclosure, the second adjustment subunit includes a first filling unit, a second filling unit, and a third filling unit. The first filling unit is configured to fill the storage rows before and after the storage position corresponding to the input data in the static memory with a first preset value to obtain first intermediate data, wherein the first intermediate data includes the input data and the filled first preset value. The second filling unit is configured to transfer the first intermediate data to the vector calculation unit, and use the shift instruction and fill instruction of the vector calculation unit to fill the two ends of each row corresponding to the first intermediate data with a second preset value to obtain second intermediate data, wherein the second intermediate data includes the first intermediate data and the filled second preset value. The third filling unit is configured to transfer the second intermediate data to the corresponding storage position in the static memory to obtain the extended data, wherein the extended data has the same content as the second intermediate data.

[0174] According to one or more embodiments of the present disclosure, the target data is represented as [1, ht, wt, (C×R×S)], where ht and wt are both integers greater than 0. The third adjustment subunit includes a first changing unit and a second changing unit. The first changing unit is configured to sequentially read data from R*N storage rows in the static memory and transmit the data to the vector calculation unit, wherein the starting address of each read is shifted by str*N storage rows according to a preset jump step str, and the preset jump step is the jump step of the sliding window in the row direction and the column direction required for performing a convolution operation on the input data with the initial convolution kernel, and the total number of times data is read from the static memory is equal to ht. The second changing unit is configured to use the vector calculation unit to convert the data in the R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain the target data, wherein L represents the number of data points that can be stored in each storage row in the static memory, ceil((C×R×S) / L) represents rounding up (C×R×S) / L, and the converted data is the target data.

[0175] According to one or more embodiments of the present disclosure, the second changing unit includes a grouping unit, a parameter determination unit, and a vector calculation unit. The grouping unit is configured to divide the data in the R*N storage rows into multiple groups of data according to the preset jump step, wherein each group of data corresponds to a sliding window in a row direction, and the number of groups of the multiple groups of data is equal to wt. The parameter determination unit is configured to determine, for each group of data, the initial position information parameters and the target position information parameters of each row of data in the sliding window corresponding to the group of data. The vector calculation unit is configured to store each group of data in a converted arrangement to a corresponding position of the target memory according to the initial position information parameters and the target position information parameters to obtain the target data, wherein the target memory is stored in units of rows, and the data transmitted to the target memory and stored on the target memory is the target data.

[0176] According to one or more embodiments of the present disclosure, the initial position information parameters include a first starting boundary coordinate, a first ending boundary coordinate, a first starting address, a first ending address, a first starting sequence number, and a first ending sequence number; the first starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the row direction of the expanded data, and the first ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the row direction of the expanded data, and the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the expanded data; the first starting address represents the address of the first starting boundary coordinate in the memory of the vector calculation unit, and the first ending address represents the address of the first ending boundary coordinate in the memory of the vector calculation unit; the first starting sequence number represents the sequence number of the data point corresponding to the first starting boundary coordinate at the first starting address, and the first ending sequence number represents the sequence number of the data point corresponding to the first ending boundary coordinate at the first ending address.

[0177] According to one or more embodiments of the present disclosure, the calculation formula of the first starting boundary coordinate is: src_row_start_index = i*str*ch, wherein src_row_start_index represents the first starting boundary coordinate, i represents the serial number of the data point corresponding to the corresponding sliding window in the size wt of the target data, str represents the jump step of the sliding window in the row direction, and ch represents the number of channels of the input data; the calculation formula of the first ending boundary coordinate is: src_row_end_index = src_row_start_index + (kernel_w*ch-1), wherein src_row_end_index represents the first ending boundary coordinate, kernel_w represents the width of the sliding window, and the size of the sliding window is equal to the size of the initial convolution kernel; the calculation formula of the first starting address is: src_row_start_address = src_row_start_index / vmem_lane+j*N, wherein sr c_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, and j represents the row number of the corresponding data in the sliding window; the calculation formula of the first end address is: src_row_end_address = src_row_end_index / vmem_lane + j*N, where src_row_end_address represents the first end address; the calculation formula of the first starting sequence number is: src_row_start_lane = src_row_start_index%vmem_lane, where src_row_start_lane represents the first starting sequence number, and % represents the modulo operation; the calculation formula of the first end sequence number is: src_row_end_lane = src_row_end_index%vmem_lane, where src_row_end_lane represents the first end sequence number.

[0178] According to one or more embodiments of the present disclosure, the target position information parameters include a second starting boundary coordinate, a second ending boundary coordinate, a second starting address, a second ending address, a second starting sequence number, and a second ending sequence number; the second starting boundary coordinate represents the relative coordinate of the starting boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)], and the second ending boundary coordinate represents the relative coordinate of the ending boundary of the corresponding sliding window in the data size of [1,1,(C×R×S)], and the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the extended data; the second starting address represents the address of the second starting boundary coordinate in the memory of the vector calculation unit, and the second ending address represents the address of the second ending boundary coordinate in the memory of the vector calculation unit; the second starting sequence number represents the sequence number of the data point corresponding to the second starting boundary coordinate at the second starting address, and the second ending sequence number represents the sequence number of the data point corresponding to the second ending boundary coordinate at the second ending address.

[0179] According to one or more embodiments of the present disclosure, the calculation formula of the second starting boundary coordinate is: dst_row_start_index = j*kernel_w*ch, wherein dst_row_start_index represents the second starting boundary coordinate, j represents the row number of the corresponding data in the sliding window, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels of the input data; the calculation formula of the second ending boundary coordinate is: dst_row_end_index = dst_row_start_index + (kernel_w*ch-1), wherein dst_row_end_index represents the second ending boundary coordinate; the calculation formula of the second starting address is: dst_row_start_address = dst_row_start_index / vmem_lane, wherein dst_ro w_start_address represents the second starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit; the calculation formula of the second end address is: dst_row_end_address = dst_row_end_index / vmem_lane, wherein dst_row_end_address represents the second end address; the calculation formula of the second starting sequence number is: dst_row_start_lane = dst_row_start_index% vmem_lane, wherein dst_row_start_lane represents the second starting sequence number, and % represents a modulo operation; the calculation formula of the second end sequence number is: dst_row_end_lane = dst_row_end_index% vmem_lane, wherein dst_row_end_lane represents the second end sequence number.

[0180] According to one or more embodiments of the present disclosure, the vector calculation unit is further configured to use a circular shift instruction based on the initial position information parameters and the target position information parameters and to splice each group of data in a converted arrangement according to a preset enable signal in the predicate register and store them in a corresponding position of the target memory to obtain the target data.

[0181] According to one or more embodiments of the present disclosure, the computing unit includes a computing subunit, and the computing subunit is configured to perform a convolution operation on the target data and the operational convolution kernel using a matrix operation unit.

[0182] According to one or more embodiments of the present disclosure, the convolution operation device is used for the first-layer convolution operation of a convolutional neural network.

[0183] According to one or more embodiments of the present disclosure, an electronic device includes the convolution operation device provided by any embodiment of the present disclosure.

[0184] According to one or more embodiments of the present disclosure, an electronic device includes: a processor; a memory, including at least one computer program module; wherein the at least one computer program module is stored in the memory and is configured to be executed by the processor, and the at least one computer program module is used to implement the convolution operation method provided by any embodiment of the present disclosure.

[0185] According to one or more embodiments of the present disclosure, a storage medium stores non-transitory computer-readable instructions, which, when executed by a computer, implement the convolution operation method provided by any embodiment of the present disclosure.

[0186] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0187] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0188] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0189] Regarding this disclosure, the following points need to be explained:

[0190] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0191] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0192] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A convolution operation method, comprising: Determine a computational convolution kernel, where the computational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], and the computational convolution kernel is expressed as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0; Adjusting the arrangement of the input data based on the number of channels of the operational convolution kernel to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; A convolution operation is performed based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel, The step of adjusting the arrangement of the input data based on the number of channels of the operational convolution kernel to obtain the target data includes: Dividing the input data into lines and storing them in a static memory; Performing a padding operation on the input data stored in the static memory to obtain expanded data; Adjusting the arrangement of the extended data to change the size and number of channels of the extended data to obtain the target data, In which, each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0, and N is obtained by multiplying the number of data points in each row of the input data after the padding operation and the number of channels, divided by the number of data points that can be stored in each storage row in the static memory, and rounding up the resulting ratio.

2. The method according to claim 1, wherein The number of channels of the target data is greater than the number of channels of the input data, and the number of channels of the operational convolution kernel is greater than the number of channels of the initial convolution kernel.

3. The method according to claim 1, wherein Dividing the input data into lines and storing the lines in the static memory includes: Storing the input data in a memory in a dense arrangement, wherein the input data includes multiple channels, and the dense arrangement means that multiple channels of the same data point are stored sequentially adjacent to each other in the memory; The input data in the memory is transferred to the static memory of the hardware accelerator by direct storage access, and the first data point of each row of the input data is stored in the first column of different rows of the static memory, so that each row of the input data is stored in the corresponding N storage rows in the static memory.

4. The method according to claim 1, wherein Performing the padding operation on the input data stored in the static memory to obtain the expanded data includes: In the static memory, filling the storage rows before and after the storage location corresponding to the input data with a first preset value to obtain first intermediate data, wherein the first intermediate data includes the input data and the filled first preset value; transmitting the first intermediate data to a vector computing unit, and using shift instructions and fill instructions of the vector computing unit to fill both ends of each row corresponding to the first intermediate data with a second preset value to obtain second intermediate data, wherein the second intermediate data includes the first intermediate data and the filled second preset value; The second intermediate data is transferred to a corresponding storage location in the static memory to obtain the extended data, wherein the extended data has the same content as the second intermediate data.

5. The method according to claim 4, wherein The target data is expressed as [1, ht, wt, (C×R×S)], where ht and wt are both integers greater than 0; Adjusting the arrangement of the extended data to change the size and number of channels of the extended data to obtain the target data includes: Reading data from R*N storage rows in the static memory one by one and transmitting the data to the vector calculation unit, wherein the starting address of each read is shifted by str*N storage rows according to a preset skip step str, where the preset skip step is the skip step in the row and column directions of the sliding window required for convolving the input data with the initial convolution kernel, and the total number of times data is read from the static memory is equal to ht; The vector calculation unit converts the data in the R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain the target data, where L represents the number of data points that can be stored in each storage row in the static memory. ceil((C×R×S) / L) represents rounding up (C×R×S) / L, and the converted data is the target data.

6. The method according to claim 5, wherein: The vector calculation unit converts the data in R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows to obtain the target data, including: Divide the data in the R*N storage rows into multiple groups of data according to the preset skipping step, wherein each group of data corresponds to a sliding window in the row direction, and the number of the multiple groups of data is equal to wt; For each set of data, determining the initial position information parameters and target position information parameters of each row of data in the sliding window corresponding to the set of data; The vector calculation unit stores each group of data in a corresponding position of the target memory in a converted arrangement according to the initial position information parameters and the target position information parameters to obtain the target data, wherein the target memory is stored in units of rows, and the data transmitted to the target memory and stored in the target memory is the target data.

7. The method according to claim 6, wherein: The initial position information parameters include a first starting boundary coordinate, a first ending boundary coordinate, a first starting address, a first ending address, a first starting sequence number, and a first ending sequence number; The first starting boundary coordinates represent relative coordinates of the starting boundary of the corresponding sliding window in the row direction of the extended data, and the first ending boundary coordinates represent relative coordinates of the ending boundary of the corresponding sliding window in the row direction of the extended data, wherein the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the extended data; The first starting address represents the address of the first starting boundary coordinate in the memory of the vector calculation unit, and the first ending address represents the address of the first ending boundary coordinate in the memory of the vector calculation unit; The first starting sequence number represents the sequence number of the data point corresponding to the first starting boundary coordinate at the first starting address, and the first ending sequence number represents the sequence number of the data point corresponding to the first ending boundary coordinate at the first ending address.

8. The method according to claim 7, wherein: The calculation formula of the first starting boundary coordinate is: src_row_start_index=i*str*ch, where src_row_start_index represents the first starting boundary coordinate, i represents the sequence number of the data point corresponding to the corresponding sliding window in the size wt of the target data, str represents the jump step of the sliding window in the row direction, and ch represents the number of channels of the input data; The calculation formula of the first end boundary coordinate is: src_row_end_index=src_row_start_index+(kernel_w*ch-1), where src_row_end_index represents the first end boundary coordinate, kernel_w represents the width of the sliding window, and the size of the sliding window is equal to the size of the initial convolution kernel; The calculation formula of the first starting address is: src_row_start_address=src_row_start_index / vmem_lane+j*N, where src_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, and j represents the row number of the corresponding data in the sliding window; The calculation formula of the first end address is: src_row_end_address=src_row_end_index / vmem_lane+j*N, where src_row_end_address represents the first end address; The calculation formula of the first starting sequence number is: src_row_start_lane=src_row_start_index%vmem_lane, wherein src_row_start_lane represents the first starting sequence number, and % represents a modulo operation; The calculation formula of the first end sequence number is: src_row_end_lane=src_row_end_index%vmem_lane, where src_row_end_lane represents the first end sequence number.

9. The method according to claim 6, wherein: The target location information parameters include a second starting boundary coordinate, a second ending boundary coordinate, a second starting address, a second ending address, a second starting sequence number, and a second ending sequence number; The second starting boundary coordinates represent relative coordinates of the starting boundary of the corresponding sliding window in the data size of [1, 1, (C×R×S)], and the second ending boundary coordinates represent relative coordinates of the ending boundary of the corresponding sliding window in the data size of [1, 1, (C×R×S)], and the starting boundary of the corresponding sliding window and the ending boundary of the corresponding sliding window are located at different positions in the row direction of the expanded data; The second starting address represents the address of the second starting boundary coordinate in the memory of the vector calculation unit, and the second ending address represents the address of the second ending boundary coordinate in the memory of the vector calculation unit; The second starting serial number represents the serial number of the data point corresponding to the second starting boundary coordinate at the second starting address, and the second ending serial number represents the serial number of the data point corresponding to the second ending boundary coordinate at the second ending address.

10. The method according to claim 9, wherein: The calculation formula of the second starting boundary coordinate is: dst_row_start_index=j*kernel_w*ch, where dst_row_start_index represents the second starting boundary coordinate, j represents the row number of the corresponding data in the sliding window, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels of the input data; The calculation formula of the second end boundary coordinate is: dst_row_end_index=dst_row_start_index+(kernel_w*ch-1), where dst_row_end_index represents the second end boundary coordinate; The calculation formula of the second starting address is: dst_row_start_address=dst_row_start_index / vmem_lane, wherein dst_row_start_address represents the second starting address, and vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit; The calculation formula of the second end address is: dst_row_end_address=dst_row_end_index / vmem_lane, where dst_row_end_address represents the second end address; The calculation formula of the second starting sequence number is: dst_row_start_lane=dst_row_start_index%vmem_lane, wherein dst_row_start_lane represents the second starting sequence number, and % represents a modulo operation; The calculation formula of the second end sequence number is: dst_row_end_lane=dst_row_end_index%vmem_lane, where dst_row_end_lane represents the second end sequence number.

11. The method according to claim 6, wherein: The vector calculation unit stores each set of data in a converted arrangement to a corresponding position of the target memory according to the initial position information parameter and the target position information parameter to obtain the target data, including: According to the initial position information parameters and the target position information parameters, the vector calculation unit uses a circular shift instruction and, according to a preset enable signal in the predicate register, splices each group of data in a converted arrangement and stores it in a corresponding position of the target memory to obtain the target data.

12. The method according to claim 1, wherein Performing a convolution operation based on the target data and the operational convolution kernel includes: A matrix operation unit is used to perform a convolution operation on the target data and the operational convolution kernel.

13. The method according to any one of claims 1 to 12, wherein: The convolution operation method is used for the first-layer convolution operation of a convolutional neural network.

14. A convolution operation device, comprising: A determining unit is configured to determine an operational convolution kernel, wherein the operational convolution kernel is obtained based on an initial convolution kernel, the initial convolution kernel is expressed as [R, S, C, K], and the operational convolution kernel is expressed as [1, 1, (C × R × S), K], where R, S, C, and K are all integers greater than 0; an adjusting unit configured to adjust an arrangement of the input data based on the number of channels of the operational convolution kernel to obtain target data, wherein a size and a number of channels of the target data are different from the size and the number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the operational convolution kernel; a computing unit configured to perform a convolution operation based on the target data and the operational convolution kernel to obtain a convolution operation result, wherein the convolution operation result of the target data and the operational convolution kernel is equal to the convolution operation result of the input data and the initial convolution kernel, Wherein, the adjustment unit is further configured as follows: Dividing the input data into lines and storing them in a static memory; Performing a padding operation on the input data stored in the static memory to obtain expanded data; Adjusting the arrangement of the extended data to change the size and number of channels of the extended data to obtain the target data, In which, each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0, and N is obtained by multiplying the number of data points in each row of the input data after the padding operation and the number of channels, divided by the number of data points that can be stored in each storage row in the static memory, and rounding up the resulting ratio.

15. An electronic device comprising the convolution operation device according to claim 14.

16. An electronic device comprising: processor; a memory comprising at least one computer program module; The at least one computer program module is stored in the memory and configured to be executed by the processor, and the at least one computer program module is used to implement the convolution operation method according to any one of claims 1 to 13.

17. A storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implements the convolution operation method according to any one of claims 1 to 13.

Images