Convolution method, convolution device, electronic device and storage medium

Abstract

A method for convolution operation, an apparatus for convolution operation, an electronic device, and a storage medium. The method for convolution operation includes: determining a convolution kernel for operation, where the convolution kernel for operation is obtained based on an initial convolution kernel, the initial convolution kernel is represented by [R, S, C, K], the convolution kernel for operation is represented by [1, 1, (C×R×S), K], and R, S, C, and K are all integers greater than 0; adjusting the array format of input data based on the number of channels of the convolution kernel for operation to obtain target data, where the size and the number of channels of the target data are different from those of the input data, and the number of channels of the target data is equal to the number of channels of the convolution kernel for operation; performing a convolution operation based on the target data and the convolution kernel for operation to obtain a result of the convolution operation. The result of the convolution operation between the target data and the convolution kernel for operation is equal to the result of the convolution operation between the input data and the initial convolution kernel. The convolution operation method can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, and shorten the time of the convolution operation.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the priority of a Chinese patent application with application number 202210610935.6 filed on May 31, 2022, and the entire content disclosed in that application is incorporated herein by reference.

[0002] [Technical Field] Embodiments of the present disclosure relate to a method for convolutional operations, an apparatus for convolutional operations, an electronic device, and a storage medium.

Background Art

[0003] With the development of technology, artificial intelligence (AI) technology has been widely used in many fields. Deep learning is one of the important technologies in AI technology. Deep learning technology based on artificial neural networks has already made great progress in fields such as object classification, text processing, image search, and human - machine interaction. Convolutional neural network (CNN) is a widely used deep learning technology that can directly input image data without requiring complex pre - processing and has great advantages in image processing and the like.

Summary of the Invention

[0004] At least one embodiment of the present disclosure is a method for a convolution operation, the method comprising the steps of: determining a convolution kernel for operation, the convolution kernel for operation is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], the convolution kernel for operation is represented as [1,1,(C×R×S),K], where R,S,C,K are all integers greater than 0; and obtaining target data by adjusting the arrangement of input data based on the number of channels of the convolution kernel for operation. The present invention provides a method for a convolution operation, comprising the steps of: a step in which the size and number of channels of the target data differ from the size and number of channels of the input data, wherein the number of channels of the target data is equal to the number of channels of the convolution kernel for calculation; and a step in which a convolution operation is performed based on the target data and the convolution kernel for calculation, and the result of the convolution operation between the target data and the convolution kernel for calculation is obtained, wherein the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0005] At least one embodiment of the present disclosure is an apparatus for a convolution operation, the apparatus comprising: a determination unit for determining a convolution kernel for calculation, the convolution kernel for calculation is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], the convolution kernel for calculation is represented as [1,1,(C×R×S),K], where R,S,C,K are all integers greater than 0; and an adjustment unit for adjusting the arrangement of input data based on the number of channels of the convolution kernel for calculation to acquire target data. The present invention provides a convolution apparatus that includes: an adjustment unit in which the size and number of channels of the acquired data differ from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the convolution kernel for calculation; and a calculation unit in which a convolution operation is performed based on the target data and the convolution kernel for calculation, and the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0006] At least one embodiment of the present disclosure further provides electronic equipment including a device for convolution operations provided by any embodiment of the present disclosure.

[0007] At least one embodiment of the present disclosure provides an electronic device further comprising a processor and a memory containing at least one computer program module, the at least one computer program module being stored in the memory and configured to be executed by the processor, and the at least one computer program module being used to implement a method for a convolution operation provided by any embodiment of the present disclosure.

[0008] At least one embodiment of the present disclosure further provides a storage medium for storing non-temporary computer-readable instructions, wherein a method for a convolution operation provided by any embodiment of the present disclosure is performed when the non-temporary computer-readable instructions are executed by a computer. [Brief explanation of the drawing]

[0009] To more clearly illustrate the technical solutions provided by the embodiments of this disclosure, some drawings of the embodiments are briefly shown below. Clearly, the drawings in the following description relate only to some embodiments of this disclosure and do not limit the disclosure.

[0010] [Figure 1] Figure 1 is a schematic diagram of the data flow of a convolution operation provided by some embodiments of this disclosure. [Figure 2] Figure 2 is a schematic flowchart of a convolution operation method provided by some embodiments of the present disclosure. [Figure 3] Figure 3 is a schematic diagram illustrating the principle of the convolution operation. [Figure 4] Figure 4 is a schematic flowchart of step S20 in Figure 2. [Figure 5] Figure 5 is a schematic flowchart of step S21 in Figure 4. [Figure 6] Figure 6 is a schematic diagram illustrating how input data is stored in memory in a convolution operation method provided by some embodiments of this disclosure. [Figure 7] Figure 7 is a schematic diagram of a method for storing input data in static memory in a convolution operation method provided by some embodiments of the present disclosure. [Figure 8] Figure 8 is a schematic flowchart of step S22 in Figure 4. [Figure 9] Figure 9 is a schematic flowchart of step S23 in Figure 4. [Figure 10] Figure 10 is a schematic flowchart of step S232 in Figure 9. [Figure 11] Figure 11 is a first schematic diagram of a data array scheme transformation in a convolution operation method provided by some embodiments of the present disclosure. [Figure 12] Figure 12 is a second schematic diagram illustrating the transformation of a data array scheme in a convolution operation method provided by some embodiments of the present disclosure. [Figure 13] Figure 13 is a schematic block diagram of a convolutional unit provided by some embodiments of the present disclosure. [Figure 14] Figure 14 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure. [Figure 15] Figure 15 is a schematic block diagram of another electronic device provided by an embodiment of the present disclosure. [Figure 16] Figure 16 is a schematic block diagram of another electronic device provided by an embodiment of the present disclosure. [Figure 17] Figure 17 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure. Forms for implementing disclosure

[0011] The embodiments of this disclosure will be described below with reference to the drawings. While specific embodiments of this disclosure are shown in the drawings, this disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. The drawings and embodiments of this disclosure are for illustrative purposes only and should not be used to limit the scope of protection of this disclosure.

[0012] It should be understood that each step described in the implementation of the method of this disclosure can be performed in a different order and / or in parallel. Furthermore, embodiments of the method may include additional steps and / or omit the execution of the illustrated steps. The scope of this disclosure is not limited in this respect.

[0013] As used herein, the term "comprising" and variations thereof are non-limiting, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment", the term "another embodiment" means "at least one additional embodiment", and the term "some embodiments" means "at least some embodiments". Related definitions of other terms are given in the following description.

[0014] It should be noted that concepts such as "first" and "second" referred to in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that the modifications of "one" and "plurality" referred to in the present disclosure are illustrative and not limiting. A person skilled in the art should understand them as "one or more" unless the context clearly indicates otherwise.

[0016] The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0017] The input data of a convolutional neural network is usually an image with 3 channels. For example, the input image of the convolution of the first layer of the Residual Network ResNet50 is [1, 224, 224, 3], that is, the input image has 3 channels, the image size of each channel is 224×224, and the shape of the convolution kernel (kernel) used in the convolution of the first layer of the Residual Network ResNet50 is [7, 7, 3, 64]. Generally used neural network accelerators are provided with a matrix operation unit (Matrix), and the matrix operation unit is mainly responsible for accelerating the matrix operations and convolution operations in the neural network. In order to accelerate the matrix operations, it is common for the matrix operation unit to increase the computational parallelism by increasing the computational scale. For example, the computational scale is set to 64×64, 128×128, etc. However, since the number of channels of the input data of the convolution of the first layer of the convolutional neural network is small (for example, 3 channels), the computing power of the matrix operation unit on the hardware accelerator (hardware accelerator) becomes low, and the computing time of the convolution of the first layer is relatively long, and the acceleration effect is not obvious. Furthermore, strictly following the data array method with aligned channels (Channel Align Tensor Layout) will significantly increase the storage space of the data and increase the data transmission time.

[0018] As shown in Figure 1, hardware accelerators are typically mounted on the host's PCIe (Peripheral Component Interconnect Express) node as slave devices to the host. PCIe is a high-speed serial computer expansion bus standard that enables high-speed data transmission. The hardware accelerator is the device side in relation to the host. When performing a convolution operation, the data input in the first layer of convolution must be transmitted from the host to the hardware accelerator via PCIe. This process is called Host2Device. For example, the Central Processing Unit (CPU) reads data from memory, transmits it to the hardware accelerator on the device side via PCIe, and stores it in the memory (e.g., DDR) on the hardware accelerator. The hardware accelerator can then perform the convolution operation using this data.

[0019] Taking the convolution of the first layer of the residual network ResNet50 as an example, the first layer requires the operation of the input data and the convolution kernel. This can be represented as [1,224,224,3] × [7,7,3,64] = [1,112,112,64]. Here, the input data is represented as [1,224,224,3], which is 3-channel data with a size of 224 × 224. The convolution kernel is represented as [7,7,3,64], which is 64 groups, each group having 3 convolution kernels, and the size of the convolution kernel is 7 × 7. The resulting output is 64-channel data with a size of 112 × 112.

[0020] Assuming the matrix unit on the hardware accelerator is 64x64, due to the limitations of the data array scheme with aligned channels, the input data for the first layer of convolution needs to be expanded on the host from [1,224,224,3] to [1,224,224,64], and all redundant channel data is filled with zeros. The memory space needs to increase by 21.33 times, and similarly, the time to transmit data from the host to the hardware accelerator also increases by 21.33 times. In this case, the computational power utilization rate of the matrix unit is only 4.68%, and regarding the convolution time, it takes the matrix unit 614,656 cycles to complete the first layer of convolution.

[0021] The first layer of a convolutional neural network faces several problems because the input data has a small number of channels, and the matrix operation unit of the hardware accelerator is large. The computational requirements and hardware characteristics do not match, leading to the following issues: First, the input data requires rearranging the array scheme using the host CPU, increasing the occupied memory space and CPU time. Second, the increased amount of rearranged input data increases the PCIe transmission time of the Host2Device. Third, the utilization rate of the matrix operation unit of the hardware accelerator is low, and its full computational power cannot be used, resulting in wasted hardware resources. Fourth, the matrix operation unit of the hardware accelerator takes a long time to perform the first layer of convolution, failing to achieve the purpose of the hardware accelerator.

[0022] At least one embodiment of the present disclosure provides a convolution operation method, a convolution operation device, an electronic device, and a storage medium. The convolution operation method can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the time of convolution operations, improve computational efficiency, and save data transmission time.

[0023] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Note that the same reference numerals in different drawings are used to refer to the same elements described.

[0024] At least one embodiment of the present disclosure provides a convolution operation method. The convolution operation method includes the steps of: determining a convolution kernel for calculation, the convolution kernel for calculation is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for calculation is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0; adjusting the arrangement of input data based on the number of channels of the convolution kernel for calculation to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the convolution kernel for calculation; and performing a convolution operation based on the target data and the convolution kernel for calculation to obtain the result of the convolution operation. The result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0025] Figure 2 is a schematic flowchart of a convolution operation method provided by some embodiments of the present disclosure. As shown in Figure 2, in some embodiments, the convolution operation method includes steps S10 to S30. Step S10: A step to determine the convolution kernel for operations, the convolution kernel for operations is obtained based on the initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for operations is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0. Step S20: Based on the number of channels in the convolution kernel for calculation, the array scheme of the input data is adjusted to obtain the target data, where the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels in the convolution kernel for calculation. Step S30: A convolution operation is performed based on the target data and the convolution kernel, and the result of the convolution operation is obtained. The result of the convolution operation between the target data and the convolution kernel is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0026] For example, the convolution method may be used for the convolution of the first layer of a convolutional neural network. Of course, the embodiments of this disclosure are not limited thereto, and the convolution method may be used for convolutions of other types of networks as well as convolutional neural networks, and may be used for convolutions of other layers as well as the first layer (the convolution of the first layer), and may be determined according to the actual needs, and the embodiments of this disclosure are not limited thereto.

[0027] For example, in step S10, the initial convolution kernel is the convolution kernel required for the convolution operation of the first layer, and is represented as [R,S,C,K]. Taking the convolution of the first layer of the residual network ResNet50 as an example, the operation required for the convolution of the first layer is [1,224,224,3]×[7,7,3,64]=[1,112,112,64], so the initial convolution kernel [R,S,C,K] is [7,7,3,64], and in this example, R=7, S=7, C=3, K=64. The parameters of the initial convolution kernel are transformed to obtain the operation convolution kernel [1,1,(C×R×S),K]. In the above example, the operation convolution kernel can be obtained based on the initial convolution kernel, and the operation convolution kernel is [1,1,147,64]. The transformation principle of the convolution kernel will be briefly explained below with reference to Figure 3.

[0028] Figure 3 is a schematic diagram illustrating the principle of convolution. As shown in Figure 3, the size of the input data is [1,3,3,5], the size of the convolution kernel is [2,2,5,4], and the size of the output data is [1,2,2,4]. For example, the calculation method for M points is shown in Figure 3. Since the size of the convolution kernel is 2×2 and there are 5 channels, M points are the result of multiplying 20 points each of the input data and the convolution kernel and accumulating them. By utilizing the properties of convolution, the convolution kernel can be transformed from R×S×C×K to 1×1×(C×R×S)×K, and the input data can be adjusted accordingly, so the overall calculation result of the convolution remains unchanged. The number of channels can be increased through such a transformation operation. For the first layer of the convolutional neural network, the convolutional kernel is adjusted from [7,7,3,64] to [1,1,147,64], expanding the number of channels from 3 to 3×7×7=147, thereby maximizing the computational power of the matrix unit (Matrix). Therefore, in step S10, the initial convolutional kernel [R,S,C,K] can be transformed to obtain the computational convolutional kernel [1,1,(C×R×S),K], thereby realizing a change in the arrangement of the convolutional kernel.

[0029] For example, the convolution kernel array may be modified offline. This is because the convolution kernel used by the neural network model is fixed during the deployment phase and does not change even if the input changes, and therefore the convolution kernel can be preprocessed into the required array scheme. In an embodiment of this disclosure, during the deployment phase of the neural network model, the convolution kernel to be used [R,S,C,K] is set to [1,1,(C×R×S),K] for later use. For example, during the model compilation phase, the code corresponding to the initial convolution kernel [R,S,C,K] can be modified and adjusted using a high-level language (e.g., Python) to obtain the operational convolution kernel [1,1,(C×R×S),K]. Of course, embodiments of this disclosure are not limited to this, and the initial convolution kernel [R,S,C,K] can also be adjusted before each convolution operation to obtain the operational convolution kernel [1,1,(C×R×S),K] that will actually be used in the current operation.

[0030] Returning to Figure 2, in step S20, the array structure of the input data is adjusted based on the number of channels in the convolution kernel used for calculation to obtain the target data. For example, the size and number of channels of the target data differ from the size and number of channels of the input data, where the number of channels in the target data is equal to the number of channels in the convolution kernel used for calculation. Taking the convolution of the first layer of the residual network ResNet50 as an example, the calculation performed is [1,224,224,3] × [7,7,3,64] = [1,112,112,64], and the convolution kernel is adjusted to [1,1,147,64]. Therefore, it is necessary to adjust the array structure of the input data so that the calculation result does not change, and the calculation is changed to [1,112,112,147] × [1,1,147,64] = [1,112,112,64]. As a result, it can be seen that the data obtained after adjusting the input data is [1,112,112,147]. For example, the data obtained after adjusting the array structure of the input data is called the target data, and the target data is the data on which the convolution kernel and convolution operation have been performed. For example, by adjusting the array structure, the size and number of channels of the target data will differ from the size and number of channels of the input data. From the above calculation formula, it can be seen that the number of channels of the target data is equal to the number of channels of the convolution kernel (for example, 147 in both cases above), which makes both convolution operations easier.

[0031] For example, if the number of channels in the target data is greater than the number of channels in the input data, and the number of channels in the convolution kernel for calculations is greater than the number of channels in the initial convolution kernel, the number of channels increases, making maximum use of the computational power of the matrix operation unit. For example, in the above example, the number of channels in the input data and the number of channels in the initial convolution kernel are both 3, and the number of channels in the target data and the number of channels in the convolution kernel for calculations are both 147, thus increasing the number of channels. For example, the transformation of the input data's array structure needs to be completed online, meaning that the array structure of the input data needs to be adjusted each time during the neural network's inference phase.

[0032] Figure 4 is a schematic flowchart of step S20 in Figure 2. For example, in some embodiments, step S20 may further include steps S21 to S23, as shown in Figure 4. Step S21: The input data is stored row by row in static memory, with each row of input data being stored in corresponding N row-stores in static memory, where N is an integer greater than 0. Step S22: Perform a fill operation on the input data stored in static memory to obtain expanded data. Step S23: Adjust the array format of the augmented data, change the size and number of channels of the augmented data, and obtain the target data.

[0033] For example, in step S21, the input data is first stored in static memory provided in the hardware accelerator, which is, for example, static random access memory (SRAM). The input data is stored in the static memory row by row; that is, each row of input data is stored in the corresponding N storage rows in the static memory, where N is an integer greater than 0. For example, the data flow shown in Figure 1 may be used to transmit the input data to the static memory.

[0034] As shown in Figure 5, step S21 may further include steps S211 and S212. Step S211: The input data is densely arranged and stored in memory. The input data includes multiple channels, and the dense arrangement means that multiple channels of the same data point are stored sequentially adjacent to each other in memory. Step S212: Using Direct Memory Access (DMA), the input data in memory is transmitted to the static memory of the hardware accelerator, and the first data point of each row of the input data is stored in the first column of a different row in the static memory, so that each row of the input data is stored in the corresponding N storage rows in the static memory.

[0035] For example, in step S211, dense arrangement means, for instance, a channel-aligned tensor layout. The input data contains multiple channels, and dense arrangement means that multiple channels of the same data point are stored sequentially adjacent in memory. As shown in Figure 6, in some examples, the input data for the first layer of convolution is [1,224,224,3], where the number 1 represents Batch Size=1, one of the numbers 224 represents Height=224, the other represents Width=224, and the number 3 represents Channel=3; in other words, the input data is a 3-channel image with a size of 224×224. A dense data arrangement storage method is employed to reduce the data storage space on the host. For example, for a pixel point in the first row and first column of the input data, the values ​​of its three channels are stored sequentially in memory space. Then, the pixel point in the second row and second column is stored, and the values ​​of its three channels are stored sequentially in memory space, and so on.

[0036] For example, the value of each channel of each pixel is represented in the FP16 data format and occupies a 2-byte address space. The input image of the first layer totals 224 × 224 × 3 × 2 bytes = 301,056 bytes, or 294 KB. Here, the data format used and the address space occupied are illustrative and do not limit the embodiments of this disclosure.

[0037] For example, in step S212, the input data in memory is transmitted to the static memory of the hardware accelerator using the Direct Memory Access (DMA) method. The data transfer method may be the method shown in Figure 1, where the input data for the first layer of convolution is transferred from the host side to the device side's DDR via PCIe, and the input data is stored in the DDR in the same way as it is stored in the host memory, thereby realizing a 1D tensor Host2Device process. For example, the input data is stored contiguously in the DDR, occupying, for example, 294KB. Next, it is necessary to transmit the input data to static memory, that is, this 294KB of data needs to be transferred from the DDR to the SRAM in the processing engine (PE, also called the hardware accelerator).

[0038] For example, when data is stored, the first data point of each row of input data is stored in the first column of a different row in static memory, and each row of input data is stored in the corresponding N storage rows in static memory.

[0039] As shown in Figure 7, in some examples, the configuration of the SRAM can be thought of abstractly as an M x N table, where one data item is stored in each table. Since the size of the input data is 224 x 224, the input data is logically divided into 224 rows, with the starting position of each row being the first column of a row in the SRAM. Because the number of SRAM columns is limited, it is difficult for an entire row of input data to be stored in a single row of SRAM. Therefore, a single row of input data is distributed across multiple rows of SRAM, i.e., different SRAM addresses. For the input data [1,224,224,3], considering the data filled by a later fill operation, each row has 229 points, and each point has 3 channels. For an SRAM with 1024 bits of memory space for each row, the number of SRAM rows required to store one row of data is ceil(229*3 / 64) = 11, where ceil represents rounding up. In other words, each data point in one row of input data is stored in 11 memory rows of SRAM. In this example, N=11, and the entire input data occupies 224 × 11 = 2464 rows of SRAM.

[0040] As shown in Figure 7, the left side shows that input data is stored contiguously in DDR without the concepts of H, W, and C, while the right side shows that after being transferred to SRAM by DMA, the data is divided in SRAM according to the rows of input data, with each row of data occupying a certain amount of SRAM space (e.g., 11 storage rows). This enables data transfer from DDR to SRAM in PE and completes the conversion from a one-dimensional tensor (1D Tensor) to a two-dimensional tensor (2D Tensor).

[0041] The DMA transfer process can be briefly explained as follows:

[0042] Assuming the input data is stored in a contiguous DDR space starting at the source address (source_address), first, the first row of 224 × 3 × 2 Bytes = 1344 Bytes of data needs to be transferred to a contiguous SRAM space starting at the destination address (destiny_address). Since one row of SRAM is 128 Bytes, these 1344 Bytes need to be stored in ceil(1344 / 128) = 11 rows of SRAM, meaning the DMA needs to continuously transmit 11 × 128 Bytes of data. After the data transfer for the first row is complete, the DMA jumps the read address from source_address to source_address + 1344 Bytes, i.e., to the DDR address at the beginning of the second row of the actual input data, and then needs to continuously transfer 11 × 128 Bytes into the SRAM space starting at destination_address + 11. By analogy, after 224 transfers are complete, all input data has already been transferred from the DDR to the SRAM within the processing engine, meaning the conversion from a one-dimensional tensor to a two-dimensional tensor is complete.

[0043] Although the amount of data transmitted each time (11 x 128 bytes) is greater than the actual amount of data in each line, meaning that the data for the next line is also included, the initial address transmitted each time is accurate, so repeated transmission of data does not affect the data itself, and this redundant data does not affect subsequent processing.

[0044] Returning to Figure 4, for example, in step S22, a padding operation is performed on the input data stored in static memory to obtain augmented data. Here, augmented data refers to the data obtained after the padding operation. For example, in some cases, let's assume that the actual convolution operation that needs to be completed is [1,224,224,3] × [7,7,3,64] = [1,112,112,64], and then the input data needs to be padded in all directions: up, down, left, and right. When padding, three points are added to the left and above the input data (filling three columns on the left and three rows above), and two points are added to the right and below the input data (filling two columns on the right and two rows below). The size of the augmented data obtained after the padding operation is [1,229,229,3].

[0045] Figure 8 is a schematic flowchart of step S22 in Figure 4. As shown in Figure 8, step S22 may further include steps S221 to S223. Step S221: In static memory, first preset values ​​are filled into the memory rows corresponding to the storage position before and after the storage position of the input data, and first intermediate data is obtained. Here, the first intermediate data includes the input data and the filled first preset values. Step S222: The first intermediate data is transmitted to the vector calculation unit, and the second preset values ​​are filled into both ends of each row corresponding to the first intermediate data using the shift command and fill command of the vector calculation unit, thereby obtaining the second intermediate data. Here, the second intermediate data includes the first intermediate data and the filled second preset values. Step S223: The second intermediate data is transmitted to the corresponding storage location in static memory to obtain the extended data. The extended data has the same content as the second intermediate data.

[0046] For example, in step S221, in static memory, a first preset value is filled into the memory rows before and after the memory location of the input data, and the first intermediate data is obtained. The first intermediate data includes the input data and the filled first preset value. In this step, for example, padding operations are performed above and below the input data. For example, in some cases, it is necessary to prepare the SRAM space required for padding above near the destination address in the SRAM, that is, it is necessary to insert several rows of data before actually inputting the first row of data. The top and bottom padding operations are performed using the vector calculation unit (Vector) of the hardware accelerator. For example, since the filled first preset value is usually 0, in order to obtain the first intermediate data, it is necessary to write the value 0 to several addresses before and after the memory space of the input data in the SRAM, thereby obtaining the first intermediate data. The first intermediate data is the data that has been padded above and below, but has not yet been padded to the left and right.

[0047] For example, in step S222, the first intermediate data is transmitted to the vector calculation unit, and the second preset values ​​are filled into both ends of each row corresponding to the first intermediate data using the shift command (e.g., vshiftri command) and fill command (e.g., SI2V command) of the vector calculation unit, thereby obtaining the second intermediate data. The second intermediate data includes the first intermediate data and the filled second preset values. In this step, for example, the fill operation is performed to the left and right of the first intermediate data.

[0048] For example, in some cases, data in the 2464 address spaces within SRAM is grouped into 11-row units, transmitted sequentially to the vector calculation unit, and stored in the vmem memory space within the vector calculation unit. Next, the vector calculation unit uses the vshiftri instruction to shift the entire data to the right, leaving space for left-side padding, and then uses the SI2V instruction to write these positions to the corresponding second preset value (for example, usually set to 0). In the case of right-side padding, after the entire data is shifted to the right, the corresponding second preset value is written after the last column of the first row of the input data. If the amount of data to be padded is too large, additional vmem space may be needed as required. For example, a pipeline approach can be used to perform left and right padding operations on multiple groups of 11-row data to improve processing efficiency.

[0049] For example, in step S223, the second intermediate data is transmitted to the corresponding storage location in static memory to obtain extended data. The extended data has the same content as the second intermediate data. In other words, the second intermediate data, for which the filling operation in vmem has been completed, is written back into the corresponding address space of SRAM, and the data stored in SRAM after the filling operation has been completed is called extended data.

[0050] Furthermore, if it is not necessary to perform a filling operation on the input data, step S22 may be omitted. In addition, in the embodiments of this disclosure, if a filling operation is necessary, the top and bottom filling may be performed first, followed by the left and right filling, or the left and right filling may be performed first, followed by the top and bottom filling, and the specific filling order is not limited. Furthermore, the instructions used when performing the filling operation are not limited to the vshiftri instruction and the SI2V instruction, and other appropriate instructions that can realize the filling operation may be used, and the embodiments of this disclosure are not limited thereto.

[0051] Returning to Figure 4, for example, in step S23, the array scheme of the extended data is adjusted, and the size and number of channels of the extended data are changed to obtain the target data. In other words, in order to ensure that the calculation result does not change in accordance with the convolution kernel, it is necessary to adjust the array scheme of the extended data and change its size and number of channels. For example, the number of channels of the target data obtained after adjustment is equal to the number of channels of the convolution kernel, and the target data is represented as [1, ht, wt, (C × R × S)], where both ht and wt are integers greater than 0.

[0052] Figure 9 is a schematic flowchart of step S23 in Figure 4. As shown in Figure 9, step S23 may further include steps S231 and S232. Step S231: The data from R×N memory rows in static memory is read one after another and transmitted to the vector calculation unit. Step S232: The vector calculation unit converts the R*N data from each received memory row into wt*ceil((C×R×S) / L) data from memory rows and obtains the target data.

[0053] For example, in step S231, data from R*N memory rows is read each time and transmitted to the vector calculation unit, which then converts the R*N memory rows received each time. For example, the starting address read each time moves through str*N memory rows according to a preset number of hops str. The preset number of hops is the number of hops in the row and column directions of the sliding window required for the convolution operation between the input data and the initial convolution kernel, and the total number of times data is read from static memory is equal to ht.

[0054] For example, in some cases, we take the convolution of the first layer of the residual network ResNet50 as an example. Since the initial convolution kernel is [R,S,C,K]=[7,7,3,64], R=7. The input data is [1,224,224,3], and one row of the input data is stored in N memory rows in SRAM. If the space of one row of SRAM is 128 bytes, then N=11. Therefore, the data in R*N=77 memory rows in static memory is read each time and transmitted to the vector calculation unit. The data stored in these 77 memory rows corresponds to one row of 224×224 input data. For example, the number of hops in the row and column directions of the sliding window required for the convolution operation between the input data [1,224,224,3] and the initial convolution kernel [7,7,3,64] is set to 2, and therefore the above-pre-set number of hops str is 2. For example, each time the starting address read is moved by str*N (i.e., 2×11=22) memory rows according to a predetermined number of hops str, thereby matching the read data with the data included in the sliding window when performing the convolution operation. According to the calculation formula [1,224,224,3]×[7,7,3,64]=[1,112,112,147]×[1,1,147,64]=[1,112,112,64], the convolution kernel is converted to [1,1,147,64], and the obtained target data [1,ht,wt,(C×R×S)] needs to be converted to [1,112,112,147], so it can be seen that ht=112 and wt=112. For example, the total number of times data is read from static memory is ht (e.g., 112), and each time data is read, it corresponds to one row of the target data after being transformed.

[0055] For example, in step S232, the vector calculation unit converts the R*N data of memory rows received each time into wt*ceil((C×R×S) / L) data of memory rows, and the converted data is the target data. That is, it adjusts the data arrangement method and changes the data size and number of channels. For example, in the calculation formula, L represents the number of data points that can be stored in each memory row of static memory, and ceil((C×R×S) / L) represents rounding up to the nearest integer for (C×R×S) / L. For example, in some cases, the vector calculation unit receives data from 7 × 11 memory rows each time, and if the space of one row of SRAM is 128 bytes, the number of data points L that can be stored in each memory row is 64, and for the initial convolution kernel [7,7,3,64], R=7, S=7, C=3, and if the target data [1,ht,wt,(C×R×S)]=[1,112,112,147], then wt=112. Therefore, wt*ceil((C×R×S) / L)=112×3, meaning the vector calculation unit converts the data from the 7 × 11 memory rows received each time into data from 112 × 3 memory rows.

[0056] Figure 10 is a schematic flowchart of step S232 in Figure 9. For example, in some cases, step S232 above further includes steps S2321 to S2323. Step S2321: Divide the data in R*N storage rows into multiple groups according to a predetermined number of hops. Step S2322: For each group of data, determine the initial position information parameter and target position information parameter for each row of data in the sliding window corresponding to the data of that group. Step S2323: The vector calculation unit stores the data of each group in the corresponding position in the target memory in the transformed array format according to the initial position information parameter and the target position information parameter, and obtains the target data.

[0057] For example, in step S2321, the data in R*N storage rows is divided into multiple groups according to a preset number of hops, and the data in each group corresponds to a sliding window in the row direction, and the number of data in multiple groups is equal to wt. For example, in some examples, the data in 7×11 storage rows is divided into 112 groups according to a preset number of hops str=2, and wt=112. The data in 7×11 storage rows corresponds to the data in one row of the input data 224×224, and the divided data in the 112 groups corresponds to 112 sliding windows created with 2 hops per row for each of the 224 data points.

[0058] For example, in step S2322, for each group of data, the initial position information parameter and the target position information parameter are determined for each row of data in the sliding window corresponding to the data of that group. The initial position information parameter is used to determine the source address where the data row in the sliding window is located, and the target position information parameter is used to determine the destination address to which this data is transferred.

[0059] The following describes the operating modes of steps S2321 to S2322 described above with an example.

[0060] For example, after performing padding operations on the top, bottom, left, and right sides of the input data [1,224,224,3], the total size of the input data becomes [1,229,229,3], occupying 229 × 11 address spaces in SRAM. Next, the shape of the input data needs to be converted to [1,112,112,147]. Basically, for convolution operations, each sliding window (Feature Window) that the convolution kernel slides through needs to complete the conversion from [7 × 7 × 3] to [1,1,147], as shown in Figure 11.

[0061] Each sliding window that a convolution kernel slides through corresponds to the original data (input data or input image) in a 7x7 grid. Therefore, during the process of sliding from the top left to the bottom right, the sliding windows swept by the convolution kernels overlap. To avoid repeatedly reading data from SRAM for column-direction overlap, 7x11=77 address spaces of data are read from SRAM each time and transmitted to the vector calculation unit for processing. For row-direction overlap, as the sliding window slides from left to right, overlapping data in the row direction is repeatedly read. Overall, the data in SRAM is divided into 112 groups, corresponding to 112 rows after conversion. The data in each group occupies 7x11 address spaces before conversion. The vector calculation unit reads and processes one group of data, outputting data in a 112x3 address space, where 112 corresponds to the converted data width and 3 corresponds to the space occupied by 147 channels (147 channels occupy 3 SRAM memory rows, which means they need to occupy 3 SRAM address spaces).

[0062] After the vector calculation unit acquires the data, the data on these 7x11 SRAM memory rows (entries) is temporarily stored in the vmem within the vector calculation unit, and its data arrangement remains unchanged. Subsequently, the data is converted into 112x3 vmem memory rows using the instructions of the vector calculation unit. Finally, the result is written back to SRAM.

[0063] The converted data width is 112 points in the row direction, with 147 channels per point, distributed across three vmem storage rows. For each point in the row direction, the sliding window corresponding to the original 7x7x3 needs to be converted to 1x1x147. Therefore, it is necessary to retrieve the 7 rows of data corresponding to each sliding window and reconstruct them into a new data array. As shown in Figure 12, for the first sliding window, the data width of the sliding window is 7x3=21 channels, with 7 rows of data (stored in 7x11 storage rows). The storage addresses of these 7 rows of data need to be determined, and their array scheme needs to be converted. Referring to the right side of Figure 12, these seven rows of data are rearranged into three rows, with the original rows 0, 1, and 2 forming the new row 1, the original rows 3 and 4 forming the new row 2, and the original rows 5 and 6 forming the new row 3. These three new rows store a total of 147 data points contained within the sliding window corresponding to the original 7 × 7 × 3. By analogy, for the next sliding window in the row direction, the data arrangement is transformed in a similar manner until all the single-row sliding windows corresponding to seven rows of data are transformed, and then the data for the next group of 7 × 11 stored rows is read.

[0064] To determine the initial and target positions of the data in each row within the sliding window, it is necessary to define initial position information parameters and target position information parameters for each row of data within the sliding window.

[0065] The initial position information parameters include the first start boundary coordinates, the first end boundary coordinates, the first start address, the first end address, the first start number, and the first end number.

[0066] The first start boundary coordinates represent the relative coordinates in the row direction of the extended data to the start boundary of the corresponding sliding window, and the first end boundary coordinates represent the relative coordinates in the row direction of the extended data to the end boundary of the corresponding sliding window. The start boundary and end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data. Since the data on which the fill operation has been performed is extended data, these coordinates and parameters are defined for the extended data. In other cases where the fill operation is not required, these coordinates and parameters may be defined directly for the input data. As shown in Figure 12, the start boundary is, for example, the left boundary of the sliding window, and the first start boundary coordinates are the relative coordinates in the row direction of the 229×3 extended data to the left boundary of the sliding window, and the end boundary is, for example, the right boundary of the sliding window, and the first end boundary coordinates are the relative coordinates in the row direction of the 229×3 extended data to the right boundary of the sliding window.

[0067] The formula for calculating the first starting boundary coordinates is src_row_start_index = i * str * ch. src_row_start_index represents the first starting boundary coordinates, i represents the number of the corresponding data point in the corresponding sliding window at the target data size wt (for example, which of the 112 sliding windows in a row it is, i.e., which of the output data width wt = 112 it is), str represents the number of hops in the row direction of the sliding window (e.g., 2), and ch represents the number of channels in the input data (e.g., 3).

[0068] The formula for calculating the first end boundary coordinates is src_row_end_index = src_row_start_index + (kernel_w * ch - 1). src_row_end_index represents the first end boundary coordinates, kernel_w represents the width of the sliding window (e.g., 7), and the size of the sliding window is equal to the size of the initial convolution kernel (e.g., both are 7x7).

[0069] The first start address represents the address in the vector calculation unit's memory (e.g., vmem) of the first start boundary coordinates, and the first end address represents the address in the vector calculation unit's memory (e.g., vmem) of the first end boundary coordinates. The first start number represents the number of the data point corresponding to the first start address of the first start boundary coordinates, and the first end number represents the number of the data point corresponding to the first end address of the first end boundary coordinates. Since vmem is stored row by row, a storage row in vmem can be found according to the first start address or the first end address, and the first start number or the first end number indicates which data item in that storage row corresponds to the data.

[0070] The formula for calculating the first starting address is src_row_start_address = src_row_start_index / vmem_lane + j * N. src_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row within each memory of the vector calculation unit, and j represents the row number in the sliding window of the corresponding data (for example, a value from 1 to 7).

[0071] The formula for calculating the first end address is src_row_end_address = src_row_end_index / vmem_lane + j * N. src_row_end_address represents the first end address.

[0072] The formula for calculating the first starting number is src_row_start_lane = src_row_start_index % vmem_lane. src_row_start_lane represents the first starting number. For example, % represents a modulo operation.

[0073] The formula for calculating the first end number is src_row_end_lane = src_row_end_index % vmem_lane. src_row_end_lane represents the first end number.

[0074] Once the above parameters are determined, the location within the vmem of the source data needed to convert the 7x7x3 can be determined. To transfer this source data to the destination address in the vmem, the corresponding destination address and associated parameters must be determined, which means that the target location information parameters must be further determined.

[0075] The target location information parameters include the second start boundary coordinates, the second end boundary coordinates, the second start address, the second end address, the second start number, and the second end number.

[0076] The second start boundary coordinate represents the relative coordinate of the start boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], and the second end boundary coordinate represents the relative coordinate of the end boundary of the corresponding sliding window within the data size [1,1,(C×R×S)]. The start boundary and end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data. For example, the target data represents [1,ht,wt,(C×R×S)]. In some examples, the target data is [1,112,112,147], and the data size corresponding to each sliding window needs to be converted from [7,7,3] to [1,1,147]. As shown in Figure 12, the starting boundary is, for example, the left boundary of the sliding window, and the second starting boundary coordinates are the relative coordinates of the left boundary of the sliding window within the data size [1,1,147], and the ending boundary is, for example, the right boundary of the sliding window, and the second ending boundary coordinates are the relative coordinates of the right boundary of the sliding window within the data size [1,1,147].

[0077] The formula for calculating the second starting boundary coordinates is dst_row_start_index = j * kernel_w * ch. dst_row_start_index represents the second ending boundary coordinates, j represents the row number in the sliding window of the corresponding data (e.g., a value between 1 and 7), kernel_w represents the width of the sliding window (e.g., 7), the size of the sliding window is equal to the size of the initial convolution kernel (e.g., both are 7x7), and ch represents the number of channels in the input data (e.g., 3).

[0078] The formula for calculating the second end boundary coordinates is dst_row_end_index = dst_row_start_index + (kernel_w*ch-1). dst_row_end_index represents the second end boundary coordinates.

[0079] The second start address represents the address in the vector calculation unit's memory (e.g., vmem) of the second start boundary coordinates, and the second end address represents the address in the vector calculation unit's memory (e.g., vmem) of the second end boundary coordinates. The second start number represents the number of the data point corresponding to the second start address of the second start boundary coordinates, and the second end number represents the number of the data point corresponding to the second end address of the second end boundary coordinates. Since vmem is stored row by row, a storage row in vmem can be found according to the second start address or the second end address, and the second start number or the second end number indicates which data item in that storage row corresponds to the data.

[0080] The formula for calculating the second starting address is dst_row_start_address = dst_row_start_index / vmem_lane. dst_row_start_address represents the second starting address, and vmem_lane represents the number of data points that can be stored in each memory row of the vector calculation unit.

[0081] The formula for calculating the second end address is dst_row_end_address = dst_row_end_index / vmem_lane. dst_row_end_address represents the second end address.

[0082] The formula for calculating the second starting number is dst_row_start_lane = dst_row_start_index % vmem_lane. dst_row_start_lane represents the second starting number.

[0083] The formula for calculating the second end number is dst_row_end_lane = dst_row_end_index % vmem_lane. dst_row_end_lane represents the second end number.

[0084] After the initial location information parameters and target location information parameters are determined, the source address and destination address required for data transfer can be determined, and the source data is then transferred to the destination address based on these parameters.

[0085] For example, in step S2323, after the initial position information parameter and the target position information parameter are determined, the vector calculation unit stores the data of each group in the converted array format at the corresponding location in the target memory. The destination address indicated by the target position information parameter is an address in the target memory, thereby obtaining the target data. For example, the target memory stores data row by row, data is transferred to the target memory, and the data stored on the target memory is the target data. For example, the target memory may be the static memory described above (in this case, the data before conversion and the data after conversion are stored at different addresses in the static memory), or it may be another storage device different from the static memory described above. The embodiments of this disclosure are not limited thereto.

[0086] For example, step S2323 may further include a step in which the vector calculation unit, using a cyclic shift instruction according to the initial position information parameter and the target position information parameter, concatenates the data of each group in a converted array format according to a pre-set enable signal in the predicate register, stores it in the corresponding position in the target memory, and obtains the target data. For example, in some examples, the vshiftri instruction of the Vector ISA instruction set architecture of the vector calculation unit may be used to cyclically shift the data at the source address several positions to the right, and then write it to the destination address according to the write enable signal in the predicate register (also called the Vector Predicate Register, VPR, or VP register), where the aforementioned pre-set enable signal is, for example, the write enable signal. In the data concatenation process that converts data corresponding to a 7x7x3 sliding window into 1x1x147 data, it is necessary to determine the VP register to be used based on a second start number dst_row_start_lane and a second end number dst_row_end_lane. The use of the vshiftri instruction and VP registers can be discussed in reference to previous designs, but will not be explained in detail here.

[0087] Using the method described above, the conversion from a 2D tensor (2DTensor) to a 3D tensor (3DTensor) is completed using the vector calculation unit.

[0088] After processing at each step, the input data [1,224,224,3] is transformed into the target data [1,112,112,147], and the convolution kernel for the operation, determined based on the initial convolution kernel [7,7,3,64], is [1,1,147,64], thereby increasing the number of channels from 3 to 147.

[0089] Returning to Figure 2, in step S30, a convolution operation is performed based on the target data and the computational convolution kernel, and the convolution result is obtained. The result of the convolution operation between the target data and the computational convolution kernel is equal to the result of the convolution operation between the input data and the initial convolution kernel. For example, step S30 may further include a step of performing a convolution operation on the target data and the computational convolution kernel using a matrix operation unit (Matrix).

[0090] For example, in some implementations, taking the convolution of the first layer of the residual network ResNet50 as an example, the input data and the operation of the convolution kernel that needs to be performed are [1,224,224,3] × [7,7,3,64] = [1,112,112,64], and since there are only three channels, the computational power of the matrix operation unit is not fully utilized. By using the convolution operation method provided by the embodiments of this disclosure, the convolution kernel for calculation obtained based on the initial convolution kernel [7,7,3,64] is [1,1,147,64], and the target data obtained after adjusting the array scheme of the input data [1,224,224,3] is [1,112,112,147]. Therefore, the actual operation performed on the target data and the convolution kernel is [1,112,112,147] × [1,1,147,64] = [1,112,112,64], and the result of this convolution operation matches the result of the convolution operation that was originally required. The number of channels is increased to 147, the computational power of the matrix operation unit can be utilized to the fullest extent, the utilization rate of the matrix operation unit is improved, the time of the convolution operation is shortened, and the computational efficiency is improved. Furthermore, since there is no need to readjust the input data arrangement method on the host CPU, and no need to expand the number of channels on the host, the occupied data space does not increase significantly, nor does the amount of data transmitted from the host to the device increase significantly. Therefore, the PCIe transmission time of Host2Device does not increase, and data transmission time can be saved.

[0091] The convolution method provided by the embodiments of this disclosure helps achieve the objective of hardware acceleration, enabling faster convolution of the first layer of a convolutional neural network (CNN), and features small memory space, short transmission time, high utilization of hardware modules, and short computation time. For example, the time required to perform the convolution of the first layer of the residual network ResNet50 using a conventional convolution method is 614,656 cycle times, but using the convolution method provided by the embodiments of this disclosure, the theoretical time required to perform the convolution of the first layer of the residual network ResNet50 is 37,632 cycle times, which is a reduction to 6.1% of the previous level, significantly shortening the convolution time of the first layer of the convolutional neural network (CNN).

[0092] In the embodiments of this disclosure, the convolution methods provided by the embodiments described above may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be clearly understood that the flow of the convolution methods described above includes multiple operations performed in a specific order, but the order of these operations is not limited. The convolution methods described above may be performed once or multiple times according to predetermined conditions.

[0093] The above explanation uses the convolution of the first layer of the residual network ResNet50 as an example, but this does not constitute a limitation to the embodiments of this disclosure. The convolution methods provided by the embodiments of this disclosure may be applied to any applicable convolution operation, and the size and number of channels of various data and various convolution kernels may be determined according to actual needs and are not limited to the specific values ​​above.

[0094] At least one embodiment of the present disclosure further provides a convolutional operation device. This convolutional operation device can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the time of convolution operations, improve computational efficiency, and save data transmission time.

[0095] Figure 13 is a schematic block diagram of a convolutional unit provided in some embodiments of the present disclosure. As shown in Figure 13, in some embodiments, the convolutional unit 100 includes a determination unit 110, an adjustment unit 120, and a calculation unit 130.

[0096] The determination unit 110 determines the convolution kernel for the operation. For example, the convolution kernel for the operation is obtained based on the initial convolution kernel, which is represented as [R,S,C,K], and the convolution kernel for the operation is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0. For example, the determination unit 110 may perform step S10 of the convolution operation method shown in Figure 2.

[0097] The adjustment unit 120 adjusts the arrangement of the input data based on the number of channels in the convolution kernel for calculation to obtain the target data. For example, the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels in the convolution kernel for calculation. For example, the adjustment unit 120 may perform step S20 of the convolution calculation method shown in Figure 2.

[0098] The calculation unit 130 performs a convolution operation based on the target data and the calculation convolution kernel, and obtains the convolution operation result. For example, the result of the convolution operation between the target data and the calculation convolution kernel is equal to the result of the convolution operation between the input data and the initial convolution kernel. For example, the calculation unit 130 may perform step S30 of the convolution operation method shown in Figure 2.

[0099] For example, the determination unit 110, the adjustment unit 120, and the calculation unit 130 may be hardware, software, firmware, or any possible combination thereof. For example, the determination unit 110, the adjustment unit 120, and the calculation unit 130 may be dedicated or general-purpose circuits, chips, or devices, or a combination of a processor and memory. The specific implementations of the determination unit 110, the adjustment unit 120, and the calculation unit 130 are not limited to those in the embodiments of this disclosure.

[0100] In the embodiments of this disclosure, each part of the convolution unit 100 corresponds to each step of the convolution method described above. Specific functions of the convolution unit 100 can be described in the related explanation of the convolution method described above, but will not be repeated here. The components and structure of the convolution unit 100 shown in Figure 13 are merely illustrative and not limiting; the convolution unit 100 may include other components and structures as needed.

[0101] At least one embodiment of the present disclosure further provides an electronic device that can improve the utilization rate of the matrix operation unit, effectively utilize the computing power of the matrix operation unit, shorten the time of convolution operations, improve computational efficiency, and save data transmission time.

[0102] Figure 14 is a schematic block diagram of an electronic device provided in some embodiments of the present disclosure. As shown in Figure 14, the electronic device 200 includes a convolutional unit 210. The convolutional unit 210 may be a convolutional unit provided in any embodiment of the present disclosure, for example, the convolutional unit 100 described above. The electronic device 200 may be any device with computing capabilities, such as a server, terminal equipment, or personal computer, but the embodiments of the present disclosure are not limited thereto.

[0103] Figure 15 is a schematic block diagram of another electronic device provided by an embodiment of the present disclosure. As shown in Figure 15, the electronic device 300 includes a processor 310 and a memory 320 for implementing a client or server. The memory 320 is used to non-temporarily store computer-executable instructions (e.g., at least one (or more) computer program modules). The processor 310 is configured to perform the computer-executable instructions, and when the computer-executable instructions are executed by the processor 310, one or more steps in the convolution method described above can be performed, thereby executing the convolution method described above. The memory 320 and the processor 310 may be interconnected by a bus system and / or other forms of connection mechanisms (not shown).

[0104] For example, the processor 310 may be a central processing unit (CPU), a graphics processing unit (GPU), or another form of processing unit having data processing and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture. The processor 310 may be a general-purpose processor or a dedicated processor capable of controlling other components in the electronic device 300 to perform desired functions.

[0105] For example, memory 320 may include any combination of at least one (e.g., one or more) computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. At least one (e.g., one or more) computer program modules may be stored in the computer-readable storage media, and the processor 310 may execute at least one (e.g., one or more) computer program modules to realize various functions of the electronic device 300. The computer-readable storage media may further store various application programs, various data, and various data used and / or generated by the application programs.

[0106] In the embodiments of this disclosure, the specific functions and technical effects of the electronic device 300 may be described above in reference to the description of the convolution method, but will not be repeated here.

[0107] Figure 16 is a schematic block diagram of another electronic device provided by an embodiment of the present disclosure. The electronic device 400 is suitable, for example, for performing a convolution operation method provided by an embodiment of the present disclosure. The electronic device 400 may be a terminal device for implementing a client or server. The electronic device 400 may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and wearable electronic devices, as well as fixed terminals such as digital TVs, desktop computers, and smart home devices. Note that the electronic device 400 shown in Figure 16 is merely an example and does not limit the functions and scope of use of the embodiments of the present disclosure.

[0108] As shown in Figure 16, the electronic device 400 may include a processing unit (e.g., a central processing unit, graphics processor, etc.) 410, which may perform various appropriate operations and processes according to a program stored in a read-only memory (ROM) 420 or a program loaded from a storage device 480 into a random access memory (RAM) 430. The RAM 430 further stores various programs and data necessary for the operation of the electronic device 400. The processing unit 410, ROM 420, and RAM 430 are interconnected via a bus 440. An edit / output (I / O) interface 450 is also connected to the bus 440.

[0109] Generally, the following devices can be connected to the I / O interface 450: for example, input devices 460 such as touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 470 such as liquid crystal displays (LCDs), speakers, and vibrators; storage devices 480 such as magnetic tape and hard disks; and communication devices 490. The communication devices 490 may allow the electronic device 400 to communicate wirelessly or wired with other electronic devices to exchange data. Figure 16 shows an electronic device 400 with various devices, but it should be understood that it is not necessary to implement or include all the devices shown, and instead, the electronic device 400 may implement or include more or fewer devices.

[0110] For example, according to embodiments of the present disclosure, the convolution method described above may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product which includes a computer program embedded in a non-temporary computer-readable medium, the computer program including program code for executing the convolution method described above. In such embodiments, the computer program may be downloaded and installed from a network via a communication device 490, or installed from a storage device 480, or from a ROM 420. When the computer program is executed by a processing unit 410, the functions defined by the convolution method provided by embodiments of the present disclosure may be executed.

[0111] At least one embodiment of the present disclosure further provides a storage medium. Using this storage medium, the utilization rate of the matrix operation unit can be improved, the computing power of the matrix operation unit can be effectively utilized, the time of convolution operations can be shortened, computation efficiency can be improved, and data transmission time can be saved.

[0112] Figure 17 is a schematic block diagram of a storage medium provided by some embodiments of the present disclosure. For example, as shown in Figure 17, the storage medium 500 may be a non-temporary computer-readable storage medium storing non-temporary computer-readable instructions 510. When the non-temporary computer-readable instructions 510 are executed by a processor, the convolution operation method described in the embodiments of the present disclosure can be realized. For example, when the non-temporary computer-readable instructions 510 are executed by a processor, one or more steps of the above convolution operation method can be performed.

[0113] For example, the storage medium 500 may be applied to the electronic device described above, and for example, the storage medium 500 may include the memory 320 in the electronic device 300.

[0114] For example, the storage medium may include a smartphone memory card, a tablet computer memory component, a personal computer hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage mediums, or other suitable storage medium.

[0115] For example, the description of the storage medium 500 may refer to the description of memory in the embodiment of the electronic device, and the same information will not be repeated. Regarding the specific functions and technical effects of the storage medium 500, the above description of the convolution operation method may refer, but it will not be repeated here.

[0116] The convolution operation method, convolution operation device, electronic device, and storage medium provided by the embodiments of this disclosure have been described above in conjunction with Figures 1 to 17. The convolution operation method provided by the embodiments of this disclosure may be used for the convolution operation of the first layer of a convolutional neural network. By adjusting the data arrangement scheme, the number of channels can be increased, and convolution operations can be performed between target data with more channels and a computational convolution kernel with more channels. This improves the utilization rate of the matrix operation unit, effectively utilizes the computing power of the matrix operation unit, shortens the convolution operation time, improves computation efficiency, and saves data transmission time.

[0117] In the context of this disclosure, a computer-readable medium may be a tangible medium that contains or can store a program used by or in combination with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this disclosure, a computer-readable medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signaling medium may include a data signal propagated in the baseband or propagated as part of a carrier wave, which incorporates computer-readable program code. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signaling medium may be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program used by or in combination with an instruction execution system, apparatus, or device.Program code embedded on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), or any suitable combination thereof.

[0118] In some embodiments, the client and server may communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect digital data communications (e.g., communication networks) in any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as currently known or future-developed networks.

[0119] The computer-readable medium described above may be included in the electronic device, or it may exist independently without being incorporated into the electronic device.

[0120] Computer program code for performing the operations of the Disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, run as a standalone software package, run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer via any type of network (local area network (LAN) or wide area network (WAN)) or to an external computer (e.g., an Internet connection via an Internet service provider).

[0121] The flowcharts and block diagrams in the drawings illustrate the feasible architectures, functionalities, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code containing one or more executable instructions for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions shown within a block may be executed in a different order than shown in the diagram. For example, two blocks shown successively may actually be executed substantially in parallel, or they may be executed in reverse order depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be realized by a dedicated hardware-based system that performs the specified function or operation, or by using a combination of special hardware and computer instructions.

[0122] The units relating to the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not necessarily limited to the module itself.

[0123] The functions described herein can be performed, at least in part, by one or more hardware logic components. For example, but not limited to, typical types of usable hardware logic components include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), and composite programmable logic devices (CPLDs).

[0124] According to one or more embodiments of the present disclosure, a convolution operation method includes the steps of: determining a convolution kernel for calculation, wherein the convolution kernel for calculation is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for calculation is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0; adjusting the arrangement of input data based on the number of channels of the convolution kernel for calculation to obtain target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the convolution kernel for calculation; and performing a convolution operation based on the target data and the convolution kernel for calculation to obtain the result of the convolution operation, wherein the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0125] According to one or more embodiments of the present disclosure, the number of channels in the target data is greater than the number of channels in the input data, and the number of channels in the computational convolution kernel is greater than the number of channels in the initial convolution kernel.

[0126] According to one or more embodiments of the present disclosure, the step of obtaining target data by adjusting the arrangement of the input data based on the number of channels of the computational convolution kernel includes the steps of storing the input data row by row in static memory, wherein each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0; performing a fill operation on the input data stored in the static memory to obtain augmented data; and adjusting the arrangement of the augmented data to change the size and number of channels of the augmented data to obtain target data.

[0127] According to one or more embodiments of the present disclosure, the step of storing the input data in the static memory row by row includes the step of storing the input data in the memory in a dense array, wherein the input data comprises a plurality of channels, and the dense array means that the plurality of channels of the same data point are stored sequentially adjacent in the memory; and the step of using direct memory access to transmit the input data in the memory to the static memory of a hardware accelerator, storing the first data point of each row of the input data in the first column of a different row of the static memory, thereby storing each row of the input data in corresponding N storage rows in the static memory.

[0128] According to one or more embodiments of the present disclosure, the step of performing the fill operation on the input data stored in the static memory to obtain the extended data includes the steps of: filling the storage rows before and after the storage location of the input data in the static memory with first preset values ​​to obtain first intermediate data including the input data and the filled first preset values; transmitting the first intermediate data to a vector calculation unit, and using the shift command and fill command of the vector calculation unit to fill second preset values ​​at both ends of each row corresponding to the first intermediate data to obtain second intermediate data including the first intermediate data and the filled second preset values; and transmitting the second intermediate data to a corresponding storage location in the static memory to obtain the extended data having the same content as the second intermediate data.

[0129] According to one or more embodiments of the present disclosure, the target data is represented as [1, ht, wt, (C × R × S)], where ht and wt are both integers greater than 0, and the step of acquiring the target data by adjusting the arrangement scheme of the extended data and changing the size and number of channels of the extended data is the step of successively reading the data of R × N storage rows in the static memory and transmitting it to the vector calculation unit, wherein the starting address read each time is moved str*N storage rows according to a preset hop number str, and the preset hop number is the convolution of the input data and the initial convolution kernel The function includes the steps of: ht being the number of hops in the row and column directions of the sliding window required for the calculation, and the total number of times data is read from the static memory being equal to ht; and the vector calculation unit converting the R*N data in the storage rows received each time into wt*ceil((C×R×S) / L) data in the storage rows to obtain the target data, where L represents the number of data points that can be stored in each storage row in the static memory, ceil((C×R×S) / L) represents rounding up to the nearest integer for (C×R×S) / L, and the converted data is the target data.

[0130] According to one or more embodiments of the present disclosure, the vector calculation unit converts the data in R*N storage rows received each time into data in wt*ceil((C×R×S) / L) storage rows, and the step of acquiring the target data includes: dividing the data in R*N storage rows into a plurality of groups according to a preset number of hops, wherein the data in each group corresponds to a sliding window in the row direction, and the number of data in the plurality of groups is equal to wt; determining an initial position information parameter and a target position information parameter for the data in each row in the sliding window corresponding to the data in that group for each group; and the vector calculation unit stores the data in each group in the converted array format according to the initial position information parameter and the target position information parameter at the corresponding position in the target memory, and acquiring the target data, wherein the target memory stores data row by row, is transmitted to the target memory, and the data stored on the target memory is the target data.

[0131] According to one or more embodiments of the present disclosure, the initial position information parameters include a first start boundary coordinate, a first end boundary coordinate, a first start address, a first end address, a first start number, and a first end number. The first start boundary coordinates represent the relative coordinates in the row direction of the extended data to the start boundary of the corresponding sliding window, the first end boundary coordinates represent the relative coordinates in the row direction of the extended data to the end boundary of the corresponding sliding window, the start boundary and the end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data, the first start address represents the address of the first start boundary coordinates in the memory (e.g., vmem) of the vector calculation unit, the first end address represents the address of the first end boundary coordinates in the memory of the vector calculation unit, the first start number represents the number of the data point corresponding to the first start address of the first start boundary coordinates, and the first end number represents the number of the data point corresponding to the first end address of the first end boundary coordinates.

[0132] According to one or more embodiments of the present disclosure, the formula for calculating the first start boundary coordinate is src_row_start_index = i * str * ch, where src_row_start_index represents the first start boundary coordinate, i represents the number of the corresponding data point in the corresponding sliding window in the size wt of the target data, str represents the number of hops in the row direction of the sliding window, and ch represents the number of channels of the input data; the formula for calculating the first end boundary coordinate is src_row_end_index = src_row_start_index + (kernel_w * ch - 1), where src_row_end_index represents the first end boundary coordinate, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel; and the formula for calculating the first start address is src_row_start_address = src_row_start_index The formula for calculating the first end address is x / vmem_lane+j*N, where src_row_start_address represents the first start address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, j represents the row number in the sliding window of the corresponding data, and the formula for calculating the first end address is src_row_end_address=src_row_end_index / vmem_lane+j*N, where src_row_end_addr ess represents the first end address, the formula for calculating the first start number is src_row_start_lane=src_row_start_index%vmem_lane, where src_row_start_lane represents the first start number, % represents the modulo operation, the formula for calculating the first end number is src_row_end_lane=src_row_end_index%vmem_lane, where src_row_end_lane represents the first end number.

[0133] According to one or more embodiments of the present disclosure, the target position information parameter includes a second start boundary coordinate, a second end boundary coordinate, a second start address, a second end address, a second start number, and a second end number. The second start boundary coordinates represent the relative coordinates of the start boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], the second end boundary coordinates represent the relative coordinates of the end boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], the start boundary of the corresponding sliding window and the end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data, the second start address represents the address of the second start boundary coordinates in the memory of the vector calculation unit, the second end address represents the address of the second end boundary coordinates in the memory of the vector calculation unit, the second start number represents the data point number corresponding to the second start address of the second start boundary coordinates, and the second end number represents the data point number corresponding to the second end address of the second end boundary coordinates.

[0134] According to one or more embodiments of the present disclosure, the formula for calculating the second start boundary coordinate is dst_row_start_index = j * kernel_w * ch, where dst_row_start_index represents the second start boundary coordinate, j represents the row number in the sliding window of the corresponding data, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels of the input data; the formula for calculating the second end boundary coordinate is dst_row_end_index = dst_row_start_index + (kernel_w * ch - 1), where dst_row_end_index represents the second end boundary coordinate; and the formula for calculating the second start address is dst_row_start_address = dst_row_start_index / vme m_lane is where dst_row_start_address represents the second start address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, the formula for calculating the second end address is dst_row_end_address=dst_row_end_index / vmem_lane, where dst_row_end_address represents the second end address, the formula for calculating the second start number is dst_row_start_lane=dst_row_start_index%vmem_lane, where dst_row_start_lane represents the second start number, where % represents modulo operation, and the formula for calculating the second end number is dst_row_end_lane=dst_row_end_index%vmem_lane, where dst_row_end_lane represents the second end number.

[0135] According to one or more embodiments of the present disclosure, the vector calculation unit stores the data of each group in a converted array format at the corresponding location in the target memory according to the initial position information parameter and the target position information parameter, and the step of acquiring the target data includes the step of the vector calculation unit concatenating the data of each group in a converted array format in accordance with the initial position information parameter and the target position information parameter, using a cyclic shift instruction and according to a preset enable signal in a predicate register, storing it at the corresponding location in the target memory, and acquiring the target data.

[0136] According to one or more embodiments of this disclosure, the step of performing a convolution operation based on the target data and the computational convolution kernel is: The process includes the step of performing a convolution operation on the target data and the convolution kernel for the operation using a matrix operation unit.

[0137] According to one or more embodiments of the present disclosure, the convolution method is used for the convolution of the first layer of a convolutional neural network.

[0138] According to one or more embodiments of the present disclosure, a convolution arithmetic unit includes: a determination unit for determining a convolution kernel for calculation, wherein the convolution kernel for calculation is obtained based on an initial convolution kernel, the initial convolution kernel is represented by [R,S,C,K], and the convolution kernel for calculation is represented by [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0; an adjustment unit for adjusting the arrangement of input data based on the number of channels of the convolution kernel for calculation to acquire target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the convolution kernel for calculation; and a calculation unit for performing a convolution operation based on the target data and the convolution kernel for calculation and acquiring the result of the convolution operation, wherein the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel.

[0139] According to one or more embodiments of the present disclosure, the number of channels in the target data is greater than the number of channels in the input data, and the number of channels in the computational convolution kernel is greater than the number of channels in the initial convolution kernel.

[0140] According to one or more embodiments of the present disclosure, the adjustment unit includes a first sub-adjustment unit, a second sub-adjustment unit, and a third sub-adjustment unit. The first sub-adjustment unit is configured to store the input data in static memory row by row, where each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0. The second sub-adjustment unit performs a fill operation on the input data stored in static memory to obtain augmented data. The third sub-adjustment unit adjusts the arrangement of the augmented data and changes the size and number of channels of the augmented data to obtain the target data.

[0141] According to one or more embodiments of the present disclosure, the first sub-adjustment unit includes a first storage unit and a second storage unit. The first storage unit is configured to store the input data in a dense array in memory, the input data comprising a plurality of channels, where the dense array means that the plurality of channels of the same data point are stored sequentially adjacent in the memory. The second storage unit transmits the input data in the memory to the static memory of a hardware accelerator using direct memory access, storing the first data point of each row of the input data in the first column of a different row in the static memory, so that each row of the input data is stored in corresponding N storage rows in the static memory.

[0142] According to one or more embodiments of the present disclosure, the second sub-adjustment unit includes a first fill unit, a second fill unit, and a third fill unit. The first fill unit fills the storage rows before and after the storage location of the input data in the static memory with first preset values ​​to acquire first intermediate data. Here, the first intermediate data includes the input data and the filled first preset values. The second fill unit is configured to transmit the first intermediate data to a vector calculation unit and, using the shift and fill commands of the vector calculation unit, fill second preset values ​​at both ends of each row corresponding to the first intermediate data to acquire second intermediate data. Here, the second intermediate data includes the first intermediate data and the filled second preset values. The third fill unit transmits the second intermediate data to the corresponding storage location in the static memory to acquire extended data. The extended data has the same content as the second intermediate data.

[0143] According to one or more embodiments of the present disclosure, the target data is represented as [1, ht, wt, (C × R × S)], where ht and wt are both integers greater than 0. The third sub-adjustment unit includes a first modification unit and a second modification unit. The first modification unit is configured to successively read data from R × N storage rows in the static memory and transmit it to the vector calculation unit, wherein the starting address read each time is moved by str * N storage rows according to a preset number of hops str, where the preset number of hops is the number of hops in the row and column directions of the sliding window required for the convolution operation between the input data and the initial convolution kernel, and the total number of times data is read from the static memory is equal to ht. The second modification unit is configured to convert the R*N data of each received memory row into wt*ceil((C×R×S) / L) data of memory rows, thereby obtaining the target data, where L represents the number of data points that can be stored in each memory row in the static memory, and ceil((C×R×S) / L) represents rounding up (C×R×S) / L to the nearest integer, and the converted data is the target data.

[0144] According to one or more embodiments of the present disclosure, the second modification includes a grouping unit, a parameter determination unit, and a vector calculation unit. The grouping unit is configured to divide the data in R*N storage rows into a plurality of groups according to the preset number of hops, where the data in each group corresponds to a sliding window in the row direction, and the number of data in the plurality of groups is equal to wt. The parameter determination unit is configured to determine, for the data in each group, an initial position information parameter and a target position information parameter for the data in each row in the sliding window corresponding to the data in that group. The vector calculation unit is configured to store the data in each group in a converted array format according to the initial position information parameter and the target position information parameter at the corresponding position in the target memory, and to acquire the target data, where the target memory is stored row by row, transmitted to the target memory, and the data stored on the target memory is the target data.

[0145] According to one or more embodiments of the present disclosure, the initial position information parameter includes a first start boundary coordinate, a first end boundary coordinate, a first start address, a first end address, a first start number, and a first end number. The first start boundary coordinate represents the relative coordinate of the start boundary of a corresponding sliding window in the row direction of the extended data, the first end boundary coordinate represents the relative coordinate of the end boundary of a corresponding sliding window in the row direction of the extended data, the start boundary and end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data, the first start address represents the address of the first start boundary coordinate in the memory of the vector calculation unit, the first end address represents the address of the first end boundary coordinate in the memory of the vector calculation unit, the first start number represents the number of the data point corresponding to the first start address of the first start boundary coordinate, and the first end number represents the number of the data point corresponding to the first end address of the first end boundary coordinate.

[0146] According to one or more embodiments of the present disclosure, the formula for calculating the first start boundary coordinate is src_row_start_index = i * str * ch, where src_row_start_index represents the first start boundary coordinate, i represents the number of the corresponding data point in the corresponding sliding window in the size wt of the target data, str represents the number of hops in the row direction of the sliding window, and ch represents the number of channels of the input data; the formula for calculating the first end boundary coordinate is src_row_end_index = src_row_start_index + (kernel_w * ch - 1), where src_row_end_index represents the first end boundary coordinate, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel; and the formula for calculating the first start address is src_row_start_address = src_row_start_ind ex / vmem_lane+j*N, where src_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, j represents the row number in the sliding window of the corresponding data, and the formula for calculating the first ending address is src_row_end_address=src_row_end_index / vmem_lane+j*N, and src_row_end_addr ess represents the first end address, the formula for calculating the first start number is src_row_start_lane=src_row_start_index%vmem_lane, where src_row_start_lane represents the first start number, % represents the modulo operation, the formula for calculating the first end number is src_row_end_lane=src_row_end_index%vmem_lane, where src_row_end_lane represents the first end number.

[0147] According to one or more embodiments of the present disclosure, the target position information parameter includes a second start boundary coordinate, a second end boundary coordinate, a second start address, a second end address, a second start number, and a second end number. The second start boundary coordinates represent the relative coordinates of the start boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], the second end boundary coordinates represent the relative coordinates of the end boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], the start boundary of the corresponding sliding window and the end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data, the second start address represents the address of the second start boundary coordinates in the memory of the vector calculation unit, the second end address represents the address of the second end boundary coordinates in the memory of the vector calculation unit, the second start number represents the data point number corresponding to the second start address of the second start boundary coordinates, and the second end number represents the data point number corresponding to the second end address of the second end boundary coordinates.

[0148] According to one or more embodiments of the present disclosure, the formula for calculating the second start boundary coordinate is dst_row_start_index = j * kernel_w * ch, where dst_row_start_index represents the second end boundary coordinate, j represents the row number in the sliding window of the corresponding data, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels of the input data. The formula for calculating the second end boundary coordinate is dst_row_end_index = dst_row_start_index + (kernel_w * ch - 1), where dst_row_end_index represents the second end boundary coordinate, and the formula for calculating the second start address is dst_row_start_address = dst_row_start_index / vme m_lane is where dst_row_start_address represents the second start address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, the formula for calculating the second end address is dst_row_end_address=dst_row_end_index / vmem_lane, where dst_row_end_address represents the second end address, the formula for calculating the second start number is dst_row_start_lane=dst_row_start_index%vmem_lane, where dst_row_start_lane represents the second start number, where % represents modulo operation, and the formula for calculating the second end number is dst_row_end_lane=dst_row_end_index%vmem_lane, where dst_row_end_lane represents the second end number.

[0149] According to one or more embodiments of the present disclosure, the vector calculation unit is further configured to acquire the target data by concatenating the data of each group in a converted array scheme in accordance with the initial position information parameter and the target position information parameter, using a cyclic shift instruction and according to a preset enable signal in a predicate register, storing it in the corresponding position in the target memory.

[0150] According to one or more embodiments of the present disclosure, the calculation unit includes the sub-calculation unit, which is configured to perform a convolution operation on the target data and the convolution kernel using a matrix operation unit.

[0151] According to one or more embodiments of the present disclosure, the convolutional unit is used for the convolution of the first layer of a convolutional neural network.

[0152] According to one or more embodiments of the present disclosure, the electronic device includes a convolutional unit provided by any embodiment of the present disclosure.

[0153] According to one or more embodiments of the present disclosure, an electronic device includes a processor and a memory containing at least one computer program module, the at least one computer program module being stored in the memory and configured to be executed by the processor, and the at least one computer program module being used to implement a convolution operation method provided by any embodiment of the present disclosure.

[0154] According to one or more embodiments of the present disclosure, a storage medium storing non-temporary computer-readable instructions is provided, and when the non-temporary computer-readable instructions are executed by a computer, a convolution operation method provided by any embodiment of the present disclosure is realized.

[0155] The above description is merely an explanation of more preferred embodiments and applicable technical principles of the present disclosure. Those skilled in the art will understand that the scope of the disclosure is not limited to technical means consisting of specific combinations of the above technical features, but should also cover other technical means formed by any combination of the above technical features or their equivalents, without departing from the concepts disclosed above. For example, technical means formed by substituting the above features with similar functional technical features disclosed in this disclosure (but not limited to these) should also be covered.

[0156] Furthermore, while various operations are presented in a specific order, this should not be understood as requiring these operations to be performed in a specific order or sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above description, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable subcombination in multiple embodiments.

[0157] While this subject matter has been described in language specific to structural features and / or the logical operation of methods, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely illustrative forms that realize the claims.

[0158] There are still a few points that need to be explained regarding this disclosure. (1) The drawings of the embodiments of the present disclosure include only structures relevant to the embodiments of the present disclosure, and common designs may be referenced for other structures. (2) The embodiments and features of the embodiments described herein can be combined with each other to obtain new embodiments, provided they do not contradict each other.

[0159] The above are merely specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto, and the scope of protection of the present disclosure should be subject to the scope of protection of the claims.

Claims

1. A method for a convolution operation performed by an electronic device, the method comprising: A step of determining a convolution kernel for calculation, wherein the convolution kernel for calculation is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for calculation is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0. A step of obtaining target data by adjusting the arrangement of input data based on the number of channels of the aforementioned convolution kernel for calculation, wherein the size and number of channels of the target data are different from the size and number of channels of the aforementioned input data, and the number of channels of the target data is equal to the number of channels of the aforementioned convolution kernel for calculation; A step of performing a convolution operation based on the target data and the convolution kernel for calculation, and obtaining the result of the convolution operation, wherein the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel. method.

2. The method according to claim 1, wherein the number of channels in the target data is greater than the number of channels in the input data, and the number of channels in the computational convolution kernel is greater than the number of channels in the initial convolution kernel.

3. The step of adjusting the array of input data based on the number of channels of the aforementioned convolution kernel for calculation to obtain target data is: A step of storing the input data in static memory row by row, wherein each row of the input data is stored in corresponding N storage rows in the static memory, where N is an integer greater than 0. The steps include: performing a fill operation on the input data stored in the static memory to obtain expanded data; The method according to claim 1, comprising the steps of adjusting the arrangement of the extended data, changing the size and number of channels of the extended data, and obtaining the target data.

4. The step of storing the aforementioned input data in the static memory row by row is: A step of storing the input data in memory in a dense arrangement, wherein the input data includes multiple channels, and the dense arrangement means that multiple channels of the same data point are stored sequentially adjacent to each other in the memory. The method according to claim 3, comprising the steps of: transmitting input data in the memory to the static memory of a hardware accelerator using direct memory access; storing the first data point of each row of the input data in the first column of a different row of the static memory; and storing each row of the input data in corresponding N storage rows in the static memory.

5. The step of performing the filling operation on the input data stored in the static memory and obtaining the expanded data is: The steps include: filling the storage rows before and after the storage location of the input data in the static memory with first preset values, and obtaining first intermediate data that includes the input data and the filled first preset values; The steps include: transmitting the first intermediate data to a vector calculation unit; using the shift command and fill command of the vector calculation unit, filling the ends of each row corresponding to the first intermediate data with second preset values; and obtaining second intermediate data including the first intermediate data and the filled second preset values. The method according to claim 3, comprising the step of transmitting the second intermediate data to a corresponding storage location in the static memory to obtain extended data having the same content as the second intermediate data.

6. The aforementioned target data is represented as [1, ht, wt, (C × R × S)], where ht and wt are both integers greater than 0. The step of adjusting the arrangement method of the extended data, changing the size and number of channels of the extended data, and obtaining the target data is: A step comprising reading data from R*N storage rows in the static memory one after another and transmitting it to the vector calculation unit, wherein the starting address read each time is such that str*N storage rows are moved according to a preset number of hops str, the preset number of hops str is the number of hops in the row and column directions of the sliding window required for the convolution operation between the input data and the initial convolution kernel, and the total number of times data is read from the static memory is equal to ht, The method according to claim 5, comprising the step of converting the data of R*N memory rows received each time into data of wt*ceil((C×R×S) / L) memory rows using the vector calculation unit to obtain the target data, wherein L represents the number of data points that can be stored in each memory row in the static memory, ceil((C×R×S) / L) represents rounding up to the nearest integer for (C×R×S) / L, and the converted data is the target data.

7. The step in which the vector calculation unit converts the R*N data of each received storage row into wt*ceil((C×R×S) / L) data of storage rows and obtains the target data is as follows: A step of dividing data in R*N storage rows into multiple groups according to a predetermined number of hops, wherein the data in each group corresponds to a sliding window in the row direction, and the number of data in the multiple groups is equal to wt. For each group of data, the steps include determining the initial position information parameters and target position information parameters for each row of data in the sliding window corresponding to the data of that group, The method according to claim 6, comprising the steps of: the vector calculation unit storing data for each group in a converted array format according to the initial position information parameter and the target position information parameter at corresponding positions in the target memory, and acquiring the target data, wherein the target memory stores data row by row, and the data stored in the target memory is the target data.

8. The initial position information parameters include a first start boundary coordinate, a first end boundary coordinate, a first start address, a first end address, a first start number, and a first end number. The first start boundary coordinates represent the relative coordinates of the start boundary of the corresponding sliding window in the row direction of the extended data, and the first end boundary coordinates represent the relative coordinates of the end boundary of the corresponding sliding window in the row direction of the extended data, and the start boundary and end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data. The first start address represents the address of the first start boundary coordinates in the memory of the vector calculation unit, and the first end address represents the address of the first end boundary coordinates in the memory of the vector calculation unit. The method according to claim 7, wherein the first start number represents the number of a data point corresponding to the first start address of the first start boundary coordinates, and the first end number represents the number of a data point corresponding to the first end address of the first end boundary coordinates.

9. The formula for calculating the first starting boundary coordinates is src_row_start_index = i * str * ch, where src_row_start_index represents the first starting boundary coordinates, i represents the number of the corresponding data point in the corresponding sliding window with a target data size wt, str represents the number of hops in the row direction of the sliding window, and ch represents the number of channels in the input data. The formula for calculating the first end boundary coordinates is src_row_end_index = src_row_start_index + (kernel_w * ch - 1), where src_row_end_index represents the first end boundary coordinates, kernel_w represents the width of the sliding window, and the size of the sliding window is equal to the size of the initial convolution kernel. The formula for calculating the first starting address is src_row_start_address = src_row_start_index / vmem_lane + j * N, where src_row_start_address represents the first starting address, vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit, and j represents the row number in the sliding window of the corresponding data. The formula for calculating the first end address is src_row_end_address = src_row_end_index / vmem_lane + j*N, where src_row_end_address represents the first end address. The formula for calculating the first starting number is src_row_start_lane = src_row_start_index % vmem_lane, where src_row_start_lane represents the first starting number and % represents the modulo operation. The method according to claim 8, wherein the formula for calculating the first end number is src_row_end_lane = src_row_end_index % vmem_lane, and src_row_end_lane represents the first end number.

10. The aforementioned target position information parameters include a second start boundary coordinate, a second end boundary coordinate, a second start address, a second end address, a second start number, and a second end number. The second start boundary coordinates represent the relative coordinates of the start boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], and the second end boundary coordinates represent the relative coordinates of the end boundary of the corresponding sliding window within the data size [1,1,(C×R×S)], and the start boundary and end boundary of the corresponding sliding window are located at different positions in the row direction of the extended data. The second start address represents the address of the second start boundary coordinates in the memory of the vector calculation unit, and the second end address represents the address of the second end boundary coordinates in the memory of the vector calculation unit. The method according to claim 7, wherein the second start number represents the number of a data point corresponding to the second start address of the second start boundary coordinates, and the second end number represents the number of a data point corresponding to the second end address of the second end boundary coordinates.

11. The formula for calculating the second starting boundary coordinates is dst_row_start_index = j * kernel_w * ch, where dst_row_start_index represents the second starting boundary coordinates, j represents the row number in the sliding window of the corresponding data, kernel_w represents the width of the sliding window, the size of the sliding window is equal to the size of the initial convolution kernel, and ch represents the number of channels in the input data. The formula for calculating the second end boundary coordinate is dst_row_end_index = dst_row_start_index + (kernel_w * ch - 1), where dst_row_end_index represents the second end boundary coordinate. The formula for calculating the second starting address is dst_row_start_address = dst_row_start_index / vmem_lane, where dst_row_start_address represents the second starting address and vmem_lane represents the number of data points that can be stored in each storage row in the memory of the vector calculation unit. The formula for calculating the second end address is dst_row_end_address = dst_row_end_index / vmem_lane, where dst_row_end_address represents the second end address. The formula for calculating the second starting number is dst_row_start_lane = dst_row_start_index % vmem_lane, where dst_row_start_lane represents the second starting number and % represents the modulo operation. The method according to claim 10, wherein the formula for calculating the second end number is dst_row_end_lane = dst_row_end_index % vmem_lane, and dst_row_end_lane represents the second end number.

12. The vector calculation unit stores the data of each group in the corresponding location in the target memory in a converted array format according to the initial position information parameter and the target position information parameter, and the step of acquiring the target data is as follows: The method according to claim 7, wherein the vector calculation unit, according to the initial position information parameter and the target position information parameter, uses a cyclic shift instruction to concatenate the data of each group in a converted array format according to a preset enable signal in a predicate register, stores it in the corresponding position in the target memory, and acquires the target data.

13. The step of performing a convolution operation based on the target data and the convolution kernel for the operation is as follows: The method according to claim 1, further comprising the step of performing a convolution operation on the target data and the convolution kernel for the operation using a matrix operation unit.

14. The method according to claim 1, wherein the convolution operation method is used for the convolution operation of the first layer of a convolutional neural network.

15. A device for convolution operations, said device, A determination unit for determining a convolution kernel for calculations, wherein the convolution kernel for calculations is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for calculations is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0. An adjustment unit that adjusts the arrangement of input data based on the number of channels of the aforementioned convolution kernel for calculation to acquire target data, wherein the size and number of channels of the target data are different from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels of the aforementioned convolution kernel for calculation, A calculation unit that performs a convolution operation based on the target data and the calculation convolution kernel and obtains the result of the convolution operation, wherein the result of the convolution operation between the target data and the calculation convolution kernel is equal to the result of the convolution operation between the input data and the initial convolution kernel, Device.

16. Electronic device including the apparatus described in claim 15.

17. Electronic equipment, said electronic equipment Processor and A memory containing at least one computer program module, and includes, The at least one computer program module is stored in the memory and, when executed by the processor, the processor... The process involves determining a convolution kernel for calculations, wherein the convolution kernel for calculations is obtained based on an initial convolution kernel, the initial convolution kernel is represented as [R,S,C,K], and the convolution kernel for calculations is represented as [1,1,(C×R×S),K], where R, S, C, and K are all integers greater than 0. The process involves adjusting the arrangement of the input data based on the number of channels in the aforementioned convolution kernel to obtain target data, wherein the size and number of channels of the target data differ from the size and number of channels of the input data, and the number of channels of the target data is equal to the number of channels in the aforementioned convolution kernel. The process involves performing a convolution operation based on the target data and the convolution kernel for calculation, and obtaining the result of the convolution operation, wherein the result of the convolution operation between the target data and the convolution kernel for calculation is equal to the result of the convolution operation between the input data and the initial convolution kernel. electronic equipment.

18. A storage medium for storing non-temporary computer-readable instructions, A storage medium in which, when the non-temporary computer-readable instruction is executed by a computer, the method according to any one of claims 1 to 14 is performed.

Images