Convolution operation method and apparatus

Abstract

The application discloses a convolution operation method and device. The specific implementation scheme is as follows: the method comprises the following steps: configuring a split parameter; inputting the split parameter, the parameter of the convolution operation and the parameter of the convolution acceleration device into a cost function to obtain the optimal value of the split parameter under the condition of meeting the constraint condition; adopting the optimal value of the split parameter to split input image data and weight data used for inputting the convolution acceleration device into a plurality of input data blocks and a plurality of weight data blocks; and controlling the plurality of input data blocks and the plurality of weight data blocks to be input into an internal local buffer, so that an operation device obtains the input data blocks and the weight data blocks required by the convolution operation from the internal local buffer, and performs the convolution operation on the input data blocks and the weight data blocks to obtain a plurality of output data blocks. The average utilization rate of the convolution acceleration device is improved, the time of the convolution operation is effectively reduced, and the balance between the utilization rate and the bandwidth demand of the convolution acceleration device is realized.

Description

Convolution operation method and device Technical Field

[0001] This application relates to the field of deep learning, and more particularly to the field of convolution operations. Background Technology

[0002] A convolution accelerator is hardware used to accelerate convolution calculations. Convolution accelerators typically fetch data from external memory. Although external memory has a large capacity, its access speed is slow. Therefore, to reduce the bandwidth requirements of external memory, a small-capacity internal local buffer (GLB) is set up inside the convolution accelerator. The GLB can be directly accessed by the arithmetic units within the convolution accelerator.

[0003] However, when performing convolution operations on large image data, the limited capacity of the GLB in the convolution accelerator necessitates splitting the large image data into data blocks, which are then input into the GLB. The GLB then sends these data blocks to the processor for convolution. However, existing splitting methods divide large image data into fixed-size blocks, and the bandwidth between external memory and the GLB cannot keep up with the computation speed, resulting in insufficient average utilization of the convolution accelerator and long computation times. Summary of the Invention

[0004] This application provides a convolution operation method and apparatus to solve the problems existing in related technologies. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a convolution operation method applied to a convolution acceleration device, the convolution acceleration device including an internal local cache and an arithmetic unit, comprising:

[0006] Configure the splitting parameters, which are derived from the data blocks that the internal local cache can store and the data blocks that the arithmetic unit can process.

[0007] The splitting parameters, convolution operation parameters, and convolution acceleration device parameters are input into the cost function, and the optimal value of the splitting parameters is obtained under the constraints.

[0008] By using the optimal value of the splitting parameter, the input image data and weight data used as input to the convolution accelerator are split into multiple input data blocks and multiple weight data blocks;

[0009] Multiple input data blocks and multiple weight data blocks are controlled to be input into an internal local buffer, so that the arithmetic unit can obtain the input data blocks and weight data blocks required for convolution operation from the internal local buffer, and perform convolution operation on them to obtain multiple output data blocks.

[0010] In one implementation, the internal local cache can simultaneously store an input data block, a weight data block, and an output data block.

[0011] In one implementation, it further includes:

[0012] Obtain the parameters for the convolution operation, including the height of the convolution kernel. and the width of the convolution kernel The number of batches N of input image data and the number of channels of input image data. Width of input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data, the height of the output image data The output image data includes output data blocks;

[0013] Obtain the parameters of the convolution accelerator, which include: the storage space of the internal local cache. The height of the arithmetic unit array in the arithmetic module and width Number of registers in the arithmetic unit .

[0014] In one implementation, configuring splitting parameters includes:

[0015] Configure the internal local buffer to store the required batch size n of input data blocks and the number of channels for the input data blocks. , height of input data block Number of channels in the output data block Height of output data block Where the number of batches n of the input data blocks is less than or equal to the number of batches N of the input image data;

[0016] Configure the number of channels for each arithmetic module in the arithmetic unit to calculate the required input data blocks. and the number of channels for output data blocks .

[0017] In one implementation, the splitting parameters, convolution operation parameters, and convolution acceleration device parameters are input into the cost function. Under certain constraints, the optimal value of the splitting parameters is obtained, including:

[0018] Based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device, the values ​​of the input image data, the values ​​of the weight data, and the average utilization of the processor are calculated.

[0019] The constraints are determined based on the parameters of the convolution accelerator and the splitting parameters.

[0020] The sum of the input image data values ​​and the weight data values, and the ratio of the average utilization of the processor, are input into the cost function. Under the condition that the constraints are met, the optimal values ​​of the splitting parameters are obtained.

[0021] In one implementation, the value of the input image data is calculated based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device, including:

[0022] Calculate the number of channels in the input data block , height of input data block and the width of the input image data The product of these two values ​​gives the amount of data that an input data block is loaded from external memory into the internal local cache. ;

[0023] Calculate the number of channel splits in the input image data. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from external memory to the internal local cache. = ;

[0024] Calculate the number of transmissions and The product of these terms yields the value of the input image data. .

[0025] In one implementation, the value of the weight data is calculated based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device, including:

[0026] height of convolution kernel width of the convolution kernel The arithmetic module calculates the number of channels required for the input data blocks. and the number of channels in the output data block The product of these values ​​yields the value of the weighted data block. ;

[0027] Calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q;

[0028] Calculate the number of channels required for the internal local buffer to store the output data blocks. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q';

[0029] Calculate the ratio Sum and ratio The product of these two operations, to obtain the output data of the internal local buffer, requires computation. The number of operations b = ;

[0030] Calculate the number of transmissions Number of operations and the values ​​of the weighted data blocks The product of these factors yields the value of the weighted data. .

[0031] In one implementation, the average utilization of the arithmetic unit is calculated based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device, including:

[0032] Based on the height of the computing unit array and width height of the convolution kernel , operation Calculate the number of channels required for the input data block The average utilization rate of the arithmetic unit .

[0033] In one implementation, determining constraints based on parameters of the convolution accelerator and splitting parameters includes:

[0034] Input data block value The value of the weighted data block and the values ​​of the output data blocks The sum is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely ;

[0035] The arithmetic module calculates the number of channels required for the output data blocks. Less than or equal to the width of the arithmetic unit array The arithmetic module calculates the number of channels required for the input data block. With the height of the convolution kernel The product is less than or equal to the height of the arithmetic unit array. Thus, the second constraint condition is obtained, namely ;

[0036] width of the convolution kernel Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely ;

[0037] The number of channels for storing the required input data blocks in the internal local buffer. The number of channels in the input image data is less than or equal to the number of channels in the input image data. Thus, the fourth constraint condition is obtained, namely ;

[0038] The arithmetic module calculates the number of channels required for the output data blocks. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely .

[0039] In one implementation, the ratio of the sum of the input image data values ​​and the weight data values ​​to the average utilization of the processor is input into the cost function. Under certain constraints, the optimal values ​​of the splitting parameters are obtained, including:

[0040] The sum of the input image data values ​​and the weight data values, divided by the average utilization rate of the processor, is input into the cost function to obtain... ;

[0041] The optimal value of the splitting parameter is obtained when all five constraints are satisfied simultaneously.

[0042] In one implementation, multiple input data blocks and multiple weight data blocks are sequentially input into an internal local buffer, so that the arithmetic unit retrieves the input data blocks and weight data blocks required for the convolution operation from the internal local buffer, performs convolution operation on them, and obtains multiple output data blocks, including:

[0043] Control the internal local buffer to send an input data block and A weighted data block is fed to the arithmetic unit, so that the arithmetic unit can adjust the weighted data block according to an input data block and One weighted data block generates one intermediate data block;

[0044] Execute in a loop The next control internal local buffer sends an input data block and The steps of sending weighted data blocks to the processor Summing the intermediate data blocks yields a single output data block.

[0045] Execute in a loop Following the steps described above to obtain an output data block, we get One output data block.

[0046] In one implementation, the arithmetic unit calculates based on an input data block and... One weighted data block generates an intermediate data block, including:

[0047] Each arithmetic unit executes the following steps: Based on... Given one input data block and one weight data block, we obtain... One intermediate data block, Less than 1 ;

[0048] row operations Performing the above steps yields an intermediate data block.

[0049] Secondly, embodiments of this application provide a convolution operation apparatus, including:

[0050] The splitting parameter configuration module is used to configure splitting parameters, which are obtained based on the data blocks that the internal local cache can store and the data blocks that the arithmetic unit can process.

[0051] The optimal value calculation module for splitting parameters is used to input the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device into the cost function, and obtain the optimal value of the splitting parameters under the constraints.

[0052] The data splitting module is used to split the input image data and weight data used to input the convolution accelerator into multiple input data blocks and multiple weight data blocks by using the optimal value of the splitting parameters.

[0053] The convolution operation module is used to control the input of multiple input data blocks and multiple weight data blocks into the internal local buffer, so that the operator can obtain the input data blocks and weight data blocks required for the convolution operation from the internal local buffer, and perform convolution operation on them to obtain multiple output data blocks.

[0054] In one implementation, the internal local cache can simultaneously store an input data block, a weight data block, and an output data block.

[0055] In one implementation, it further includes:

[0056] The first acquisition module is used to acquire the parameters for the convolution operation, including the height of the convolution kernel. and the width of the convolution kernel The number of batches N of input image data and the number of channels of input image data. Width of input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data, the height of the output image data The output image data includes output data blocks;

[0057] The second acquisition module is used to acquire the parameters of the convolution accelerator, including the storage space of the internal local cache. The height of the arithmetic unit array in the arithmetic module and width Number of registers in the arithmetic unit .

[0058] In one implementation, the parameter configuration module is split, including:

[0059] The first configuration submodule is used to configure the number of batches n of the required input data blocks and the number of channels of the input data blocks stored in the internal local buffer. , height of input data block Number of channels in the output data block Height of output data block Where the number of batches n of the input data blocks is less than or equal to the number of batches N of the input image data;

[0060] The second configuration submodule is used to configure the number of channels for the input data blocks required for calculation by each arithmetic module in the arithmetic unit. and the number of channels for output data blocks .

[0061] In one implementation, the parameter optimal value calculation module is split, including:

[0062] The first calculation submodule is used to calculate the value of the input image data, the value of the weight data, and the average utilization of the processor based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device.

[0063] The constraint determination submodule is used to determine the constraints based on the parameters of the convolution accelerator and the splitting parameters.

[0064] The second calculation submodule is used to input the sum of the image data values ​​and the weight data values, and the ratio of the average utilization rate of the processor into the cost function. Under the condition that the constraints are met, the optimal value of the splitting parameters is obtained.

[0065] In one implementation, the first computing submodule includes:

[0066] The first calculation unit is used to calculate the number of channels in the input data block. , height of input data block and the width of the input image data The product of these two values ​​gives the amount of data that an input data block is loaded from external memory into the internal local cache. ;

[0067] The second calculation unit is used to calculate the number of times the input image data is split into channels. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from external memory to the internal local cache. = ;

[0068] The third calculation unit is used to calculate the number of transmissions. and The product of these terms yields the value of the input image data. .

[0069] In one implementation, the first computing submodule further includes:

[0070] The fourth computational unit is used for the height of the convolution kernel. width of the convolution kernel The arithmetic module calculates the number of channels required for the input data blocks. and the number of channels in the output data block The product of these values ​​yields the value of the weighted data block. ;

[0071] The fifth calculation unit is used to calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q;

[0072] The sixth calculation unit is used to calculate the number of channels required for storing the output data blocks in the internal local buffer. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q';

[0073] The seventh calculation unit is used to calculate the ratio. Sum and ratio The product of these two operations, to obtain the output data of the internal local buffer, requires computation. The number of operations b = ;

[0074] The eighth calculation unit is used to calculate the number of transmissions. Number of operations and the values ​​of the weighted data blocks The product of these factors yields the value of the weighted data. .

[0075] In one implementation, the first computing submodule further includes:

[0076] The ninth computing unit is used to determine the height of the arithmetic unit array. and width height of the convolution kernel , operation Calculate the number of channels required for the input data block The average utilization rate of the arithmetic unit .

[0077] In one implementation, the constraint determination submodule includes:

[0078] The first constraint calculation unit is used to input the values ​​of the data block. The value of the weighted data block and the values ​​of the output data blocks The sum is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely ;

[0079] The second constraint calculation unit is used by the arithmetic module to calculate the number of channels required for the output data block. Less than or equal to the width of the arithmetic unit array The arithmetic module calculates the number of channels required for the input data block. With the height of the convolution kernel The product is less than or equal to the height of the arithmetic unit array. Thus, the second constraint condition is obtained, namely ;

[0080] The third constraint calculation unit is used for the width of the convolution kernel. Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely ;

[0081] The fourth constraint calculation unit is used to store the number of channels of the required input data blocks in the internal local buffer. The number of channels in the input image data is less than or equal to the number of channels in the input image data. Thus, the fourth constraint condition is obtained, namely ;

[0082] The fifth constraint calculation unit is used by the arithmetic module to calculate the number of channels required for the output data block. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely .

[0083] In one implementation, the second computing submodule includes:

[0084] The tenth calculation unit is used to input the sum of the image data values ​​and the weight data values, and the ratio of this sum to the average utilization rate of the processor, which is then input into the cost function to obtain the result. ;

[0085] The eleventh calculation unit is used to obtain the optimal value of the splitting parameters while simultaneously satisfying the first to fifth constraints.

[0086] In one implementation, the convolution operation module includes:

[0087] The intermediate data block computation submodule is used to control the internal local buffer to send an input data block and A weighted data block is fed to the arithmetic unit, so that the arithmetic unit can adjust the weighted data block according to an input data block and One weighted data block generates one intermediate data block;

[0088] The output data block calculation submodule is used for cyclic execution. The next control internal local buffer sends an input data block and The steps of sending weighted data blocks to the processor Summing the intermediate data blocks yields a single output data block.

[0089] The loop calculation submodule is used for loop execution. Following the steps described above to obtain an output data block, we get One output data block.

[0090] In one implementation, the intermediate data block computation submodule includes:

[0091] Some intermediate data calculation units are used by each calculation unit to execute steps: according to Given one input data block and one weight data block, we obtain... One intermediate data block, Less than 1 ;

[0092] Summation unit, used for row operations Performing the above steps yields an intermediate data block.

[0093] Thirdly, an electronic device is provided, comprising:

[0094] At least one processor; and

[0095] A memory that is communicatively connected to at least one processor; wherein,

[0096] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform any of the methods described above.

[0097] Fourthly, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods described above is provided.

[0098] One embodiment of the above application has the following advantages or beneficial effects: For large-size convolution operations, the splitting parameters are configured according to the data blocks that the internal local buffer can store and the data blocks that the arithmetic unit can operate on. Using the optimal value of the splitting parameters allows any large-size input image data and weight data to be split into multiple input data blocks and multiple weight data blocks, enabling the convolution acceleration device to perform convolution operations on the input data blocks and weight data blocks to obtain output data blocks. The optimal value of the splitting parameters allows convolution operations of any size to adaptively reduce the bandwidth requirements from DDR to GLB, improve the average utilization of the convolution acceleration device, and effectively reduce the time of convolution operations, achieving a balance between the utilization of the convolution acceleration device and the bandwidth requirements.

[0099] Other effects of the above-mentioned alternative methods will be described below in conjunction with specific embodiments. Attached Figure Description

[0100] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0101] Figure 1 is a schematic diagram of a convolution operation method according to an embodiment of this application;

[0102] Figure 2 is a scene diagram of a convolution operation method according to another embodiment of this application;

[0103] Figure 3 is a schematic diagram of a method for calculating the value of input image data, the value of weight data, and the average utilization rate of the arithmetic unit according to an embodiment of this application;

[0104] Figure 4 is a schematic diagram of a method for splitting input data blocks and weight data blocks according to an embodiment of this application;

[0105] Figure 5 is a schematic diagram of a split convolution operation method according to an embodiment of this application;

[0106] Figure 6 is a schematic diagram of a convolution operation device according to an embodiment of the present application;

[0107] Figure 7 is a schematic diagram of a convolution operation device according to another embodiment of this application;

[0108] Figure 8 is a schematic diagram of a first computing submodule according to another embodiment of this application;

[0109] Figure 9 is a schematic diagram of a second computing submodule according to another embodiment of this application;

[0110] Figure 10 is a schematic diagram of a convolution operation module according to another embodiment of this application;

[0111] Figure 11 is a block diagram of an electronic device used to implement a convolution operation method according to an embodiment of this application. Detailed Implementation

[0112] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0113] In image processing, convolutional neural networks (CNNs) are used to process images, such as extracting features, processing those features, sharpening images, and detecting edges. Typically, convolution operations involve using the convolution kernel in a CNN to perform a series of operations on each pixel in an image. When current storage conditions and hardware computing speed cannot meet the computational demands of complex CNNs, convolution acceleration devices are often used to improve the speed of convolution operations.

[0114] The following is a detailed description of the scheme proposed in this application.

[0115] Figure 1 is a flowchart illustrating a convolution operation method provided in some embodiments of this application. This method is applied to a convolution acceleration device, which may include a FPGA-based (Field-Programmable Gate Array) convolutional neural network accelerator, a CPU-based convolutional neural network accelerator, etc. The convolution acceleration device includes an internal local buffer (GLB) and an arithmetic unit. In the prior art, convolution acceleration devices typically acquire image data from external DDR (Double Data Rate synchronous dynamic random-access memory) for accelerated convolution operations. The execution entity of the convolution operation method provided in some embodiments of this application includes a data splitting device and a convolution acceleration device. The data splitting device may include a splitting parameter configuration module, a splitting parameter optimal value calculation module, a data splitting module, and a data transmission control module. The data splitting device can be located between the external DDR (Double Data Rate SDRAM) and the convolution acceleration device, or it can be located within the convolution acceleration device, with the data splitting device, internal local buffer, and arithmetic unit connected sequentially. The data splitting device is used to obtain image data from DDR and split the image data into multiple data blocks. The data blocks are sent to the convolution acceleration device for accelerated convolution operation, that is, the operation is performed in multiple steps until all data blocks are processed.

[0116] The process in Figure 1 may include the following steps:

[0117] Step S110: Configure the splitting parameters, which are obtained based on the data blocks that the internal local cache can store and the data blocks that the arithmetic unit can process;

[0118] Step S120: Input the splitting parameters, convolution operation parameters, and convolution acceleration device parameters into the cost function, and obtain the optimal value of the splitting parameters under the constraints.

[0119] Step S130: Using the optimal value of the splitting parameter, the input image data and weight data used to input the convolution acceleration device are split into multiple input data blocks and multiple weight data blocks;

[0120] Step S140: Control multiple input data blocks and multiple weight data blocks to be input into the internal local buffer, so that the arithmetic unit can obtain the input data blocks and weight data blocks required for convolution operation from the internal local buffer, and perform convolution operation on them to obtain multiple output data blocks.

[0121] In one example, since different convolution operation parameters lead to different bandwidth requirements and computation speeds, in order to improve computation speed, reduce computation time, and ensure that the bandwidth requirement meets the high-speed computation, thereby maximizing the utilization of the convolution acceleration device, the convolution operation method provided in this embodiment, for large-size convolution operations, breaks down the convolution operation into multiple steps. Therefore, it is necessary to find the optimal values ​​for the parameters of the convolution operation in each individual convolution operation step. Specifically, the large-size input image data and weight data are split into multiple input data blocks and multiple weight data blocks, and each block is calculated in multiple steps to reduce bandwidth requirements. Then, the split input data blocks and weight data blocks are controlled to be input into the GLB for buffering. The GLB then sends the input data blocks and weight data blocks to the arithmetic unit for convolution operation to obtain the output data block. Finally, the output data block is returned to the GLB and then to the DDR. This process is repeated multiple times until all input data blocks and weight data blocks have been processed.

[0122] Since each step utilizes the GLB and the arithmetic unit to process the input data block and weight data block, the input data block and weight data block of the convolution accelerator should be adapted to the storage of image data by the GLB and the processing of image data by the arithmetic unit at each step. Therefore, a splitting parameter is introduced to determine the size of the input data block and the weight data block to be split into the current input image data and weight data.

[0123] In some embodiments of this application, the splitting parameters may include tensors of input data blocks that the GLB can store: the number of batches (n) of the input data blocks, the number of channels of the input data blocks. , height of input data block Here, 'n' refers to the number of batches of input data contained in an input data block, and the number of batches of input data contained in an output data block. The splitting parameters can also include the tensor of the output data block that the GLB can store: the number of channels in the output data block. Height of output data block The splitting parameters may also include: the number of channels for the input data blocks required for each computation module to compute. Each processing module calculates the number of channels required for the output data block. The splitting parameters may also include: calculating the number of channels for the input data blocks that need to be stored in the GLB. The number of channels required for each processing module to calculate the input data block That is, the number of channels of the input data block stored in GLB is used for the number of operations q of the smallest processing unit. The arithmetic unit includes multiple processing modules (PU, Process Unit), which are matrices composed of multiple smallest processing units (PE, Process Element) and control logic, enabling the processing modules to perform convolution operations.

[0124] In some embodiments of this application, the parameters of the convolution operation include parameters of the input image data, parameters of the convolution kernel, and parameters of the output image data. The parameters of the input image data include: a four-dimensional tensor (N, H, W, C) of the input image data, where N (batch) is the number of images input at one time (which can be 1 or other numbers), and the height H and width of the input image data. And the dilation rate V of the input image data. The parameters of the convolution kernel include: the height of the convolution kernel. width of the convolution kernel The parameters of the output image data include: the stride U of the convolution kernel and the dilation rate D of the convolution kernel. The width F of the output image data, the height of the output image data At the hardware level, parameters of the convolution acceleration device include: GLB storage space, etc. The number of computing modules (c) and the height of the computing unit array. and width Number of registers in the arithmetic unit .

[0125] In one example, since the time required for convolution operations is directly proportional to the bandwidth requirement from DDR to GLB, and inversely proportional to the average utilization of the processor, optimizing the splitting parameters (optimizing the size of the input data blocks and weight data blocks after splitting) minimizes the DDR to GLB bandwidth requirement in each step while maximizing the average utilization of the processor. Specifically, first, the splitting parameter calculation module determines constraints based on the parameters of the convolution accelerator and the splitting parameters. Then, based on the splitting parameters, the convolution operation parameters, and the parameters of the convolution accelerator, it calculates the values ​​of the input image data, the weight data, and the average utilization of the processor. The sum of the input image data and the weight data represents the DDR to GLB bandwidth requirement. Next, the splitting parameter calculation module inputs the ratio of the sum of the input image data and weight data to the average utilization of the processor into the cost function. Under the constraints, it obtains the optimal value of the splitting parameters. Finally, the data splitting module uses the optimal value of the splitting parameters to split the input image data used to input the convolution accelerator into multiple input data blocks and the weight data into multiple weight data blocks. The data transmission control module controls multiple input data blocks and multiple weight data blocks to be input sequentially into the internal local buffer, so that the arithmetic unit can obtain the input data blocks and weight data blocks required for convolution operation from the internal local buffer, and perform convolution operation on them to obtain multiple output data blocks.

[0126] In this embodiment, for large-sized convolution operations, splitting parameters are configured based on the data blocks that the internal local buffer can store and the data blocks that the arithmetic unit can process. Using the optimal value of the splitting parameters allows any large-sized input image data and weight data to be split into multiple input data blocks and multiple weight data blocks. This enables the convolution acceleration device to perform convolution operations on the input data blocks and weight data blocks to obtain the output data blocks. The optimal value of the splitting parameters allows for an adaptive reduction in the DDR to GLB bandwidth requirement for convolution operations of any size, improving the average utilization of the convolution acceleration device while effectively reducing the convolution operation time, thus achieving a balance between the utilization of the convolution acceleration device and bandwidth requirements.

[0127] Based on the method in Figure 1, some embodiments of this application also provide some specific implementation schemes and extended schemes of the method, which are described below.

[0128] In one implementation, the internal local cache can simultaneously store an input data block, a weight data block, and an output data block.

[0129] In one example, after the DDR sends the input image data and weight data to the convolution operation device, it is split into input data blocks and weight data blocks. The GLB can temporarily cache an input data block and a weight data block and send them to the arithmetic unit for operation. When the arithmetic unit calculates an output data block, it can return an output data block to the GLB.

[0130] In one embodiment, as shown in FIG2, it further includes:

[0131] Step S111: Obtain the parameters for the convolution operation. The parameters for the convolution operation include: the height of the convolution kernel. and the width of the convolution kernel The number of batches N of input image data and the number of channels of input image data. Width of input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data, the height of the output image data The output image data includes output data blocks;

[0132] Step S112: Obtain the parameters of the convolution accelerator, including: the storage space of the internal local cache. The height of the arithmetic unit array in the arithmetic module and width Number of registers in the arithmetic unit .

[0133] In one example, during the convolution operation, parameters of the convolutional neural network are involved, such as the size of the convolution kernel, the size of the input image data, and the size of the output image data. For example, the input is a 32 ( The image is 32 (H) * 3 (C) pixels, and the number of channels in the input image data is 32 (H) * 3 (C). =3 indicates RGB three channels, each convolutional kernel is 5(R)*5(S)*3(C), and one convolutional kernel produces one feature map. If there are 6 5*5*3 convolutional kernels, then 6 feature maps are output, with a size of 28(F)*28 ( )*6 ( The parameters for convolution operations can be configured according to user needs, including the size of the input image data to be processed, the kernel size, and the output image data size. They can also be configured based on the size of the input image data sent from the DDR to the data splitting device, the kernel size, and the output image data size. The parameters of the convolution acceleration device are hardware-level parameters, representing the storage capacity of the GLB and the computing power of the processor. During the convolution operation, the values ​​of the convolution operation parameters and the parameters of the convolution acceleration device are configured according to different requirements and are known values.

[0134] In one embodiment, as shown in FIG2, step S110 includes:

[0135] Step S113: Configure the internal local buffer to store the required batch number n of input data blocks and the number of channels of the input data blocks. , height of input data block Number of channels in the output data block Height of output data block Where the number of batches n of the input data blocks is less than or equal to the number of batches N of the input image data;

[0136] Step S114: Configure the number of channels for the input data blocks required for calculation by each arithmetic module in the arithmetic unit. and the number of channels for output data blocks .

[0137] In one example, the storage size of the GLB and the computing power of the processor vary in different convolution acceleration devices. Since the GLB can store an input data block, a weight data block, and an output data block, the parameters configuring the GLB to store the size of an input data block, such as the number of channels in the input data block, can be configured. , height of input data block And the number of batches n of the input data blocks. Wherein, the height of the input data block... ,in, The stride of the convolution kernel. The height of the convolution kernel. This specifies the height of the output data block that the GLB can store. It configures the size of an output data block that the GLB can store, for example, the number of channels in the output data block. Height of output data block The number of output data blocks, n. It should be noted that the number of batches n refers to the number of input data blocks containing n batches of input data. The number of output data batches is equal to the number of batches in the input data blocks, both being n. The number of channels for the input data blocks required for calculation by each processing module is configured according to the processing power of the arithmetic unit. and the number of channels for output data blocks .

[0138] It is important to note that due to the limited capacity of the GLB fast memory, large convolutions need to be split into multiple smaller convolutions for computation by the convolution accelerator. Therefore, when configuring the splitting parameters based on the data blocks that the GLB can store and the data blocks that the arithmetic unit can process, there are various possible values ​​for the splitting parameters. As long as they do not exceed the GLB's storage capacity, the arithmetic unit can be further utilized for smaller convolution operations. However, different splitting parameters will lead to different computation speed and bandwidth requirements. Therefore, it is necessary to find a set of splitting parameters that minimizes computation time, maximizes the utilization of the convolution accelerator, and ensures that the bandwidth keeps up with the computation speed—that is, the optimal values ​​of the splitting parameters. Therefore, when configuring the splitting parameters, the parameters are unknown, and the method provided in this embodiment is needed to find the optimal values ​​of the splitting parameters.

[0139] In one embodiment, as shown in FIG2, step S120 includes:

[0140] Step S121: Calculate the values ​​of the input image data, the weight data, and the average utilization of the processor based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device;

[0141] Step S122: Determine the constraints based on the parameters of the convolution accelerator and the splitting parameters;

[0142] Step S123: The sum of the input image data values ​​and the weight data values, and the ratio of the average utilization rate of the processor, are input into the cost function. Under the condition that the constraints are met, the optimal values ​​of the splitting parameters are obtained.

[0143] In one example, the parameters of the convolution operation and the convolution accelerator are typically known, while the splitting parameters are unknown. Based on the splitting parameters, the convolution operation parameters, and the convolution accelerator parameters, the sum of the input image data values ​​and the weight data values ​​is calculated to obtain the bandwidth requirement from DDR to GLB, and the average utilization of the processor is calculated. Under the constraints, when the ratio of the DDR to GLB bandwidth requirement to the average utilization of the processor is minimized, the average utilization of the processor is maximized, the DDR to GLB bandwidth requirement is minimized, and the computation time is minimized. At this point, the optimal value of the splitting parameters is calculated.

[0144] In one embodiment, as shown in FIG3, step S121 includes:

[0145] Step S1211: Calculate the number of channels in the input data block. , height of input data block and the width of the input image data The product of these two numbers represents the amount of data loaded from external memory into the internal local cache for an input data block. ;

[0146] Step S1212: Calculate the number of channel splits in the input image data. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from external memory to the internal local cache. ;

[0147] Step S1213: Calculate the number of transmissions and The product of these terms yields the value of the input image data. .

[0148] In one example, the number of data transfers from DDR to GLB is calculated. Number of transmissions and The product of these terms yields the value of the input image data. , which represents the size of the input image data.

[0149] In one embodiment, as shown in FIG3, step S121 includes:

[0150] Step S1214: Height of the convolution kernel width of the convolution kernel The arithmetic module calculates the number of channels required for the input data blocks. and the number of channels in the output data block The product of these values ​​yields the value of the weighted data block. ;

[0151] Step S1215: Calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q;

[0152] Step S1216: Calculate the number of channels required for the internal local buffer to store the output data blocks. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q';

[0153] Step S1217: Calculate the ratio Sum and ratio The product of these numbers, b, is the number of operations required by the computation module to obtain the output data of the internal local buffer.

[0154] Step S1218: Calculate the number of transmissions Number of operations and the values ​​of the weighted data blocks The product of these factors yields the value of the weighted data. .

[0155] In one example, calculating the output data of the GLB requires b number of operations from the computation module. Number of transmissions Number of operations and the values ​​of the weighted data blocks The product of these factors yields the value of the weighted data. , which represents the size of the weighted data.

[0156] In one embodiment, as shown in FIG3, step S121 includes:

[0157] Step S1219: Based on the height of the computing unit array and width height of the convolution kernel The arithmetic module calculates the number of channels required for the input data blocks. The average utilization rate of the arithmetic unit .

[0158] In one implementation, step S122 includes:

[0159] Step S1221: Input the value of the data block The value of the weighted data block and the values ​​of the output data blocks The sum is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely ;

[0160] Step S1222: The arithmetic module calculates the number of channels required for the output data block. Less than or equal to the width of the arithmetic unit array The arithmetic module calculates the number of channels required for the input data block. With the height of the convolution kernel The product is less than or equal to the height of the arithmetic unit array. Thus, the second constraint condition is obtained, namely ;

[0161] Step S1223: Width of the convolution kernel Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely ;

[0162] Step S1224: The internal local buffer stores the number of channels of the required input data blocks. The number of channels in the input image data is less than or equal to the number of channels in the input image data. Thus, the fourth constraint condition is obtained, namely ;

[0163] Step S1225: The arithmetic module calculates the number of channels required for the output data block. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely .

[0164] In one example, the optimal values ​​of the splitting parameters are calculated while satisfying five constraints simultaneously. The first constraint is the spatial constraint of GLB: The second constraint is the size constraint of the computation unit: , The third constraint is the size constraint of the registers in the arithmetic unit: The fourth constraint is the size constraint of the input data block: ;

[0165] The fifth constraint is the size constraint of the output data block: .

[0166] In one embodiment, as shown in FIG4, step S130 includes:

[0167] Step S131: The sum of the input image data values ​​and the weight data values, divided by the ratio of this sum to the average utilization rate of the processor, is input into the cost function to obtain... ;

[0168] Step S132: Under the condition that the first to fifth constraints are satisfied at the same time, the optimal value of the splitting parameters is obtained.

[0169] In one example, the optimal values ​​of the splitting parameters are found through enumeration. Each splitting parameter corresponds to a range of values. The cost function is calculated based on the combinations of different value ranges, and the splitting parameter corresponding to the minimum value of the cost function is the optimal value. For example, given N=1, R=5, S=5, M=16, C=3, H=128, W=256, E=127, F=253, Jw=32, Jh=24, Q=786432, RF=8, all values ​​of the variables that satisfy the first to fifth constraints are enumerated and substituted into the cost function. The group with the smallest cost function value is selected, resulting in n=1, e=30, m=16, d=3, r=3, t=16.

[0170] In one embodiment, as shown in FIG5, step S140 includes:

[0171] Step S141: Control the internal local buffer to send an input data block and A weighted data block is fed to the arithmetic unit, so that the arithmetic unit can adjust the weighted data block according to an input data block and One weighted data block generates one intermediate data block;

[0172] Step S142: Execute repeatedly The next control internal local buffer sends an input data block and The steps of sending weighted data blocks to the processor Summing the intermediate data blocks yields a single output data block.

[0173] Step S143: Execute repeatedly Following the steps described above to obtain an output data block, we get One output data block.

[0174] In one implementation, in step S141, the arithmetic unit calculates based on an input data block and One weighted data block generates an intermediate data block, including:

[0175] Each arithmetic unit executes the following steps: Based on... Given one input data block and one weight data block, we obtain... Intermediate data, Less than 1 ;

[0176] Each processing unit performs the above steps to obtain an intermediate data block.

[0177] In one example, the data transport control module controls the DDR to send an input data block and weight data to the GLB for storage, and the arithmetic unit loads an input data block and weight data from the GLB. Weighted data blocks ( (More than one weighted data block), in the arithmetic unit, based on an input data block and Each weighted data block generates an intermediate data block (psum), which is then stored in the GLB. Each computational unit in the arithmetic unit loads... Given one input data block (RrW is less than one input data block) and one weight data block, we get... intermediate data blocks ( ), e rows of operation units, The intermediate data blocks are summed e times to obtain a single intermediate data block. The arithmetic unit then loads another input data block from the GLB. Each weighted data block is used to obtain an intermediate data block, until this step is completed. After repeating this process 10 times, one output data block is obtained and stored in GLB. The above steps are repeated EM / em times, outputting EM / em output data blocks.

[0178] Based on the same idea, some embodiments of this application also provide apparatus, devices and non-volatile computer storage media corresponding to the above methods.

[0179] Figure 6 is a schematic diagram of a convolution operation device corresponding to Figure 1 provided in some embodiments of this application.

[0180] The splitting parameter configuration module 110 is used to configure the splitting parameters, which are obtained based on the data blocks that the internal local cache can store and the data blocks that the arithmetic unit can process.

[0181] The optimal value calculation module 120 for splitting parameters is used to input the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device into the cost function, and obtain the optimal value of the splitting parameters under the condition of meeting the constraints.

[0182] The data splitting module 130 is used to split the input image data and weight data used to input the convolution accelerator into multiple input data blocks and multiple weight data blocks by using the optimal value of the splitting parameters.

[0183] The convolution operation module 140 is used to control the input of multiple input data blocks and multiple weight data blocks into an internal local buffer, so that the operator can obtain the input data blocks and weight data blocks required for the convolution operation from the internal local buffer and perform convolution operation on them to obtain multiple output data blocks.

[0184] In one implementation, the internal local cache can simultaneously store an input data block, a weight data block, and an output data block.

[0185] In one embodiment, as shown in FIG7, it further includes:

[0186] The first acquisition module 111 is used to acquire the parameters of the convolution operation, including the height of the convolution kernel. and the width of the convolution kernel The number of batches N of input image data and the number of channels of input image data. Width of input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data, the height of the output image data The output image data includes output data blocks;

[0187] The second acquisition module 112 is used to acquire parameters of the convolution accelerator, including: the storage space of the internal local cache. The height of the arithmetic unit array in the arithmetic module and width Number of registers in the arithmetic unit .

[0188] In one implementation, as shown in FIG7, the split parameter configuration module 110 includes:

[0189] The first configuration submodule 113 is used to configure the number of batches n of the required input data blocks and the number of channels of the input data blocks stored in the internal local buffer. , height of input data block Number of channels in the output data block Height of output data block Where the number of batches n of the input data blocks is less than or equal to the number of batches N of the input image data;

[0190] The second configuration submodule 114 is used to configure the number of channels for the input data blocks required for calculation by each arithmetic module in the arithmetic unit. and the number of channels for output data blocks .

[0191] In one implementation, as shown in Figure 7, the optimal value calculation module 120 for the splitting parameters includes:

[0192] The first calculation submodule 121 is used to calculate the value of the input image data, the value of the weight data, and the average utilization rate of the arithmetic unit based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device.

[0193] The constraint determination submodule 122 is used to determine the constraint conditions based on the parameters of the convolution accelerator and the splitting parameters.

[0194] The second calculation submodule 123 is used to input the sum of the values ​​of the image data and the weight data, and the ratio of the average utilization rate of the arithmetic unit into the cost function, so as to obtain the optimal value of the splitting parameters under the condition that the constraints are met.

[0195] In one implementation, as shown in FIG8, the first calculation submodule 121 includes:

[0196] The first calculation unit 1211 is used to calculate the number of channels in the input data block. , height of input data block and the width of the input image data The product of these two values ​​gives the amount of data that an input data block is loaded from external memory into the internal local cache. ;

[0197] The second calculation unit 1212 is used to calculate the number of times the input image data is split into channels. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from external memory to the internal local cache. = ;

[0198] The third calculation unit 1213 is used to calculate the number of transmissions. and The product of these terms yields the value of the input image data. .

[0199] In one embodiment, as shown in FIG8, the first calculation submodule 121 further includes:

[0200] The fourth computational unit 1214 is used for the height of the convolution kernel. width of the convolution kernel The arithmetic module calculates the number of channels required for the input data blocks. and the number of channels in the output data block The product of these values ​​yields the value of the weighted data block. ;

[0201] The fifth calculation unit 1215 is used to calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q;

[0202] The sixth calculation unit 1216 is used to calculate the number of channels required for the internal local buffer to store the output data blocks. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q';

[0203] The seventh calculation unit 1217 is used to calculate the ratio. Sum and ratio The product of these two operations, to obtain the output data of the internal local buffer, requires computation. The number of operations b = ;

[0204] The eighth calculation unit 1218 is used to calculate the number of transmissions. Number of operations and the values ​​of the weighted data blocks The product of these factors yields the value of the weighted data. .

[0205] In one embodiment, as shown in FIG8, the first calculation submodule 121 further includes:

[0206] The ninth computing unit 1219 is used to determine the height of the computing unit array. and width height of the convolution kernel , operation Calculate the number of channels required for the input data block The average utilization rate of the arithmetic unit .

[0207] In one implementation, the constraint determination submodule 122 includes:

[0208] The first constraint calculation unit is used to input the values ​​of the data block. The value of the weighted data block and the values ​​of the output data blocks The sum is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely ;

[0209] The second constraint calculation unit is used by the arithmetic module to calculate the number of channels required for the output data block. Less than or equal to the width of the arithmetic unit array The arithmetic module calculates the number of channels required for the input data block. With the height of the convolution kernel The product is less than or equal to the height of the arithmetic unit array. Thus, the second constraint condition is obtained, namely ;

[0210] The third constraint calculation unit is used for the width of the convolution kernel. Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely ;

[0211] The fourth constraint calculation unit is used to store the number of channels of the required input data blocks in the internal local buffer. The number of channels in the input image data is less than or equal to the number of channels in the input image data. Thus, the fourth constraint condition is obtained, namely ;

[0212] The fifth constraint calculation unit is used by the arithmetic module to calculate the number of channels required for the output data block. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely .

[0213] In one implementation, as shown in FIG9, the second calculation submodule 123 includes:

[0214] The tenth calculation unit 1231 is used to input the sum of the image data values ​​and the weight data values, and the ratio of this sum to the average utilization rate of the processor, into the cost function to obtain... ;

[0215] The eleventh calculation unit 1232 is used to obtain the optimal value of the splitting parameters while simultaneously satisfying the first to fifth constraints.

[0216] In one embodiment, as shown in FIG10, the convolution operation module 140 includes:

[0217] Intermediate data block calculation submodule 141 is used to control the internal local buffer to send an input data block and A weighted data block is fed to the arithmetic unit, so that the arithmetic unit can adjust the weighted data block according to an input data block and One weighted data block generates one intermediate data block;

[0218] Output data block calculation submodule 142 is used for cyclic execution. The next control internal local buffer sends an input data block and The steps of sending weighted data blocks to the processor Summing the intermediate data blocks yields a single output data block.

[0219] Loop calculation submodule 143 is used for loop execution. Following the steps described above to obtain an output data block, we get One output data block.

[0220] In one implementation, the intermediate data block calculation submodule 141 includes:

[0221] Some intermediate data calculation units are used by each calculation unit to execute steps: according to Given one input data block and one weight data block, we obtain... One intermediate data block, Less than 1 ;

[0222] Summation unit, used for row operations Performing the above steps yields an intermediate data block.

[0223] The functions of each module in each device in the embodiments of this application can be found in the corresponding descriptions in the above methods, and will not be repeated here.

[0224] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0225] Figure 11 is a block diagram of an electronic device according to an embodiment of the convolution operation method of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0226] As shown in Figure 11, the electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a graphical user interface (GUI) on external input / output devices (such as display devices coupled to the interfaces). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 11 uses a single processor 1101 as an example.

[0227] The memory 1102 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform a convolution operation method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform a convolution operation method provided in this application.

[0228] The memory 1102, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to a convolution operation method in this application embodiment (e.g., the splitting parameter configuration module 110, the splitting parameter optimal value calculation module 120, the data splitting module 130, and the convolution operation module 140 shown in Figure 6). The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing a convolution operation method in the above method embodiment.

[0229] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device according to a convolution operation method. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0230] The aforementioned electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means, as shown in Figure 11, which illustrates a connection via a bus.

[0231] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the aforementioned electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0232] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0235] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0236] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0237] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0238] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A convolution operation method applied to a convolution acceleration device, the convolution acceleration device comprising an internal local buffer and an arithmetic unit, characterized in that, include: Configure splitting parameters, which are obtained based on the data blocks that the internal local cache can store and the data blocks that the processor can operate on. Input the splitting parameters, convolution operation parameters, and parameters of the convolution acceleration device into a cost function, and obtain the optimal value of the splitting parameters under the constraints. The cost function is constructed as the ratio of the sum of the input image data values ​​and the weight data values ​​to the average utilization of the processor. The input image data values ​​refer to the total amount of input image-related data transferred from the external memory to the internal local cache, based on input data blocks as the basic unit. The weight data values ​​refer to the total amount of weight-related data transferred from the external memory to the internal local cache, which is matched and operated on with the input data blocks. The average utilization of the arithmetic unit is determined by the ratio of the product of the height of the convolution kernel and the number of channels of the input data block and the number of channels of the output data block required for calculation by the arithmetic module to the product of the width and height of the arithmetic unit array in the arithmetic module. The constraint is determined based on the parameters of the convolution acceleration device and the splitting parameters. Using the optimal value of the splitting parameters, the input image data and weight data used to input the convolution acceleration device are split into multiple input data blocks and multiple weight data blocks. The multiple input data blocks and the multiple weight data blocks are controlled to be input into the internal local buffer, so that the arithmetic unit obtains the input data blocks and weight data blocks required for convolution operation from the internal local buffer and performs convolution operation on them to obtain multiple output data blocks.

2. The method according to claim 1, characterized in that, The internal local cache can simultaneously store one input data block, one weight data block, and one output data block.

3. The method according to claim 1, characterized in that, Also includes: Obtain the parameters for the convolution operation, including the height of the convolution kernel. and the width of the convolution kernel The number of batches N of the input image data and the number of channels of the input image data. The width of the input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data and the height of the output image data The output image data includes the output data block; the parameters of the convolution acceleration device are obtained, and the parameters of the convolution acceleration device include: the storage space of the internal local cache. The height of the computing unit array in the computing module and width The number of registers in the arithmetic unit 。 4. The method according to claim 3, characterized in that, The configuration splitting parameters include: configuring the number of batches n of the input data blocks required for storage in the internal local cache, and the number of channels of the input data blocks. , height of input data block Number of channels in the output data block Height of output data block Wherein, the batch number n of the input data block is less than or equal to the batch number N of the input image data; the number of channels of each processing module in the processor is configured to calculate the required input data block. and the number of channels for output data blocks 。 5. The method according to claim 4, characterized in that, The step of inputting the splitting parameters, convolution operation parameters, and convolution acceleration device parameters into the cost function to obtain the optimal value of the splitting parameters under the condition of satisfying the constraints includes: calculating the value of the input image data, the value of the weight data, and the average utilization rate of the processor based on the splitting parameters, the convolution operation parameters, and the convolution acceleration device parameters; determining the constraints based on the parameters of the convolution acceleration device and the splitting parameters; inputting the ratio of the sum of the input image data value and the weight data value to the average utilization rate of the processor into the cost function, and obtaining the optimal value of the splitting parameters under the condition of satisfying the constraints.

6. The method according to claim 5, characterized in that, The step of calculating the value of the input image data based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device includes: calculating the number of channels in the input data block. The height of the input data block and the width of the input image data The product of these two values ​​yields the amount of data that the input data block is loaded from external memory into the internal local cache. ; Calculate the number of times the input image data is split into channels. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from the external memory to the internal local cache. = ; Calculate the number of transmissions and The product of these terms yields the value of the input image data. 。 7. The method according to claim 6, characterized in that, The step of calculating the weight data value based on the splitting parameters, the convolution operation parameters, and the convolution acceleration device parameters includes: the height of the convolution kernel. The width of the convolution kernel The calculation module calculates the number of channels for the required input data blocks. and the number of channels of the output data block. The product of these factors yields the value of the weighted data block. ; Calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations q; the number of channels required for the internal local buffer to store the output data blocks. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations q'; calculate the ratio Sum and ratio The product of these operations, to obtain the output data of the internal local buffer, requires the following computation. The number of operations b = ; Calculate the number of transmissions The number of operations and the value of the weighted data block The product of these factors yields the value of the weighted data. 。 8. The method according to claim 7, characterized in that, The average utilization of the arithmetic unit is calculated based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device, including: based on the height of the arithmetic unit array. and width The height of the convolution kernel The operation Calculate the number of channels required for the input data block Calculate the average utilization rate of the arithmetic unit. 。 9. The method according to claim 8, characterized in that, Determining the constraint conditions based on the parameters of the convolution acceleration device and the splitting parameters includes: the value of the input data block. The value of the weighted data block and the value of the output data block The sum of these values ​​is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely The calculation module calculates the number of channels required for the output data block. Less than or equal to the width of the arithmetic unit array The computing module calculates the number of channels for the required input data blocks. With respect to the height of the convolution kernel The product is less than or equal to the height of the array of computing units. Thus, the second constraint condition is obtained, namely The width of the convolution kernel Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely The internal local buffer stores the number of channels for the required input data blocks. The number of channels of the input image data is less than or equal to Thus, the fourth constraint condition is obtained, namely The calculation module calculates the number of channels required for the output data block. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely 。 10. The method according to claim 9, characterized in that, The ratio of the sum of the input image data values ​​and the weight data values ​​to the average utilization rate of the processor is input into the cost function. Under the constraints, the optimal value of the splitting parameter is obtained, including: the ratio of the sum of the input image data values ​​and the weight data values ​​to the average utilization rate of the processor is input into the cost function to obtain... The optimal value of the splitting parameter is obtained when the first to the fifth constraints are satisfied simultaneously.

11. The method according to claim 10, characterized in that, The process involves controlling the multiple input data blocks and the multiple weight data blocks to be sequentially input into the internal local buffer, so that the arithmetic unit obtains the input data blocks and weight data blocks required for the convolution operation from the internal local buffer, and performs convolution operation on them to obtain multiple output data blocks. This includes controlling the internal local buffer to send one of the input data blocks and weight data blocks. The weighted data block is fed to the arithmetic unit, so that the arithmetic unit calculates the weighted data block according to the input data block and... One intermediate data block is generated from each of the weighted data blocks; the process is repeated cyclically. The next step involves controlling the internal local buffer to send an input data block and... The step of sending the weighted data blocks to the processor. Summing each intermediate data block yields one output data block; this process is repeated. The steps described above for obtaining one of the output data blocks result in... The output data blocks are described.

12. The method according to claim 11, characterized in that, The arithmetic unit is based on one of the input data blocks and Generate an intermediate data block from the weighted data blocks, including: each of the computational units performing the following steps: based on... Given one input data block and one weight data block, we obtain The aforementioned intermediate data blocks Less than 1 ； The operation described in the row By performing the above steps, an intermediate data block is obtained.

13. A convolution operation device, characterized in that, include: A splitting parameter configuration module is used to configure splitting parameters, which are obtained based on the data blocks that the internal local cache can store and the data blocks that the arithmetic unit can operate on. A splitting parameter optimal value calculation module is used to input the splitting parameters, convolution operation parameters, and convolution acceleration device parameters into a cost function, and obtain the optimal value of the splitting parameters under certain constraints. The cost function is constructed as the ratio of the sum of the input image data values ​​and the weight data values ​​to the average utilization rate of the arithmetic unit. The input image data values ​​refer to the total amount of input image-related data transferred from external memory to the internal local cache, based on input data blocks as the basic unit. The weight data values ​​refer to the total amount of weight-related data transferred from external memory to the internal local cache, matching the input data blocks for computation. The average utilization rate of the arithmetic unit is determined by the ratio of the height of the convolution kernel, the product of the number of channels in the input data blocks required for computation by the arithmetic module, and the number of channels in the output data blocks, to the product of the width and height of the arithmetic unit array in the arithmetic module. The constraints are determined based on the parameters of the convolution acceleration device and the splitting parameters. The data splitting module is used to split the input image data and weight data used to input the convolution acceleration device into multiple input data blocks and multiple weight data blocks using the optimal value of the splitting parameters; the convolution operation module is used to control the multiple input data blocks and the multiple weight data blocks to be input into the internal local buffer, so that the operator can obtain the input data blocks and weight data blocks required for the convolution operation from the internal local buffer and perform convolution operation on them to obtain multiple output data blocks.

14. The apparatus according to claim 13, characterized in that, The internal local cache can simultaneously store one input data block, one weight data block, and one output data block.

15. The apparatus according to claim 13, characterized in that, Also includes: The first acquisition module is used to acquire the parameters of the convolution operation, including the height of the convolution kernel. and the width of the convolution kernel The number of batches N of the input image data and the number of channels of the input image data. The width of the input image data The height H of the input image data; the number of channels of the output image data. The width F of the output image data and the height of the output image data The output image data includes the output data block; the second acquisition module is used to acquire the parameters of the convolution acceleration device, the parameters of the convolution acceleration device including: the storage space of the internal local cache. The height of the computing unit array in the computing module and width The number of registers in the arithmetic unit 。 16. The apparatus according to claim 15, characterized in that, The splitting parameter configuration module includes: a first configuration submodule, used to configure the number of batches n of the input data blocks required for storage in the internal local cache, and the number of channels of the input data blocks. , height of input data block Number of channels in the output data block Height of output data block Wherein, the batch number n of the input data block is less than or equal to the batch number N of the input image data; the second configuration submodule is used to configure the number of channels of the input data block required for calculation by each computing module in the computing unit. and the number of channels for output data blocks 。 17. The apparatus according to claim 16, characterized in that, The optimal value calculation module for the splitting parameters includes: a first calculation submodule, used to calculate the value of the input image data, the value of the weight data, and the average utilization rate of the processor based on the splitting parameters, the parameters of the convolution operation, and the parameters of the convolution acceleration device; a constraint determination submodule, used to determine constraint conditions based on the parameters of the convolution acceleration device and the splitting parameters; and a second calculation submodule, used to input the ratio of the sum of the value of the input image data and the value of the weight data to the average utilization rate of the processor into the cost function, and obtain the optimal value of the splitting parameters under the condition that the constraint conditions are met.

18. The apparatus according to claim 17, characterized in that, The first calculation submodule includes: a first calculation unit, used to calculate the number of channels of the input data block. The height of the input data block and the width of the input image data The product of these two values ​​yields the amount of data that the input data block is loaded from external memory into the internal local cache. The second calculation unit is used to calculate the number of times the input image data is split into channels. The number of times the height of the output image data is split. and the number of times the output image data channels are split. The product of these numbers yields the number of data transfers from the external memory to the internal local cache. = The third calculation unit is used to calculate the number of transmissions. and The product of these terms yields the value of the input image data. 。 19. The apparatus according to claim 18, characterized in that, The first calculation submodule further includes: a fourth calculation unit, used for the height of the convolution kernel. The width of the convolution kernel The calculation module calculates the number of channels for the required input data blocks. and the number of channels of the output data block. The product of these factors yields the value of the weighted data block. The fifth calculation unit is used to calculate the number of channels required for the internal local buffer to store the input data blocks. and Number of channels in the input data block ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations, q; the sixth calculation unit, used to calculate the number of channels required for the internal local buffer to store the output data blocks. ratio That is, the number of channels for storing the required input data blocks in the internal local buffer. The number of operations q'; the seventh calculation unit, used to calculate the ratio. Sum and ratio The product of these operations, to obtain the output data of the internal local buffer, requires the following computation. The number of operations b = The eighth calculation unit is used to calculate the number of transmissions. The number of operations and the value of the weighted data block The product of these factors yields the value of the weighted data. 。 20. The apparatus according to claim 19, characterized in that, The first calculation submodule further includes: a ninth calculation unit, configured to calculate based on the height of the calculation unit array. and width The height of the convolution kernel The operation Calculate the number of channels required for the input data block Calculate the average utilization rate of the arithmetic unit. 。 21. The apparatus according to claim 20, characterized in that, The constraint determination submodule includes: a first constraint calculation unit, used for calculating the value of the input data block. The value of the weighted data block and the value of the output data block The sum of these values ​​is less than or equal to the storage space of the internal local cache. Thus, the first constraint condition is obtained, namely The second constraint calculation unit is used by the arithmetic module to calculate the number of channels for the required output data blocks. Less than or equal to the width of the arithmetic unit array The computing module calculates the number of channels for the required input data blocks. With respect to the height of the convolution kernel The product is less than or equal to the height of the array of computing units. Thus, the second constraint condition is obtained, namely The third constraint calculation unit is used for calculating the width of the convolution kernel. Less than or equal to the number of registers in the arithmetic module Thus, the third constraint condition is obtained, namely The fourth constraint calculation unit is used to store the number of channels for the required input data blocks in the internal local buffer. The number of channels of the input image data is less than or equal to Thus, the fourth constraint condition is obtained, namely The fifth constraint calculation unit is used by the arithmetic module to calculate the number of channels for the required output data blocks. The number of channels less than or equal to the number of output data blocks required for storage in the internal local buffer. Thus, the fifth constraint condition is obtained, namely 。 22. The apparatus according to claim 21, characterized in that, The second calculation submodule includes: a tenth calculation unit, used to input the sum of the input image data value and the weight data value, and the ratio of the average utilization rate of the processor, into the cost function to obtain... The eleventh calculation unit is used to obtain the optimal value of the splitting parameter while simultaneously satisfying the first to the fifth constraint conditions.

23. The apparatus according to claim 22, characterized in that, The convolution operation module includes: an intermediate data block calculation submodule, used to control the internal local buffer to send one of the input data blocks and The weighted data block is fed to the arithmetic unit, so that the arithmetic unit calculates the weighted data block according to the input data block and... The weighted data blocks generate an intermediate data block; the output data block calculation submodule is used for cyclic execution. The next step involves controlling the internal local buffer to send an input data block and... The step of sending the weighted data blocks to the processor. The intermediate data blocks are summed to obtain one output data block; the loop calculation submodule is used to perform loop calculations. The steps described above for obtaining one of the output data blocks result in... The output data blocks are described.

24. The apparatus according to claim 23, characterized in that, The intermediate data block calculation submodule includes: a partial intermediate data calculation unit, used by each of the calculation units to perform the following steps: according to Given one input data block and one weight data block, we obtain The aforementioned intermediate data blocks Less than 1 Summation unit, used for The operation described in the row By performing the above steps, an intermediate data block is obtained.

25. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

Images