Hardware accelerators, processors, chips, and electronic devices

Abstract

Embodiments of the present application provide a hardware accelerator, a processor, a chip and an electronic device, wherein the hardware accelerator comprises a PE array, an internal cache unit, and a data scheduler arranged between the PE array and the internal cache unit; the data scheduler is configured to sequentially obtain a plurality of image rows to be processed from the internal cache unit, and schedule the PE array to sequentially perform MAC operation processing on the plurality of image rows; there are overlapping pixel rows between adjacent image rows, and the overlapping pixel rows are subjected to MAC operation processing in the adjacent image rows to which the overlapping pixel rows belong; in the process of performing MAC operation processing on each image row, the PEs of the PE array perform MAC operation processing on a plurality of row tiles contained in each image row in units of tiles; for adjacent row tiles, the operation results of the overlapping part of the previous row tile and the next row tile are cached, and the operation results of the non-overlapping part of the next row tile are combined as the MAC operation processing result of the next row tile.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of hardware acceleration, and particularly relate to a hardware accelerator, a processor, a chip, and an electronic device. BACKGROUND

[0002] Convolutional Neural Network (CNN) is one of the most important algorithms in deep learning, and is widely used in fields such as autonomous driving, computer vision, and speech recognition due to its high precision and small weight amount. In order to deploy CNN more efficiently on terminals, the industry has developed corresponding neural network hardware accelerators for different neural networks.

[0003] The existing neural network hardware accelerators are mostly used for CNNs for classification tasks, and process CNNs according to the layer order of convolutional layers. In this processing mode, the feature data and weight data are loaded into the memory of the hardware accelerator in different orders for reuse within the layer. After the operation of one layer is completed through the off-chip memory, the operation of the next layer is started.

[0004] However, compared with CNNs for classification tasks, image enhancement tasks do not frequently downsample images such as feature maps, which results in that the data amount and operation amount of feature data for image enhancement tasks are much higher than those for classification tasks when processing input image data of the same size. The frequent data interaction between the neural network hardware accelerator and the off-chip memory thus causes serious calculation delay and processing power consumption problems. SUMMARY

[0005] Therefore, embodiments of the present application provide a hardware acceleration scheme to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a hardware accelerator is provided, comprising: a processing element (PE) array, an internal cache unit of the hardware accelerator, and a data scheduler arranged between the PE array and the internal cache unit; wherein the data scheduler is configured to: sequentially obtain a plurality of image rows to be processed from the internal cache unit, and schedule the PE array to sequentially perform MAC operation processing on the plurality of image rows, wherein there are overlapping pixel rows between adjacent image rows, and the overlapping pixel rows are subjected to MAC operation processing in both of the adjacent image rows to which the overlapping pixel rows belong; and in the process of performing MAC operation processing on each image row, schedule a PE in the PE array that processes the current image row to perform MAC operation processing on a plurality of row tiles contained in each image row in units of tiles, wherein for adjacent row tiles, cache the operation results of the overlapping part of the previous row tile and the non-overlapping part of the next row tile as the MAC operation processing result of the next row tile.

[0007] According to a second aspect of the embodiments of the present application, a processor is provided, comprising the hardware accelerator according to the first aspect.

[0008] According to a third aspect of the embodiments of the present application, a chip is provided, comprising the processor according to the second aspect.

[0009] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, comprising the chip according to the third aspect.

[0010] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the method according to the first aspect.

[0011] According to the scheme provided by the embodiments of the present application, in the hardware accelerator that can be used for image enhancement tasks, a data scheduler is additionally added to perform data scheduling for MAC (Multiply Accumulate, multiply-accumulate operation) operation processing between the internal cache unit and the PE (Processing Element, processing unit) array of the hardware accelerator. When scheduling, the input image row is longer in the X dimension direction and shorter in the Y dimension direction. When performing operation processing by MAC, the overlapping pixel row between adjacent image rows in the Y dimension direction is processed by MAC operation processing (re-computation). The overlapping part between the row tiles of the image row in the X dimension direction is cached and immediately used in the next row tile, that is, the operation result of the non-overlapping part of the next row tile is combined as the MAC operation processing result of the next row tile. In this way, on the one hand, part of the data is re-computed and part of the data is cached, so as to balance the on-chip cache and the re-computation amount of the overlapping part between the row tiles, and avoid frequent read and write operations to the off-chip memory (such as external global cache). On the other hand, the MAC operation processing is performed in units of row tiles, and the MAC operation capability of the PE is fully utilized except for the left and right row tiles, thereby avoiding the problem of decreasing receptive field width layer by layer in the traditional pyramid layer integration data flow processing mode, and avoiding the decrease of PE utilization rate. Thus, the computing resources of the PE are fully utilized, and the computing delay and processing power consumption caused by frequent data interaction between the hardware accelerator and the off-chip memory are avoided. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0013] Figure 1 Structure diagram of exemplary hardware accelerator for implementing the embodiments of the present application;

[0014] Figure 2 Diagram of row cut of an image according to an embodiment of the present application;

[0015] Figure 3 Diagram of writing image rows to an external global cache in a sliding window manner according to an embodiment of the present application;

[0016] Figure 4 Diagram of row cut processing procedure according to an embodiment of the present application;

[0017] Figure 5 Diagram of inter-tile data scheduling procedure according to an embodiment of the present application;

[0018] Figure 6 Diagram of MAC operation processing procedure according to an embodiment of the present application;

[0019] Figure 7 Diagram of relationship between weight data and image data / feature data in a MAC operation processing procedure according to an embodiment of the present application;

[0020] Figure 8 Structure block diagram of a processor according to an embodiment of the present application;

[0021] Figure 9 Structure block diagram of a chip according to an embodiment of the present application;

[0022] Figure 10 Structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make persons skilled in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. All other embodiments obtained by persons skilled in the art based on the embodiments in the embodiments of the present application shall belong to the scope of protection of the embodiments of the present application.

[0024] The specific implementation of the embodiments of the present application will be further described below with reference to the drawings in the embodiments of the present application.

[0025] Hardware accelerators can accelerate various computations, such as convolution calculations. Since convolution calculations are widely used in tasks such as classification and image processing, hardware accelerators can also be widely applied to these tasks. In this example, the hardware accelerator can be applied to image enhancement tasks, leveraging the large data volume and computational complexity of the feature data in image enhancement tasks to achieve hardware acceleration. However, it should be noted that the application is not limited to image enhancement tasks; other convolution calculation tasks with large data volumes and computational complexity are also applicable to the solutions in this embodiment.

[0026] Below, we will first combine Figure 1 The hardware accelerator provided in the embodiments of this application will be described. Figure 1 An exemplary hardware accelerator applicable to embodiments of this application is shown, such as... Figure 1 As shown, this hardware accelerator interacts with other devices via a bus, such as... Figure 1 The ISP (Image Signal Processor), AP (Application Processor), DRAM (Dynamic Random Access Memory), and SRAM (Static Random-Access Memory) shown are, in this example, shared SRAM, which serves as an external global cache interfacing with the hardware accelerator. In this example, a data scheduler is added inside the hardware accelerator, located between the hardware accelerator's internal cache units (including but not limited to the Weight Buffer, Feature Ping-Pong Buffer, and Overlap Buffer) and the hardware accelerator's PE array. This data scheduler efficiently supports the input and output of the image sensor (not shown in the figure) and / or the ISP. In one feasible approach, the ISP scans the image pixel by pixel in raster scan order, performing pixel row input and output; multiple pixel rows form an image row. This improves the efficiency of image data input and output. Based on this, the data scheduler can control the data scheduling between and within image rows by dividing the image into rows and blocks, respectively, in the external global buffer, internal buffer units (Weight Buffer, Feature Ping-Pong Buffer, Overlap Buffer) connected to the hardware accelerator, and between PEs, to complete the layer fusion data stream.

[0027] In addition, such as Figure 1As shown in FIG. 1, the internal cache units of the hardware accelerator can also include a bias cache unit Bias Buffer for caching the bias generated in the convolution calculation, and a residual cache unit Residual Buffer for caching the residual generated in the convolution calculation. It should be noted that, based on the calculation characteristics of the convolution calculation, the data scheduler in this example mainly schedules data in the Weight Buffer, Feature Ping-Pong Buffer, and Overlap Buffer.

[0028] As shown in FIG. 1, the internal cache units of the hardware accelerator can also include a bias cache unit Bias Buffer for caching the bias generated in the convolution calculation, and a residual cache unit Residual Buffer for caching the residual generated in the convolution calculation. It should be noted that, based on the calculation characteristics of the convolution calculation, the data scheduler in this example mainly schedules data in the Weight Buffer, Feature Ping-Pong Buffer, and Overlap Buffer. Figure 1 As shown in FIG. 1, for an image (input image or feature map after convolution), the data scheduler in this example will continuously obtain data from the Weight Buffer, Feature Ping-Pong Buffer, and Overlap Buffer according to the instructions of the instruction engine (Command Engine) in the hardware accelerator, and then schedule the PEs in the PE array to perform corresponding MAC operation processing based on the obtained data, until the image data of the image is completely processed. The results processed by the PEs will be input to the accumulator Accumulator in the hardware accelerator together with the bias and residual corresponding to the image cached in the Bias Buffer and Residual Buffer, and then the processing results will be transmitted to the post-processing engine Post-processing engine of the hardware accelerator for preset post-processing, so as to obtain the convolution processing results of the image.

[0029] In the above process, the Global Buffer is used for data interaction between the ISP and the hardware accelerator, the ISP writes pixel rows in raster scan order and finally forms image rows, and after the hardware accelerator completes the calculation, the image rows are written back; the input and output data of the Feature Ping-Pong Buffer is the row tile after tiling the image rows. In order to cope with the overlapping part between the tiles, the Overlap Buffer caches the calculation results of the right boundary overlapping part between the row tiles of each layer.

[0030] Wherein, tiling means dividing the input image / feature map in X-dimension (horizontal dimension) and Y-dimension (vertical dimension), and a schematic of a tile is shown in FIG. 2. Figure 2 Figure 2 ​In the embodiment, the whole image (input image or feature map) is divided into 4x8 tiles. In the scheduling process of the data scheduler, the Feature Ping-Pong Buffer loads one tile at a time, and then the data scheduler schedules the PE in the PE array to process. In actual application, the tile operation is performed by the data scheduler when it is determined that at least one complete image line has been cached in the external global cache, instead of processing the image in advance. In the embodiment, each PE in the PE array is used for single MAC operation, which can be considered as having a multiplier and an adder to realize MAC operation processing.

[0031] Based on the structure of the hardware accelerator shown in the embodiment, the hardware acceleration scheme of the present application is described below. Figure 1 Based on the structure of the hardware accelerator shown in the embodiment, the hardware acceleration scheme of the present application is described below.

[0032] The hardware accelerator provided in the embodiment at least includes: a PE array, an internal cache unit of the hardware accelerator, and a data scheduler arranged between the PE array and the internal cache unit.

[0033] The data scheduler is configured to: sequentially obtain a plurality of image lines to be processed from the internal cache unit, and schedule the PE array to sequentially perform MAC operation processing on the plurality of image lines, wherein there are overlapping pixel lines between adjacent image lines, and the overlapping pixel lines are subjected to MAC operation processing in the adjacent image lines to which the overlapping pixel lines belong; and in the process of performing MAC operation processing on each image line, schedule the PE in the PE array that processes the current image line to perform MAC operation processing on a plurality of row tiles contained in each image line in a tile unit, wherein for adjacent row tiles, cache the operation result of the overlapping part of the previous row tile with the next row tile, and combine the operation result of the non-overlapping part of the next row tile as the MAC operation processing result of the next row tile.

[0034] From a macro perspective, the hardware accelerator processes related operations in units of images, such as image enhancement processing. However, at a slightly microscopic level, the hardware accelerator in the embodiment processes images in units of image lines. For example, the hardware accelerator in the embodiment is configured to perform MAC operation processing on each image line in the image in a tile unit. Figure 2As shown, it is divided into 4 image rows. It should be noted that the image row is the unit of image processing, and usually includes multiple pixel rows in an image row. In the traditional way, after processing an image row, the new image row is loaded and processed, which will cause the interruption of loading and processing, thereby affecting the processing efficiency. Therefore, in the embodiment of the present application, a sliding window method is used to load the image row. In this way, the sliding window moves at a certain step (usually in pixel row units), and thus there is an overlapping pixel row part between the image rows formed by the two sliding windows. At a more microscopic level, when the PE array performs specific operation processing, each PE performs processing respectively, and each PE realizes it in the row tile unit of the image row. Specifically, in the embodiment of the present application, it can be considered that each PE is provided with a multiplier-accumulator for MAC operation processing, and the multiplier-accumulator performs MAC operation processing on the Tile, and other parts of the processing except the MAC operation processing can be realized by other parts of the PE. In the embodiment of the present application, the processing realized by the other parts is not limited. For example, Figure 2 As shown, it includes 4 image rows, and each image row includes 8 Tiles. Based on the convolution operation property of the convolution kernel, in the front and back two convolution operations, there will be repeated calculation of the data in the adjacent two Tiles, usually the data of the boundary part of the adjacent Tile. Therefore, the hardware accelerator can process the data of the overlapping pixel row part and the repeated calculation Tile boundary part, so as to improve the speed and efficiency of the whole image enhancement processing, and achieve the effect of hardware acceleration.

[0035] Specifically, in the embodiment of the present application, the ISP writes the pixel row into the external global cache in the raster scan order and forms the image row. The hardware accelerator interfaces with the external global cache, so that the multiple image rows written in the raster scan order can be obtained through the external global cache. Thus, the row-first input and processing of the image are realized. Also, because the raster scan order is used, the width of the image row is much larger than the length. In actual application, the data scheduler can start data scheduling after identifying that enough pixel rows (such as 10-30 pixel rows) can constitute at least one image row. The number of pixel rows can be parameterized designed, so that at least one image row output can be obtained after MAC operation processing.

[0036] After a complete image row is cached, the data scheduler splits the image row cached in the external global buffer to obtain a plurality of row tiles corresponding to the image row. The splitting of the image row is a tiling operation, which splits the image row in the X dimension direction, so that the length of each row tile Tile obtained after splitting exactly fills the PE processing the current image row in the X dimension direction, so as to improve the utilization of the PE as much as possible. The tiling number can be parameterized to match the data amount of the Tile that can be processed by the PE. Therefore, the size of the Tile can be configured by those skilled in the art according to the actual situation, so that the scheme of the embodiments of the present application can be adapted to a plurality of different Tile sizes. After the tiling operation is completed, the data scheduler loads the first Tile from the Feature Ping-Pong Buffer to schedule to an idle PE for MAC operation, and writes the output obtained after the PE processes an image row, i.e., the operation result, back to the Global Buffer. Initially, the tiling operation is performed on the image row of the input image, and in the subsequent process, the tiling operation is performed on the feature map obtained after convolution.

[0037] To facilitate input, in a feasible manner, a sliding window mode can be used to write the image row to the external global buffer. For example, as shown in FIG. 6, Figure 3 Figure 3 In FIG. 6, the upper dashed box shows the image row written to the external global buffer for the first time in the sliding window mode. For ease of distinction, the image row data is shown as sliding window 0 in FIG. 6. Figure 3 The sliding window slides at a preset step size, Figure 3 In FIG. 6, the image row written to the external global buffer for the second time in the sliding window mode is shown in the lower dashed box. For ease of distinction, the image row data is shown as sliding window 1 in FIG. 6. Figure 3 As can be seen, there is partially overlapping data in the image rows corresponding to the sliding window 0 and the sliding window 1, i.e., partially overlapping pixel rows. This part of the overlapping pixel rows is calculated whether in the MAC operation processing of the image row corresponding to the sliding window 0 or in the MAC operation processing of the image row corresponding to the sliding window 1. That is, the overlapping pixel rows between adjacent image rows are MAC operated in the adjacent image rows to which they belong. Therefore, for the final row tile processing in the Tile unit, the overlapping part of the Tile in the X dimension is processed in the form of recalculation, thereby reducing the demand for on-chip cache and avoiding the power consumption caused by frequent interaction between the on-chip cache and the off-chip cache.

[0038] ​In one possible implementation, the internal cache unit comprises at least an image data unit for caching image rows in units of row tiles. For example, the Feature Ping-Pong Buffer as shown in Figure 1 In this case, the data scheduler sequentially obtaining the image rows to be processed from the internal cache unit can be implemented as: the data scheduler sequentially obtaining the row tiles corresponding to the image rows from the image data unit. This way effectively utilizes the structure of the existing hardware accelerator, and further hardware acceleration can be achieved with minor modification of the hardware accelerator.

[0039] For example, after the tiling operation of a certain image row is completed, the data scheduler schedules the Feature Ping-Pong Buffer to load the first Tile of the image row and dispatch it to the PE for operation processing. At the same time, the Feature Ping-Pong Buffer continues to load other Tiles of the image row in the order of Tiles. The processing of the Tile by the PE can be regarded as one convolution processing of the convolution kernel. Since the convolution kernel moves by a certain step size when performing convolution on the image, there is an overlap between the data of two convolution operations, and the amount of data in the overlap can be determined according to the size of the convolution kernel and the moving step size of the convolution kernel. In the present example, after the PE performs MAC operation processing on the previous Tile, it caches the result of the MAC operation processing of the overlapping part of the previous Tile and the next Tile for use in the MAC operation processing of the next Tile.

[0040] When the internal cache unit comprises an overlap cache unit, for adjacent row tile Tiles, the operation result of the overlapping part of the previous row tile Tile and the next row tile Tile can be implemented as: determining the overlapping part of the adjacent row tile Tiles according to the moving step size of the convolution kernel; and caching the MAC operation result of the overlapping part to the overlap cache unit when performing MAC operation processing on each row tile Tile by the PE. For example, the overlap cache unit can be the OverlapBuffer as shown in Figure 1

[0041] An example of the above process is shown in Figure 4 The MAC operation processing process for 4 image rows is shown in Figure 4 For each image row, as shown in Figure 4 ​As shown in FIG. 2, each PE processes in the unit of row tile. Taking the first image row as an example, the first tile of the first image row, the MAC operation result of the slant line part of the right boundary of the tile is overlapped with the second tile, and the MAC operation result of this part is cached in the Overlap Buffer. When the PE performs MAC operation on the second tile, the cached data in the Overlap Buffer is read as part of the MAC operation result of the second tile, as shown in the gray part of the left boundary of the second tile in FIG. 2. Figure 4 Meanwhile, the slant line part of the right boundary of the second tile is overlapped with the third tile, and after the MAC operation result of this part is obtained, it is cached in the Overlap Buffer for use in the MAC operation of the third tile, as shown in the gray part of the left boundary of the third tile. In this way, the MAC operation of the last tile of the current image row is performed. That is, until the MAC operation of all tiles of the image row in the Global Buffer is completed, the MAC operation result of the image row is obtained. Then, the result is scheduled and cached in the Global Buffer, and then output to the ISP for further processing.

[0042] It should be noted that while the PE performs the above-mentioned tile operation of the image row, the image row reading operation of the external global cache and the image row caching operation of the image data unit are also performed synchronously. Therefore, the data scheduler will also obtain the newly cached image row (in the unit of row tile) from the internal cache unit while scheduling the PE to perform the MAC operation of the image row. There is an overlapping pixel row between the newly cached image row and the previously cached image row (such as the overlapping part between the sliding window 0 and the sliding window 1 in FIG. 2). Figure 3 As mentioned above, the overlapping pixel row is involved in the MAC operation in the previously cached image row and the newly cached image row, respectively. As shown in FIG. 2, Figure 4 As shown in the gray part of the leftmost of the 2nd-4th image row in FIG. 2. In this way, different operations can be performed in parallel, further improving the speed and efficiency of the hardware accelerator in image processing such as image enhancement processing.

[0043] In a specific implementation, when performing MAC operation on the image row, the overlapped pixel row in the previously cached image row can be cached in the register of the PE array for use in the MAC operation of the newly cached image row. Specifically, the overlapped pixel row cached in the register is also cached in the unit of Tile. Since the overlapped pixel row is divided into multiple parts corresponding to the Tile according to the division of the Tile, and the PE performs operation in the unit of Tile, the part of the overlapped pixel row cached in the register is the part corresponding to the Tile currently processed by the PE. In this way, on the one hand, the number of on-chip caches can be saved; on the other hand, the data cached in the register can be applied to the processing of the next image row more quickly.

[0044] An example of the above Tile-based inter-block data scheduling process is shown in the following table: Figure 5 As shown in the table, the process can include the following steps: Figure 5

[0045] Step A: The external global cache Global Buffer caches the pixel row data output by the ISP.

[0046] Step B: The data scheduler determines whether enough rows of pixel rows have been cached in the Global Buffer to form at least one image row. If yes, after the image row is formed, the data scheduler tiles the image row formed in the Global Buffer to obtain multiple Tiles. If no, return to step A.

[0047] In an example, the enough rows can be 10-30 rows.

[0048] Step C: The data scheduler loads the Tile to the internal cache unit (specifically, the image (feature) data cache unit) of the hardware accelerator and schedules it to the PE for operation.

[0049] The operation process is as described above and will not be repeated here.

[0050] Step D: The data scheduler determines whether the Tile currently processed by the PE is the last Tile in the horizontal direction of the image row to which it belongs. If no, the data scheduler caches the MAC operation result of the overlapping part (right boundary) of the Tile and the next Tile, then increments the Tile counter (tile = tile + 1) and returns to step C. If yes, execute step E.

[0051] Step E: The data scheduler writes the MAC operation result of the image row back to the external global cache Global Buffer and returns to step B. ​

[0052] It can be seen that in the scheme provided in the embodiments of the present application, the hardware accelerator can support input and output of image rows in raster scan order; different Tile sizes can be supported; and in combination with the above-mentioned inter-block data scheduling for row tiles, the data scheduler can ensure balance of on-chip cache and overlapping part of inter-Tile recalculation under the premise of priority row input and output and high utilization of MAC, solving the problem of frequent off-chip memory read and write of the hardware accelerator when performing a computing imaging task. Moreover, due to row priority, the X dimension direction of the input image row is relatively long, so for the overlapping part in the X dimension direction between tiles, the data scheduler adopts a recalculation strategy, and for the overlapping part in the Y dimension direction between tiles which is relatively short, the data scheduler stores the data of the overlapping part layer by layer and immediately uses the data in the next Tile calculation, so that a smaller on-chip cache can be used to reduce the recalculation amount, and at the same time, the problem of decreasing receptive field width layer by layer in the traditional pyramid layer fusion data flow mode can be solved, and there will be no problem of decreasing utilization of the computing unit except for the left and right tiles. In the traditional pyramid layer fusion data flow mode, with the increase of the depth direction of the neural network, the size of the feature map will gradually decrease, and the data mapped to the PE of the hardware accelerator will also gradually decrease, resulting in a decrease in utilization of the hardware accelerator, especially the PE, and the overlapping part of the feature map between different pyramids will also cause a large amount of recalculation. Through the above-mentioned mode in the scheme of the present application, the problem is effectively avoided.

[0053] In summary, according to the scheme provided in the embodiments of the present application, a data scheduler is added in the hardware accelerator that can be used for image enhancement tasks to schedule data for MAC multiplication and addition operation processing between the internal cache unit and the PE of the hardware accelerator. When scheduling, the row input mode is adopted, so that the input image row is longer in the X dimension direction and shorter in the Y dimension direction. When performing operation processing by MAC, the overlapping pixel row between adjacent image rows in the Y dimension direction is processed by MAC (recalculated); and the overlapping part between the row tiles of the image row in the X dimension direction is cached and immediately used in the next row tile, that is, the operation result of the non-overlapping part of the next row tile is combined as the MAC operation processing result of the next row tile. In this way, on the one hand, part of the data is recalculated and part of the data is cached during calculation, so as to balance the on-chip cache and the recalculation amount of the overlapping part between the row tiles, and avoid frequent read and write operations to the off-chip memory (such as external global cache, etc.); on the other hand, the MAC operation processing is performed in units of row tiles, and the operation capability of the PE is fully utilized except for the left and right row tiles, thereby avoiding the problem of decreasing receptive field width layer by layer in the traditional pyramid layer integration data flow processing mode, and avoiding the decrease of PE utilization. In this way, the calculation resources of the PE are fully utilized, and the calculation delay and processing power consumption caused by frequent data interaction between the hardware accelerator and the off-chip memory are avoided.

[0054] However, in order to further improve the data processing efficiency of the hardware accelerator, the PE array can also be improved. In a feasible manner, the PE array can be divided into sub-arrays by the data scheduler according to the currently used convolution kernel size. Optionally, the PE array can be divided into sub-arrays from the height direction, such as 1 / 3 / 5 / 7 / 9 height sub-array grouping, and the feature data and weight data are input to the PE array. The height of the PE array is a multiple of 30, and the width is a configurable parameter.

[0055] On the basis of dividing the PE array into sub-arrays, the data scheduler can schedule the PE array to sequentially perform MAC operation processing on multiple image rows, which can be implemented as follows: according to the size of the convolution kernel, the PE array is divided into sub-arrays from the height direction to obtain multiple row groups; and the PE array is scheduled to sequentially perform MAC operation processing on multiple image rows through the multiple row groups. In this way, the MAC resources can be fully utilized, and parallel calculation can be performed to further improve the calculation efficiency. Further, the scheduling of the PE array to sequentially perform MAC operation processing on multiple image rows through the multiple row groups includes: for each image row, the PE array is scheduled to obtain weight row group data corresponding to the image row through the multiple row groups; and based on the weight row group data, the image row is processed by MAC to obtain corresponding image feature data.

[0056] Since the hardware accelerator adopts the form of PE Array, it can be further divided in the height direction to form multiple sub-arrays, each of which can be considered as a row group, and each row group can perform operation processing on multiple Tiles of one image row at a time. Thus, the resource utilization rate of the PE Array is further improved, and the overall efficiency of image enhancement processing is improved. Because in the convolution calculation, each image row corresponds to a corresponding weight, and the weight corresponding to each Tile in the image row is the same as the weight of the image row to which it belongs. In the case where the PE Array is divided into row groups, after obtaining the input image data / feature data (Tile) and the weight data corresponding to the image data / feature data, operation processing can be performed. In the embodiments of the present application, “ / ” can represent the relationship of “or”.

[0057] In one feasible manner, when the internal cache unit includes a weight cache unit (such as the WeightBuffer shown in Figure 1 , the weight data is cached in the weight cache unit, and then the data scheduler can group the weight data cached in the weight cache unit by rows according to the size of the convolution kernel to obtain multiple groups of weight row group data. Correspondingly, the data scheduler schedules the PE array to obtain the weight row group data corresponding to the image row through multiple row groups, and the weight group data is obtained from the multiple groups of weight row group data cached in the weight cache unit.

[0058] On this basis, the MAC operation processing on the image row based on the weight row group data can be implemented as follows: scheduling the weight row group data to be input to the multiple row groups of the PE array along the row direction, and scheduling the row tiles of the image row to be input to the multiple row groups of the PE array along the diagonal direction; and performing MAC operation processing on the image row based on the input of the multiple row groups.

[0059] An exemplary process of performing MAC operation processing on the image row through the multiple row groups of the PE array is shown in Figure 6 .

[0060] In the example shown in Figure 6 , the data scheduler groups the convolution kernel in the row direction to form multiple convolution rows. After forming the multiple convolution rows, the weight row group data corresponding to the weight data input each time, and the image data / feature data input are the data in the sliding window corresponding to the weight data on the input image / feature map. Further, the data scheduler schedules the weight data to be input along the row direction of the row group, the data in the sliding window to be input along the diagonal direction, and the corresponding partial sum to be accumulated along the column direction.

[0061] An exemplary correspondence between the weight data and the image data / feature data is shown inFigure 7 As shown, Figure 7 For example, a 3*3 convolution kernel is used to convolve an input feature map including 5 image rows, and a 3*3 output feature map is obtained.

[0062] For example, a 3*3 convolution kernel is used to convolve an input feature map including 5 image rows, and a 3*3 output feature map is obtained.

[0063] For example, a 3*3 convolution kernel is used to convolve an input feature map including 5 image rows, and a 3*3 output feature map is obtained.

[0064] For example, a 3*3 convolution kernel is used to convolve an input feature map including 5 image rows, and a 3*3 output feature map is obtained.

[0065] For example, a 3*3 convolution kernel is used to convolve an input feature map including 5 image rows, and a 3*3 output feature map is obtained. Figure 7In the specific implementation, the second row of the input feature map is input into the "PE Group 1" and the "PE Group 3" along the diagonal direction; the third row of the input feature map is input into the "PE Group 2", the "PE Group 4" and the "PE Group 6" along the diagonal direction; and the fourth row of the input feature map is input into the "PE Group 5" and the "PE Group 7" along the diagonal direction. In addition, the MAC operation is performed in parallel in each row group. In this way, the efficiency of the MAC operation processing is improved.

[0066] After the operation on the image row (including a plurality of tiles) corresponding to a sliding window is completed, the obtained feature data is fixed, and a new set of weight data is scheduled by the data scheduler for operation until all the weight data is used up. For the overlapping pixel rows between image rows, in terms of tile processing, it can be considered that the overlapping data parts of different sliding windows in the tile will not be repeatedly scheduled, but will be stored in the registers of the PE array and reused in the PE array.

[0067] Through the above processing, different convolution dimensions can be flexibly mapped on the row groups of the PE array. In combination with the processing data flow of each row group of the PE array for the image row and the sufficient reuse of the input data, the access to the on-chip memory and the off-chip memory in the convolution operation process can be effectively optimized, and the high utilization rate of the PE array for any workload can be ensured.

[0068] Based on the hardware accelerator described above, an embodiment of the present application further provides a processor, Figure 8 A structural block diagram of the processor is shown. As shown in Figure 8 The processor can include the hardware accelerator as described above.

[0069] In a feasible manner, the hardware accelerator is a neural network hardware accelerator.

[0070] In addition, in another feasible manner, the processor can be implemented as an NPU (Neural Processing Unit, neural network processor).

[0071] In addition, an embodiment of the present application further provides a chip, Figure 9 A structural block diagram of the chip is shown. As shown in Figure 9 The chip can include the processor as described above, for example, the NPU described above.

[0072] Further, an embodiment of the present application further provides an electronic device, which can include the chip as described above.

[0073] In a specific example, as Figure 10As shown, it is a structural schematic diagram of an electronic device, and the embodiments of the present application do not limit the specific implementation of the electronic device.

[0074] As shown, the electronic device can include: a chip 502, a communications interface 504, a memory 506, and a communications bus 508. Figure 10

[0075] Among them:

[0076] The chip 502, the communications interface 504, and the memory 506 complete the communication with each other through the communications bus 508.

[0077] The communications interface 504 is configured to communicate with other electronic devices or servers.

[0078] The chip 502 is a chip as described above, in which a hardware accelerator is deployed, and which can provide hardware acceleration function for image processing such as image enhancement processing.

[0079] The chip 502 can be an NPU, which can independently perform related processing, or form a heterogeneous system with a CPU and the like to implement tasks related to image processing, especially image enhancement processing.

[0080] The memory 506 is configured to store programs and data required for application running in the electronic device. The memory 506 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0081] The specific process of the electronic device of the embodiment to implement corresponding functions through the chip 502 can refer to the description in the above embodiments, and has corresponding beneficial effects, which will not be described here.

[0082] ​In summary, by means of the scheme of the embodiments of the present application, a data scheduler is added to the hardware accelerator, which can support data input and output in raster scan order; in combination with the inter-block data scheduling of the PE when processing in the unit of Tile, different Tile sizes can be supported, and the data scheduler can ensure the balance of the on-chip cache and the overlapping part of the re-computation between Tiles under the premise of the priority row input and row output and the high utilization of the PE, solve the frequent off-chip memory read-write problem of the hardware accelerator when performing the computing imaging task; and due to the row priority feature, the image row and the corresponding Tile are relatively long in the X dimension direction, so for the overlap in the Y dimension direction between Tiles, the data scheduler adopts the re-computation strategy, and for the shorter overlap in the X dimension direction between Tiles, the data scheduler stores the data of the overlapping part layer by layer in the cache and uses the data in the next Tile computation, which can reduce the re-computation amount with a smaller on-chip cache and solve the problem of the field of view width decreasing layer by layer in the pyramid layer fusion data flow, so there is no problem of the PE utilization rate decreasing except for the Tiles on the left and right sides; in combination with the intra-block data scheduling of the PE for the Tile, different convolution dimensions are flexibly mapped on the PE array, the optimized row fixed data flow and the full reuse of the input feature data are used to maximize the optimization of the access to the on-chip memory and the off-chip memory in the convolution operation process, and the high utilization of the PE for any workload is ensured.

[0083] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, so as to achieve the purpose of the embodiments of the present application.

[0084] The above implementation manners are only used to illustrate the embodiments of the present application, but not limit the embodiments of the present application, and the ordinary skilled in the technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, so all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A hardware accelerator comprising: A processing element (PE) array, an internal cache unit of the hardware accelerator, and a data scheduler disposed between the PE array and the internal cache unit; The data scheduler is configured to: sequentially obtain a plurality of image rows to be processed from the internal cache unit, and sequentially schedule the PE array to perform MAC operation processing on the plurality of image rows, wherein there are overlapping pixel rows between adjacent image rows, and the overlapping pixel rows are subjected to MAC operation processing in both of the adjacent image rows; and, during the MAC operation processing on each image row, schedule a PE in the PE array that processes a current image row to perform MAC operation processing on a plurality of row tiles included in each image row in a tile unit, wherein, for adjacent row tiles, cache the operation result of an overlapping part of a previous row tile with a next row tile, and combine the operation result of a non-overlapping part of the next row tile as the MAC operation processing result of the next row tile.

2. The hardware accelerator of claim 1, wherein, The hardware accelerator is connected with an external global cache to obtain the plurality of image rows written in a raster scan order through the external global cache.

3. The hardware accelerator of claim 2, wherein, The data scheduler is further configured to split the image rows cached in the external global cache to obtain a plurality of row tiles corresponding to each image row.

4. The hardware accelerator of any of claims 1-3, wherein, The internal cache unit at least includes an image data unit configured to cache the row tiles of the image rows in a row tile unit; The sequentially obtaining a plurality of image rows to be processed from the internal cache unit includes sequentially obtaining the row tiles corresponding to each image row from the image data unit.

5. The hardware accelerator of claim 4, wherein, The internal cache unit further includes an overlapping cache unit; The, for adjacent row tiles, caching the operation result of an overlapping part of a previous row tile with a next row tile includes determining the overlapping part of adjacent row tiles according to a moving step of a convolution kernel, and caching the MAC operation result of the overlapping part to the overlapping cache unit when the PE performs MAC operation processing on each row tile.

6. The hardware accelerator of any of claims 1-3, wherein, The data scheduler is further configured to, while scheduling the PE array to sequentially perform MAC operation processing on the plurality of image rows, obtain a newly cached image row from the internal cache unit, wherein there are overlapping pixel rows between the newly cached image row and a previously cached image row, and the overlapping pixel rows are subjected to MAC operation processing in both of the previously cached image row and the newly cached image row.

7. The hardware accelerator of claim 6, wherein, When performing MAC operation processing on the image rows, the overlapping pixel rows in the previously cached image row are cached to a register of the PE array for use by the newly cached image row in MAC operation processing.

8. The hardware accelerator of any of claims 1-3, wherein, The scheduling the PE array to sequentially perform MAC operation processing on the plurality of image rows includes: dividing the PE array into a plurality of row groups in a height direction according to the size of the convolution kernel; scheduling the PE array to sequentially perform MAC operation processing on the plurality of image rows through the plurality of row groups.

9. The hardware accelerator of claim 8, wherein, The scheduling the PE array to sequentially perform MAC operation processing on the plurality of image rows through the plurality of row groups includes: For each image row, the plurality of row groupings are obtained, and weight row grouping data corresponding to the image row is obtained; Based on the weight row grouping data, MAC operation processing is performed on the image row to obtain corresponding image feature data.

10. The hardware accelerator of claim 9, wherein, The internal cache unit further includes a weight cache unit; The data scheduler is further configured to group the weight data cached in the weight cache unit by rows according to the size of the convolution kernel to obtain a plurality of sets of weight row grouping data.

11. The hardware accelerator of claim 9, wherein, The MAC operation processing on the image row based on the weight row grouping data includes: The weight row grouping data is scheduled to be input to a plurality of row groupings of the PE array along a row direction, and the row tiles of the image row are scheduled to be input to a plurality of row groupings of the PE array along a diagonal direction; Based on the input of the plurality of row groupings, MAC operation processing is performed on the image row.

12. A processor comprising: The hardware accelerator of any one of claims 1-11.

13. A chip comprising: The processor of claim 12.

14. An electronic device comprising: The chip of claim 13.

Images