A data processing method and device, and a storage medium

Abstract

The embodiment of the application provides a data processing method and device, and a storage medium, comprising: acquiring hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model respectively; the deep learning accelerator is used for running the neural network model; comparing memory access times in a plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters to obtain a comparison result; and determining a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes based on the comparison result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a data processing method and device and a storage medium. BACKGROUND

[0002] Deep learning can enable machines to imitate human activities such as audio-visual and thinking, and solve many complex pattern recognition problems. The neural network used by deep learning has made great progress in many technical fields such as natural language processing, image recognition, speech recognition, and data mining. At present, for a deep learning accelerator, a fixed data scheduling mode is adopted when executing a neural network model, which results in a large difference in performance of each algorithm layer when executing, and further increases hardware power consumption. SUMMARY

[0003] The embodiments of the present application provide a data processing method and device and a storage medium, which can reduce hardware power consumption.

[0004] The technical solution of the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a data processing method, which comprises:

[0006] Respectively acquiring a hardware parameter of a deep learning accelerator and an algorithm layer parameter corresponding to a first algorithm layer in a neural network model; the deep learning accelerator is used to run the neural network model;

[0007] According to the hardware parameter and the algorithm layer parameter, comparing the memory access times under a plurality of data scheduling modes to obtain a comparison result;

[0008] Based on the comparison result, determining a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes.

[0009] In a second aspect, the embodiments of the present application provide a data processing device, wherein the data processing device is deployed with the above deep learning accelerator; the deep learning accelerator is an array of computing units;

[0010] A set of convolution kernel data corresponding to one channel and window feature data in a sliding window corresponding to one channel are sequentially loaded in each group of computing units along a first direction in the array of computing units;

[0011] The computing unit array sequentially processes a group of convolution kernel data corresponding to a group of channels in a convolution kernel and a group of window feature data in a group of channels corresponding to a sliding window in each group of computing units along the second direction in the computing unit array; and a group of computing units along the second direction in the computing unit array are cascaded to perform an accumulation operation on the output feature data of each computing unit in the group of computing units, to obtain an output feature data corresponding to the group of computing units.

[0012] In a third aspect, an embodiment of the present application provides a data processing apparatus, which comprises:

[0013] An obtaining unit is configured to obtain a hardware parameter of a deep learning accelerator and an algorithm layer parameter corresponding to a first algorithm layer in a neural network model respectively; the deep learning accelerator is configured to run the neural network model;

[0014] A comparing unit is configured to compare memory access times in a plurality of data scheduling modes according to the hardware parameter and the algorithm layer parameter, to obtain a comparison result;

[0015] A determining unit is configured to determine a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes based on the comparison result.

[0016] In a fourth aspect, an embodiment of the present application provides a data processing apparatus, which comprises: a processor, a memory and a communication bus; and the processor implements the data processing method as described above when executing a running program stored in the memory.

[0017] In a fifth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the data processing method as described above.

[0018] This application provides a data processing method, apparatus, and storage medium. The method includes: acquiring hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model; using the deep learning accelerator to run the neural network model; comparing memory access counts under various data scheduling methods based on the hardware parameters and algorithm layer parameters to obtain comparison results; and determining a target data scheduling method for the first algorithm layer on the deep learning accelerator based on the comparison results. By adopting the above implementation scheme, the data flow under various data scheduling methods is analyzed based on the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer, obtaining a comparison result of memory access counts among various data scheduling methods. Since memory access counts greatly affect the energy consumption of data transmission, the target data scheduling method determined based on the comparison results takes into account the memory access counts of the first algorithm layer on the deep learning accelerator, which can improve the performance of the algorithm layer executing on the deep learning accelerator and thus reduce hardware power consumption. Attached Figure Description

[0019] Figure 1 A convolutional diagram illustrating a weighted data reuse method. Figure 1 ;

[0020] Figure 2 A convolutional diagram illustrating a weighted data reuse method. Figure 2 ;

[0021] Figure 3 A convolutional diagram illustrating a weighted data reuse method. Figure 3 ;

[0022] Figure 4 A convolutional diagram illustrating a weighted data reuse method. Figure 4 ;

[0023] Figure 5 A convolutional diagram illustrating a method of reusing input feature data. Figure 1 ;

[0024] Figure 6 A convolutional diagram illustrating a method of reusing input feature data. Figure 2 ;

[0025] Figure 7 A convolutional diagram illustrating a method of reusing input feature data. Figure 3 ;

[0026] Figure 8 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0027] Figure 9 A schematic diagram of a 4×4 computing unit array provided for an embodiment of this application;

[0028] Figure 10 A unit structure schematic diagram of a computing unit provided for an embodiment of the present application;

[0029] Figure 11 A schematic diagram of mapping convolution operation to a computing unit provided for an embodiment of the present application;

[0030] Figure 12 A schematic diagram of mapping data stream to a deep learning accelerator in weight data multiplexing provided for an embodiment of the present application;

[0031] Figure 13 A schematic diagram of mapping data stream to a deep learning accelerator in input feature data multiplexing provided for an embodiment of the present application;

[0032] Figure 14 A structure schematic diagram of a data processing apparatus 1 provided for an embodiment of the present application Figure 1 ;

[0033] Figure 15 A structure schematic diagram of a data processing apparatus 1 provided for an embodiment of the present application Figure 2 .DETAILED DESCRIPTION

[0034] It should be understood that the specific embodiments described herein merely exemplify the application. They are not intended to limit the application.

[0035] At present, data scheduling modes include weight data multiplexing and input feature data multiplexing. The two data scheduling modes are described in detail below.

[0036] For the weight data multiplexing mode, the multiplexing criterion is to first read a group of channels in a convolution kernel into the computing unit, and then start traversing the input feature image data. The feature image window is traversed along the horizontal direction or the vertical direction. After the traversal is completed, the next channel group is switched to. After all channels in a convolution kernel are traversed, the convolution kernel is switched.

[0037] Specifically, first, the channel 0 of the convolution kernel 0 is fixed. Then, the sliding window is started to be traversed in the input feature data corresponding to the channel 0. As shown in Figure 1 , the sliding can be started from the leftmost sliding window in the input feature data. Until all sliding windows in the input feature data corresponding to the channel 0 complete the calculation, as shown in Figure 2 , at this time, for the output feature data, an intermediate result 1 is generated. Then, as shown in Figure 3 , the channel 1 of the convolution kernel 0 is switched to, and the input feature data corresponding to the channel 1 is traversed to generate an intermediate result 2. The intermediate result 1 and the intermediate result 2 are superimposed to obtain a superimposed result 1. In turn, as shown in Figure 4As shown in FIG. 6, until the last channel N of the convolution kernel 0 is switched, and the input feature data corresponding to the channel N is traversed, an intermediate result N is generated. The intermediate result N and the superimposed result N-1 are superimposed to obtain a group of output feature data corresponding to the convolution kernel 0. Then, the above operation is performed again by fixing the convolution kernel 1.

[0038] For the input feature data multiplexing mode, the multiplexing criterion is to first fix a channel group data of the input feature data, load it into the calculation unit array, then traverse the channel groups corresponding to different convolution kernels, after the traversal is completed, switch to the next channel group, until the channel group is completed, and then switch to the next convolution kernel, until all the convolution kernels are traversed.

[0039] Specifically, first, the sliding window at the top left corner of the input feature data corresponding to the channel 0 is fixed, then the convolution kernel 0 of the channel 0 is started, the weight multiplexing mode is referred to as Figure 1 , the convolution kernels of the channel 0 are traversed in turn, until all the convolution kernels of the channel 0 are traversed, the Figure 5 is referred to, and an intermediate result 1 is obtained. Then, the channel 1 is switched, and the sliding window at the top left corner of the input feature data corresponding to the channel 1 is fixed, then the convolution kernel 0 of the channel 1 is started, the convolution kernels of the channel 1 are traversed in turn, until all the convolution kernels of the channel 1 are traversed, the Figure 6 is referred to, and an intermediate result 2 is obtained. The intermediate result 1 and the intermediate result 2 are superimposed to obtain a superimposed result 1, and the above operation is performed in turn, as shown in Figure 7 , until the sliding window at the top left corner of the input feature data corresponding to the channel 0 is switched, and the convolution kernel of the channel N is traversed, an intermediate result N is generated. The intermediate result N and the superimposed result N-1 are superimposed to obtain a group of output feature data corresponding to the sliding window at the top left corner of the input feature data. Then, the above operation is performed by fixing the second sliding window of the input feature data.

[0040] As can be seen from the above, different data scheduling modes correspond to different data access modes. To solve the problem that the fixed data scheduling mode leads to large performance difference of each algorithm layer when it is executed, and further increases the hardware power consumption, the embodiment of the present application provides a data processing method, as shown in Figure 8 , which can include the following steps.

[0041] In S101, hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model are acquired respectively. The deep learning accelerator is used to run the neural network model.

[0042] The data processing method provided in the embodiment of the present application is applicable to the scene of running the neural network model on the deep learning accelerator.

[0043] In an embodiment of the present application, the neural network model can be a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), or the like. The specific type of neural network model can be selected according to actual conditions, and the present application does not make specific limitations.

[0044] In an embodiment of the present application, in the process of running the neural network model on the deep learning accelerator, when the deep learning accelerator implements the algorithm logic of the first algorithm layer in the neural network model, the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer are acquired respectively.

[0045] It should be noted that the hardware parameters of the deep learning accelerator are parameters after spatial expansion on the deep learning accelerator, that is, the data granularity that the deep learning accelerator can process at a time in three dimensions of input feature tensors, output tensors, and weight tensors; wherein the data granularity that the deep learning accelerator can process at a time in the input feature tensor dimension includes the number of channels that the deep learning accelerator can process at a time and the number of convolution kernels that the deep learning accelerator can load at a time, the data granularity that the deep learning accelerator can process at a time in the output tensor dimension includes the size information of a feature image output by one computing unit in the deep learning accelerator, and the data granularity that the deep learning accelerator can process at a time in the weight tensor dimension includes the convolution kernel size information that one multiplication accumulation tree in one computing unit can process at a time; the specific type can be selected according to actual conditions, and the present application does not make specific limitations.

[0046] It should be noted that the deep learning accelerator is an array of computing units, in the vertical direction of the array of computing units, each computing unit can process data of one channel of the input feature tensor, if the vertical direction of the array of computing units includes N computing units, then the number of channels that the deep learning accelerator can process at a time is N; in the horizontal direction of the array of computing units, each computing unit can load one convolution kernel data, if the horizontal direction of the array of computing units includes C computing units, then the number of convolution kernels that the deep learning accelerator can load at a time is C.

[0047] It should be noted that for each computing unit in the deep learning accelerator, there are M multiplication accumulation trees, which can process M windows at a time, and each multiplication accumulation tree can process data of one AxA window, and the corresponding convolution kernel size that can be processed at a time is AxA.

[0048] In the embodiment of the present application, the hardware parameters of the deep learning accelerator can at least include: size information of a feature image output by one computing unit in the deep learning accelerator, a number of convolution kernels loaded at one time and a number of channels of the deep learning accelerator; wherein the size information of the feature image output by one computing unit includes width information and height information of the feature image output by one computing unit.

[0049] In the embodiment of the present application, the neural network model includes a plurality of algorithm layers, and each algorithm layer includes a set of algorithm layer parameters, wherein the algorithm layer parameters of a first algorithm layer in the neural network model can at least include: a total number of convolution kernels corresponding to the first algorithm layer, and size information of feature data output by the first algorithm layer. It should be noted that the algorithm layer parameters of the first algorithm layer can also include: a total number of channels required to be processed by the first algorithm layer, and width information and height information of an input feature tensor. Specifically, selection can be made according to actual conditions, and the embodiment of the present application does not make specific limitations.

[0050] S102, comparing the memory access times in the plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters to obtain a comparison result.

[0051] In the embodiment of the present application, the data scheduling modes can include weight data multiplexing and input feature data multiplexing, and the data scheduling modes can also be increased and / or modified according to actual conditions, and the embodiment of the present application does not make specific limitations. In the embodiment of the present application, the weight data multiplexing and the input feature data multiplexing are used for subsequent description.

[0052] In the embodiment of the present application, the memory access times in each data scheduling mode include weight data memory access times and input feature data memory access times, then according to the size information of the feature image output by one computing unit and the size information of the feature data output by the first algorithm layer, a ratio of the weight data memory access times corresponding to the plurality of data scheduling modes is determined; according to the number of convolution kernels loaded at one time by the deep learning accelerator, the total number of convolution kernels corresponding to the first algorithm layer, and the width information of the feature image output by one computing unit, a ratio of the input feature data memory access times corresponding to the plurality of data scheduling modes is determined; the width information of the feature image output by one computing unit is the width information in the size information of the feature image output by one computing unit; the comparison result of the weight data memory access times and the comparison result of the input feature data memory access times are the comparison results obtained by comparing the memory access times in the plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters.

[0053] The hardware parameters of the deep learning accelerator are defined as HOW, HOH, HIC, HOC, HKW, and HKH, wherein HOW is the width information of a feature map output by a calculation unit, HOH is the height information of a feature map output by a calculation unit, HIC is the number of channels that can be processed by the deep learning accelerator at a time, HOC is the number of convolution kernels that can be loaded by the deep learning accelerator at a time, HKW is the width information of a convolution kernel that can be processed by one multiplication and accumulation tree in a calculation unit at a time, HKH is the height information of a convolution kernel that can be processed by one multiplication and accumulation tree in a calculation unit at a time, and HKW and HKH together constitute the size information of a convolution kernel that can be processed by a calculation unit at a time; the algorithm layer parameters of the first algorithm layer are defined as IC, TC, TW, TH, IW, and IH, wherein IC is the total number of channels that need to be processed by the first algorithm layer, TC is the total number of convolution kernels corresponding to the first algorithm layer, TW is the width information of feature data output by the first algorithm layer, TH is the height information of feature data output by the first algorithm layer, TW and TH together constitute the size information of feature data output by the first algorithm layer, IW is the width information of an input feature tensor, and IH is the height information of an input feature tensor.

[0054] For example, in the weight data multiplexing mode, the weight data memory access times are as shown in formula (1),

[0055] HKHxHKHxTCxIC (1)

[0056] In the weight data multiplexing mode, the input feature data memory access times are as shown in formula (2),

[0057]

[0058] In the input feature data multiplexing mode, the weight data memory access times are as shown in formula (3),

[0059]

[0060] In the input feature data multiplexing mode, the input feature data memory access times are as shown in formula (4),

[0061]

[0062] Based on the memory access times calculated above, the comparison result of the weight data memory access times obtained by using formula (1) divided by formula (3) is defined as the weight data memory access times ratio, and is specifically as shown in formula (5),

[0063]

[0064] Based on the calculated access frequency, the comparison result of input feature data access frequency is obtained by using formula (2) divided by formula (4), which is defined as the ratio of input feature data access frequency, as shown in formula (6),

[0065]

[0066] S103, based on the comparison result, determining the target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes.

[0067] Since fewer access frequencies represent lower power consumption, after determining the comparison result, the target data scheduling mode with the least access frequency is determined from the plurality of data scheduling modes based on the ratio of weight data access frequency corresponding to the plurality of data scheduling modes and the ratio of input feature data access frequency corresponding to the plurality of data scheduling modes, as the target data scheduling mode of the first algorithm layer on the deep learning accelerator.

[0068] For example, according to formula (5) and formula (6), for different algorithm layers in the neural network model, in the case of small TW and TH, but large TC, the total access frequency in the weight data reuse mode is larger, and in the case of large TW and TH, but small TC, the total access frequency in the input feature data reuse mode is larger.

[0069] For example, if the ratio of weight data access frequency of formula (5) is less than 1 and the ratio of input feature data access frequency of formula (6) is less than 1 for the first algorithm layer, it can be determined that the access frequency of the weight data reuse mode is less than that of the input feature data reuse mode, that is, the weight data reuse mode can be determined as the target data scheduling mode of the first algorithm layer on the deep learning accelerator hardware architecture. Conversely, if the ratio of weight data access frequency of formula (5) is greater than 1 and the ratio of input feature data access frequency of formula (6) is greater than 1 for the first algorithm layer, it can be determined that the access frequency of the weight data reuse mode is greater than that of the input feature data reuse mode, that is, the input feature data reuse mode can be determined as the target data scheduling mode of the first algorithm layer on the deep learning accelerator hardware architecture. In addition, if the ratio of weight data access frequency of formula (5) is less than 1 and the ratio of input feature data access frequency of formula (6) is greater than 1, or the ratio of weight data access frequency of formula (5) is greater than 1 and the ratio of input feature data access frequency of formula (6) is less than 1, the access frequency of the weight data reuse mode and the access frequency of the input feature data reuse mode need to be calculated according to the above formulas (1)(2)(3)(4) to determine the data scheduling mode with the least access frequency as the target data scheduling mode.

[0070] In summary, when the upper-layer compiler performs data scheduling, the different data multiplexing modes can be switched according to the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to each algorithm layer in the neural network model, so that the data scheduling process is more reasonable.

[0071] Further, after determining the target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes, the input feature data and the convolution kernel data are sequentially traversed according to the target data scheduling mode, and the input feature data and the convolution kernel data traversed each time are loaded into the deep learning accelerator for data processing.

[0072] It should be noted that the process of sequentially traversing the input feature data and the convolution kernel data according to the target data scheduling mode, and loading the input feature data and the convolution kernel data traversed each time into the deep learning accelerator for data processing is described in detail in combination with the composition of the deep learning accelerator below, and will not be described here.

[0073] It can be understood that, according to the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer, the data flow under the plurality of data scheduling modes is analyzed to obtain the memory access frequency comparison result between the plurality of data scheduling modes. Since the memory access frequency greatly affects the energy consumption of data transmission, the target data scheduling mode determined based on the comparison result considers the memory access frequency of the first algorithm layer on the deep learning accelerator, which can improve the performance of the algorithm layer executing on the deep learning accelerator, and further reduce the hardware power consumption.

[0074] Further, before obtaining the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer in the neural network model, the process of designing the deep learning accelerator and fine-tuning the algorithm layer parameters corresponding to the first algorithm layer needs to be performed first. Specifically, a plurality of permutation and combination modes of hardware resources and a plurality of sets of hardware parameters corresponding to the plurality of permutation and combination modes are determined; the initial algorithm layer parameters of the first algorithm layer are obtained; the hardware parameters are determined from the plurality of sets of hardware parameters based on a preset algorithm performance index and a preset hardware performance index, and the initial algorithm layer parameters are adjusted to obtain the algorithm layer parameters corresponding to the first algorithm layer; and the hardware parameters are used to model the deep learning accelerator.

[0075] In this embodiment, the hardware resources can be the total number of computing units. The various arrangements and combinations of hardware resources involve arranging all computing units into multiple computing unit arrays. The hardware parameters corresponding to each arrangement and combination can include the array's width and height, the size information of the feature image output by one computing unit, and the size information of the convolution kernel that a multiply-accumulate tree in one computing unit can process in one operation. For example, if the hardware resources consist of 16 computing units, the computing unit arrays corresponding to these 16 units can include: a 1×16 array, a 2×8 array, a 4×4 array, an 8×2 array, and a 16×1 array.

[0076] In this embodiment, given multiple sets of hardware parameters and initial algorithm layer parameters, hardware parameters can be simultaneously determined from these sets, and the initial algorithm layer parameters can be adjusted until preset algorithm performance indicators and preset hardware performance indicators are met. This yields the algorithm layer parameters corresponding to the first algorithm layer and the hardware parameters corresponding to the deep learning accelerator. Therefore, a balance can be struck between hardware parameters and algorithm layer parameters to determine suitable hardware and algorithm layer parameters, achieving optimal utilization of hardware resources while ensuring algorithm performance.

[0077] It should be noted that, based on the above embodiments, this application also proposes a data processing apparatus, in which the aforementioned deep learning accelerator is deployed, wherein the deep learning accelerator is a computing unit array.

[0078] In the array of computing units along the first direction, each group of computing units sequentially loads a set of convolutional kernel data corresponding to one channel and window feature data within a sliding window corresponding to one channel.

[0079] In each group of computing units along the second direction in the computing unit array, a set of convolution kernel data corresponding to a set of channels in a convolution kernel and a set of window feature data corresponding to a set of channels in a sliding window are processed sequentially; the group of computing units along the second direction in the computing unit array are cascaded to accumulate the output feature data of each computing unit in the group of computing units to obtain one output feature data corresponding to the group of computing units.

[0080] In this embodiment of the application, the data processing device for deploying the deep learning accelerator can be any device with data processing and storage functions, such as tablet computers, mobile phones, personal computers (PCs), laptops, servers, etc.

[0081] In the embodiments of the present application, the first direction is the horizontal direction, and the second direction is the vertical direction; or the first direction is the vertical direction, and the second direction is the horizontal direction. The specific selection can be made according to the actual situation, and the embodiments of the present application do not make specific limitations. In the embodiments of the present application, the first direction is taken as the horizontal direction, and the second direction is taken as the vertical direction, which are taken as examples for description.

[0082] In the embodiments of the present application, a 4x4 calculation unit array is taken as an example for description. The 4x4 calculation unit array is shown in FIG. 4, and there are 4 calculation units in the horizontal direction and the vertical direction. Each calculation unit in the horizontal direction loads different convolution kernel data of one channel and loads window feature data in a sliding window corresponding to one channel. Each calculation unit in the vertical direction processes one convolution kernel data corresponding to different channels in one convolution kernel and one window feature data corresponding to different channels in one sliding window. The calculation units in the vertical direction are cascaded to perform accumulation operation on the output feature data of each calculation unit in the vertical direction. Figure 9

[0083] Reference is made to FIG. 1. Figure 9 ​For example, in the right-to-left order of the 4x4 array of calculation units, the four calculation units in the first row load the convolution kernel data of channel 0 in the convolution kernels K0-K3, the four calculation units in the second row load the convolution kernel data of channel 1 in the convolution kernels K0-K3, the four calculation units in the third row load the convolution kernel data of channel 2 in the convolution kernels K0-K3, and the four calculation units in the fourth row load the convolution kernel data of channel 3 in the convolution kernels K0-K3. Meanwhile, the four calculation units in the first row load the window feature data of channel 0 corresponding to the 0th sliding window to the 4th sliding window in the time sequence; the four calculation units in the second row load the window feature data of channel 1 corresponding to the 0th sliding window to the 4th sliding window in the time sequence, the four calculation units in the third row load the window feature data of channel 2 corresponding to the 0th sliding window to the 4th sliding window in the time sequence, and the four calculation units in the fourth row load the window feature data of channel 3 corresponding to the 0th sliding window to the 4th sliding window in the time sequence. At this time, in the vertical direction, in the right-to-left order, in the first clock cycle, the four calculation units in the first column process the convolution kernel data of the convolution kernel K0 in the channels 0-3 and the window feature data corresponding to the 0th sliding window, respectively; in the second clock cycle, the four calculation units in the first column process the convolution kernel data of the convolution kernel K0 in the channels 0-3 and the window feature data corresponding to the 1st sliding window, respectively, and the four calculation units in the second column process the convolution kernel data of the convolution kernel K1 in the channels 0-3 and the window feature data corresponding to the 0th sliding window, respectively; in the third clock cycle, the four calculation units in the first column process the convolution kernel data of the convolution kernel K0 in the channels 0-3 and the window feature data corresponding to the 2nd sliding window, respectively, the four calculation units in the second column process the convolution kernel data of the convolution kernel K1 in the channels 0-3 and the window feature data corresponding to the 1st sliding window, respectively, and the four calculation units in the third column process the convolution kernel data of the convolution kernel K2 in the channels 0-3 and the window feature data corresponding to the 0th sliding window, respectively; in the fourth clock cycle, the four calculation units in the first column process the convolution kernel data of the convolution kernel K0 in the channels 0-3 and the window feature data corresponding to the 3rd sliding window, respectively, the four calculation units in the second column process the convolution kernel data of the convolution kernel K1 in the channels 0-3 and the window feature data corresponding to the 2nd sliding window, respectively, the four calculation units in the third column process the convolution kernel data of the convolution kernel K2 in the channels 0-3 and the window feature data corresponding to the 1st sliding window, respectively, and the four calculation units in the fourth column process the convolution kernel data of the convolution kernel K3 in the channels 0-3 and the window feature data corresponding to the 0th sliding window, respectively.

[0084] In the embodiment of the present application, one computing unit in the computing unit array is used to process one convolution kernel data corresponding to one channel and window feature data in one sliding window; one computing unit includes a plurality of multiply-accumulate trees; wherein the number of multiply-accumulate trees is the same as the number of convolution windows in one sliding window, and the length of each multiply-accumulate tree is the same as the number of feature data in one convolution window.

[0085] For example, the unit structure diagram of the computing unit is as shown in Figure 10 If the computing unit includes three multiply-accumulate trees, the computing unit can process window feature data of three convolution windows, the weight data loaded in each multiply-accumulate tree is the same, the length of each multiply-accumulate tree is 9, that is, each multiply-accumulate tree can perform multiplication operation of window feature data and weight data for 9 times and then perform addition once, and each group of multiplication factors in the multiply-accumulate tree is one weight data F in the convolution kernel and one window feature data W, as shown in Figure 10 Figure 11 The convolution operation can be mapped to the computing unit, specifically, frame 2 in the input feature data represents a sliding window region, each sliding window region corresponds to W 3*3 convolution windows, window feature data of W 3*3 convolution windows is read in sequence and is subjected to flattening operation, that is, W data columns with length of 9 are obtained, weight information required by convolution operation is pre-stored in each computing unit, each computing unit shares one convolution kernel W0, then W data columns with length of 9 are loaded into W multiply-accumulate trees in the computing unit, the computing unit performs multiplication and accumulation operation, and finally output feature data shown in frame 2 is generated.

[0086] In an optional embodiment, the data scheduling mode for weight data multiplexing, specifically the way of mapping data stream to the deep learning accelerator is as shown in Figure 12 K0_C0~C3 represents convolution kernel data of channels 0 to 3 in the convolution kernel K0, and for a feature image, Win0_C0~C3 represents window feature data of channels 0 to 3 corresponding to the 0th sliding window in the input feature data.

[0087] ​In the right-to-left order, first, K0-C0~C3 are loaded into the first column, K1-C0~C3 are loaded into the second column, K2-C0~C3 are loaded into the third column, and K3-C0~C3 are loaded into the fourth column; then, Win0-C0~C3 are input into the first column, the second column, the third column and the fourth column for convolution calculation, and Win1-C0~C3 are input into the first column, the second column, the third column and the fourth column for convolution calculation; until WinN-C0~C3 are input into the first column, the second column, the third column and the fourth column for convolution calculation, the first round of weight reuse is completed. At this time, K0_C4~C7, K1_C4~C7, K2_C4~C7 and K3_C4~C7 are loaded, and data in Win0-C4~C7 to WinN-C4~C7 are traversed for convolution calculation, and the same is repeated until all the weight reuse processes are completed.

[0088] In another optional embodiment, for the data scheduling mode of input feature data multiplexing, the specific way of mapping the data stream to the deep learning accelerator is as follows: Figure 13 In the right-to-left order, first, Win0-C0~C3 are loaded into the first column, the second column, the third column and the fourth column, respectively; then, K0-C0~C3 are input into the first column for convolution operation, K1-C0~C3 are input into the second column for convolution operation, K2-C0~C3 are input into the third column for convolution operation, and K3-C0~C3 are input into the fourth column for convolution operation; then, the input feature data remains unchanged, K4-C0~C3 are input into the first column for convolution operation, K5-C0~C3 are input into the second column for convolution operation, K6-C0~C3 are input into the third column for convolution operation, and K7-C0~C3 are input into the fourth column for convolution operation, to complete the first round of image data multiplexing; then, Win0-C4~C7 are loaded into the first column, the second column, the third column and the fourth column, respectively, and data in K0-C4~C7 to K7-C4~C7 are traversed for convolution calculation, and the same is repeated until all the input feature data multiplexing processes are completed.

[0089] Based on the above embodiments, an embodiment of the present application provides a data processing apparatus 1. As shown in Figure 14 The apparatus 1 comprises:

[0090] An acquisition unit 10 is configured to acquire hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model, respectively; the deep learning accelerator is configured to run the neural network model;

[0091] A comparison unit 11 is configured to compare memory access times in a plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters, and obtain a comparison result.

[0092] determining unit 12 is configured to determine a target data scheduling manner of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling manners based on the comparison result.

[0093] Optionally, the obtaining unit 10 is further configured to respectively obtain size information of a feature image output by one computing unit in the deep learning accelerator, a number of convolution kernels loaded at one time by the deep learning accelerator, and a channel number; and determine the size information of the feature image output by the one computing unit and the number of convolution kernels loaded at one time by the deep learning accelerator as the hardware parameters; respectively obtain a total number of convolution kernels corresponding to the first algorithm layer, and size information of feature data output by the first algorithm layer; and determine the total number of convolution kernels corresponding to the first algorithm layer, the channel number, and the size information of the feature data output by the first algorithm layer as the algorithm layer parameters.

[0094] Optionally, the comparison unit 11 is further configured to determine a ratio of weight data access times corresponding to the plurality of data scheduling manners according to the size information of the feature image output by the one computing unit and the size information of the feature data output by the first algorithm layer; and determine a ratio of input feature data access times corresponding to the plurality of data scheduling manners according to the number of convolution kernels loaded at one time by the deep learning accelerator, the total number of convolution kernels corresponding to the first algorithm layer, and width information of the feature image output by the one computing unit; the width information of the feature image output by the one computing unit is width information in the size information of the feature image output by the one computing unit.

[0095] Optionally, the determining unit 12 is further configured to determine, based on the ratio of weight data access times corresponding to the plurality of data scheduling manners and the ratio of input feature data access times corresponding to the plurality of data scheduling manners, a data scheduling manner with least access times from the plurality of data scheduling manners as the target data scheduling manner of the first algorithm layer on the deep learning accelerator.

[0096] Optionally, the apparatus further includes a modeling unit.

[0097] The determining unit 12 is further configured to determine a plurality of permutation and combination manners of hardware resources and a plurality of groups of hardware parameters corresponding to the plurality of permutation and combination manners; obtain initial algorithm layer parameters of the first algorithm layer; determine the hardware parameters from the plurality of groups of hardware parameters based on a preset algorithm performance index and a preset hardware performance index, and adjust the initial algorithm layer parameters to obtain algorithm layer parameters corresponding to the first algorithm layer.

[0098] The modeling unit is configured to model the deep learning accelerator by using the hardware parameters.

[0099] Optionally, the apparatus further comprises a traversal unit and a data processing unit.

[0100] The traversal unit is configured to sequentially traverse the input feature data and the convolution kernel data according to the target data scheduling manner.

[0101] The data processing unit is configured to load the input feature data and the convolution kernel data of each traversal into the deep learning accelerator for data processing.

[0102] The data processing apparatus provided in the embodiment of the present application obtains the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer in the neural network model respectively; the deep learning accelerator is used to run the neural network model; the memory access times under a plurality of data scheduling manners are compared according to the hardware parameters and the algorithm layer parameters, and a comparison result is obtained; and the target data scheduling manner of the first algorithm layer on the deep learning accelerator is determined from the plurality of data scheduling manners based on the comparison result. As can be seen, the data processing apparatus provided in the embodiment of the present application analyzes the data flow under a plurality of data scheduling manners according to the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer, obtains the comparison result of the memory access times between the plurality of data scheduling manners, and determines the target data scheduling manner based on the comparison result. Since the memory access times greatly affect the energy consumption of data transmission, the target data scheduling manner determined based on the comparison result considers the memory access times of the first algorithm layer on the deep learning accelerator, which can improve the performance of the algorithm layer executing on the deep learning accelerator, and further reduce the hardware power consumption.

[0103] Figure 15 The component structure of the data processing apparatus 1 provided in the embodiment of the present application is shown in Figure 2 In actual applications, based on the same disclosure concept of the above embodiment, as shown in Figure 15 The data processing apparatus 1 of the embodiment of the present application comprises a processor 13, a memory 14 and a communication bus 15.

[0104] In the process of the specific embodiment, the above acquisition unit 10, comparison unit 11, determination unit 12, modeling unit, traversal unit and data processing unit can be implemented by a processor 13 located on the data processing device 1. The processor 13 can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, or a microprocessor. It can be understood that, for different devices, the electronic device for implementing the above processor function can also be other devices, and the embodiment is not limited in this regard.

[0105] In the embodiment of the present application, the communication bus 15 is used to realize the connection and communication between the processor 13 and the memory 14. When the processor 13 executes the running program stored in the memory 14, the following data processing method is realized:

[0106] Respectively acquire hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model; the deep learning accelerator is used to run the neural network model; compare memory access times in a plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters, and obtain a comparison result; and determine a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes based on the comparison result.

[0107] Further, the processor 13 is further configured to respectively acquire size information of feature images output by one calculation unit in the deep learning accelerator, a number of convolution kernels loaded at a time by the deep learning accelerator and a channel number, and determine the size information of the feature images output by the one calculation unit and the number of convolution kernels loaded at a time by the deep learning accelerator as the hardware parameters; and respectively acquire a total number of convolution kernels corresponding to the first algorithm layer, and size information of feature data output by the first algorithm layer, and determine the total number of convolution kernels corresponding to the first algorithm layer, the channel number and the size information of the feature data output by the first algorithm layer as the algorithm layer parameters.

[0108] Further, the processor 13 is further configured to determine a ratio of weight data access times corresponding to the plurality of data scheduling manners according to size information of the feature image output by the one calculation unit and size information of the feature data output by the first algorithm layer; and determine a ratio of input feature data access times corresponding to the plurality of data scheduling manners according to a number of convolution kernels loaded at one time by the deep learning accelerator, a total number of convolution kernels corresponding to the first algorithm layer, and width information of the feature image output by the one calculation unit, wherein the width information of the feature image output by the one calculation unit is width information in the size information of the feature image output by the one calculation unit.

[0109] Further, the processor 13 is further configured to determine a data scheduling manner with least access times from the plurality of data scheduling manners as a target data scheduling manner of the first algorithm layer on the deep learning accelerator based on the ratio of weight data access times corresponding to the plurality of data scheduling manners and the ratio of input feature data access times corresponding to the plurality of data scheduling manners.

[0110] Further, the processor 13 is further configured to determine a plurality of permutation and combination manners of hardware resources and a plurality of groups of hardware parameters corresponding to the plurality of permutation and combination manners; obtain initial algorithm layer parameters of the first algorithm layer; determine the hardware parameter from the plurality of groups of hardware parameters based on a preset algorithm performance index and a preset hardware performance index, and adjust the initial algorithm layer parameters to obtain algorithm layer parameters corresponding to the first algorithm layer; and model the deep learning accelerator by using the hardware parameter.

[0111] Further, the processor 13 is further configured to traverse input feature data and convolution kernel data in sequence according to the target data scheduling manner; and load the input feature data and the convolution kernel data traversed each time into the deep learning accelerator for data processing.

[0112] The embodiment of the present application provides a storage medium having a computer program stored thereon, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors, and the one or more programs are applied to a data processing device, and the computer program implements the data processing method.

[0113] It should be noted that, in this text, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0114] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing an image display device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present disclosure.

[0115] The above description is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: respectively acquiring hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model; the deep learning accelerator is used to run the neural network model; the hardware parameters comprise size information of feature images output by one calculation unit, a number of convolution kernels loaded at one time by the deep learning accelerator, and channel numbers; the algorithm layer parameters comprise a total number of convolution kernels corresponding to the first algorithm layer and size information of feature data output by the first algorithm layer; comparing memory access times in multiple data scheduling modes according to the hardware parameters and the algorithm layer parameters to obtain a comparison result; determining a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the multiple data scheduling modes based on the comparison result.

2. The method of claim 1, wherein, The method further comprises: respectively acquiring the size information of the feature images output by the one calculation unit, the number of the convolution kernels loaded at one time by the deep learning accelerator, and the channel numbers; respectively acquiring the total number of the convolution kernels corresponding to the first algorithm layer and the size information of the feature data output by the first algorithm layer.

3. The method of claim 2, wherein, The method further comprises: determining a ratio of weight data memory access times corresponding to the multiple data scheduling modes according to the size information of the feature images output by the one calculation unit and the size information of the feature data output by the first algorithm layer; determining a ratio of input feature data memory access times corresponding to the multiple data scheduling modes according to the number of the convolution kernels loaded at one time by the deep learning accelerator, the total number of the convolution kernels corresponding to the first algorithm layer, and width information of the feature images output by the one calculation unit; the width information of the feature images output by the one calculation unit is width information in the size information of the feature images output by the one calculation unit.

4. The method of claim 3, wherein, The method further comprises: determining a data scheduling mode with the least memory access times in the multiple data scheduling modes as the target data scheduling mode of the first algorithm layer on the deep learning accelerator based on the ratio of the weight data memory access times corresponding to the multiple data scheduling modes and the ratio of the input feature data memory access times corresponding to the multiple data scheduling modes.

5. The method of claim 1, wherein, Before the step of respectively acquiring the hardware parameters of the deep learning accelerator and the algorithm layer parameters corresponding to the first algorithm layer in the neural network model, the method further comprises: determining multiple permutation and combination modes of hardware resources and multiple groups of hardware parameters corresponding to the multiple permutation and combination modes; and acquiring initial algorithm layer parameters of the first algorithm layer; determining the hardware parameters from the multiple groups of hardware parameters based on preset algorithm performance indexes and preset hardware performance indexes, and adjusting the initial algorithm layer parameters to obtain the algorithm layer parameters corresponding to the first algorithm layer; and determining the target data scheduling mode of the first algorithm layer on the deep learning accelerator from the multiple data scheduling modes based on the comparison result. Modeling of the deep learning accelerator is performed using the hardware parameters.

6. The method of claim 1, wherein, After determining the target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes based on the comparison result, the method further comprises: According to the target data scheduling mode, input feature data and convolution kernel data are sequentially traversed, and the input feature data and the convolution kernel data traversed each time are loaded into the deep learning accelerator for data processing.

7. A data processing apparatus, characterized by The data processing device is provided with the deep learning accelerator of any one of claims 1-6; and the deep learning accelerator is an array of calculation units. In each group of calculation units in the array of calculation units along the first direction, a group of convolution kernel data corresponding to one channel and window feature data in one sliding window corresponding to one channel are sequentially loaded. In each group of calculation units along the second direction in the array of calculation units, a group of convolution kernel data corresponding to a group of channels in one convolution kernel and a group of window feature data corresponding to a group of channels in one sliding window are sequentially processed. The calculation units along the second direction in the array of calculation units are cascaded to perform accumulation operation on the output feature data of each calculation unit in a group of calculation units, so as to obtain one output feature data corresponding to a group of calculation units.

8. The apparatus of claim 7, wherein, One calculation unit in the array of calculation units is used to process one convolution kernel data corresponding to one channel and window feature data in one sliding window; the one calculation unit comprises a plurality of multiply-accumulate trees; the number of the multiply-accumulate trees is the same as the number of convolution windows in one sliding window, and the length of each multiply-accumulate tree is the same as the number of feature data in one convolution window.

9. A data processing apparatus, characterized by, The data processing device comprises: An acquisition unit is configured to acquire hardware parameters of a deep learning accelerator and algorithm layer parameters corresponding to a first algorithm layer in a neural network model respectively; the deep learning accelerator is configured to run the neural network model; the hardware parameters comprise size information of feature images output by one calculation unit, a number of convolution kernels loaded at one time and a number of channels of the deep learning accelerator; and the algorithm layer parameters comprise a total number of convolution kernels corresponding to the first algorithm layer and size information of feature data output by the first algorithm layer. A comparison unit is configured to compare access frequencies in a plurality of data scheduling modes according to the hardware parameters and the algorithm layer parameters, and obtain a comparison result. A determination unit is configured to determine a target data scheduling mode of the first algorithm layer on the deep learning accelerator from the plurality of data scheduling modes based on the comparison result.

10. A data processing apparatus, characterized by, The device comprises a processor, a memory and a communication bus; the processor implements the method of any one of claims 1-6 when executing a running program stored in the memory.

11. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images