Data scheduling methods and electronic devices applicable to neural network accelerators

Abstract

This invention relates to the field of neural network accelerator technology, and provides a data scheduling method and electronic device suitable for neural network accelerators. The method includes: obtaining an expanded index of input features based on a padding mask, and determining the mapping relationship between the expanded index and the original index; performing convolution calculations on the input features based on the mapping relationship, the kernel size, and the convolution traversal order to obtain multiple convolution results, wherein the convolution traversal order is determined based on the pooling traversal order; performing activation and pooling calculations on each obtained convolution result to obtain a corresponding pooling result, wherein the input for each pooling calculation includes the activation result corresponding to the current pooling calculation and the pooling result corresponding to the previous pooling calculation; calculating a quantization result based on the pooling result corresponding to the Mth pooling calculation, and storing it in a buffer, where M = a × N, a is a positive integer, and N is the size of the pooling window. This addresses the problem of high time and space overhead in data scheduling for neural network accelerators.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network accelerator, and particularly relates to a data scheduling method suitable for a neural network accelerator and an electronic device. BACKGROUND

[0002] With the popularity of deep learning models in the fields of computer vision and natural language processing, special accelerators for neural networks, such as neural network processing units (NPUs) and tensor processing units (TPUs), have become the key hardware for solving the insufficient computing power and low energy efficiency of general-purpose processors. The special accelerators need to sequentially schedule padding operators, convolution operators, activation operators, pooling operators and quantization operators for related calculations, but the time and space overhead of the related scheduling methods is large.

[0003] For example, in related technologies, when scheduling a padding operator for data padding, a large number of 0 elements are pre-stored in the memory in advance, which will squeeze the valid data space, resulting in waste of memory bandwidth and increase of data interaction delay; or the pixel position is judged in real time through hardware logic when the data is loaded, 0 elements are generated and valid data is spliced, but only fixed padding parameters are adapted, the flexibility is poor, and the additional judgment and splicing circuit occupies chip area and consumes power, reducing the energy efficiency. When scheduling a pooling operator for pooling calculation in related technologies, the convolution operation needs to be completed, and then the convolution results stored in the cache are taken out for pooling calculation, which increases the calculation delay and storage overhead.

[0004] At present, no effective solution has been proposed to solve the problem of large time and space overhead of the data scheduling method for the neural network accelerator in related technologies. SUMMARY

[0005] The data scheduling method suitable for a neural network accelerator and the electronic device provided by the embodiments of the present application at least solve the problem of large time and space overhead of the data scheduling method for the neural network accelerator in related technologies.

[0006] The data scheduling method suitable for the neural network accelerator provided by the embodiment of the application comprises: determining a mapping relationship between an extended index and an original index of input features, wherein the extended index is obtained by using a padding mask by the input features; scheduling a convolution operator based on the mapping relationship, a convolution kernel size and a convolution traversal order, performing convolution calculation on the input features to obtain a plurality of convolution results, wherein the convolution traversal order is determined based on a pooling traversal order, and the pooling traversal order is used to schedule a pooling operator to perform pooling calculation; performing activation calculation and pooling calculation in sequence for each obtained convolution result to obtain a corresponding pooling result, wherein the input of each pooling calculation comprises an activation result of the activation calculation corresponding to the current time of pooling calculation, and a pooling result corresponding to the previous time of pooling calculation of the current time; performing quantization calculation on the pooling result corresponding to the Mth time of pooling calculation to obtain a quantization result, and storing the quantization result in a buffer, wherein M=a×N, a is a positive integer, and N is the size of the pooling window.

[0007] Preferably, the mapping relationship between the extended index and the original index of the input features comprises: determining a boundary of a padding region based on the convolution kernel size and the feature map size of the input features to obtain the padding region; determining the extended index based on the feature map of the input features and the padding region; in the case that the convolution element of the current convolution window is located in the padding region, the extended index corresponding to the convolution element has no mapping relationship with the original index, and a padding element is used to replace the convolution element for subsequent calculation; in the case that the convolution element is located in the region corresponding to the feature map, the mapping relationship between the extended index corresponding to the convolution element and the original index corresponding to the convolution element is determined.

[0008] Preferably, the scheduling of the convolution operator based on the mapping relationship, the convolution kernel size and the convolution traversal order to perform convolution calculation on the input features to obtain a plurality of convolution results comprises: taking the convolution kernel as a convolution window, sliding the convolution window according to the convolution traversal order, and performing index traversal on the input features; reading the input features corresponding to the current convolution window from the memory based on the mapping relationship to calculate the convolution result corresponding to the current convolution window; in the case that all the input features are indexed and traversed by the convolution window, the plurality of convolution results are obtained, and the convolution calculation is stopped.

[0009] Preferably, the input features are indexed according to the convolution window in the convolution traversal order, including: setting a feature width increment enable signal, a feature width count signal, a feature width reset signal, a feature height increment enable signal, a feature height count signal and a feature height reset signal, wherein the initial values of the feature width count signal and the feature height count signal are 0; the feature width increment enable signal is set to an enable value based on the convolution traversal order, and the value of the feature width count signal increases with the sliding of the convolution window in the width direction of the feature map, wherein the feature map is the feature map of all input features; whenever the value of the feature width count signal is equal to the width of the feature map, the feature width reset signal resets the feature width count signal to 0, sets the feature height increment enable signal to an enable value, and the value of the feature height count signal is incremented by 1; when the value of the feature height count signal is equal to the height of the feature map, the feature height reset signal resets the feature height count signal to 0 and stops the indexing traversal.

[0010] Preferably, the input features corresponding to the current convolution window are read from the memory based on the mapping relationship, and the convolution result corresponding to the current convolution window is calculated, including: determining the input features corresponding to the convolution elements in the current convolution window based on the mapping relationship; multiplying the convolution elements and the input features corresponding to the convolution elements to obtain corresponding convolution intermediate values; and adding the convolution intermediate values corresponding to the convolution elements in the current convolution window to obtain the convolution result corresponding to the current convolution window.

[0011] Preferably, in the case that all input features are indexed by the convolution window, before obtaining a plurality of convolution results, the above method further includes: setting a convolution kernel increment enable signal, a convolution kernel element count signal and a convolution kernel reset enable signal; when the current convolution window is indexed, the convolution kernel increment enable signal is set to an enable value, and the convolution kernel element count signal is started, wherein the value of the convolution kernel element count signal increases with the number of convolution elements corresponding to the determined input features in the current convolution window; when the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the convolution kernel reset enable signal resets the convolution kernel element count signal to 0, and determines that the convolution result corresponding to the current convolution window is calculated.

[0012] Preferably, after each convolution result is obtained, the activation calculation and the pooling calculation are sequentially performed to obtain the corresponding pooling result, and the above method further includes: determining the index information of the pooling result of the current pooling calculation in the corresponding pooling window; remapping the storage address of the input feature corresponding to the current pooling calculation based on the index information to obtain verification information; and updating the extended index and the mapping relationship based on the verification information.

[0013] Preferably, the index information of the pooling result of the current pooling calculation in the corresponding pooling window is determined by determining the relative coordinates of the pooling result of the current pooling calculation based on the size of the pooling window, determining the absolute coordinates of the pooling result of the current pooling calculation based on the size of the global feature map, wherein the global feature map is a feature map obtained by performing padding mask on the input feature, and determining the index information based on the relative coordinates and the absolute coordinates.

[0014] Preferably, before the input feature is convoluted to obtain a plurality of convolution results based on the mapping relationship, the convolution kernel size and the convolution traversal order, the method further comprises: determining a pooling traversal order based on the size of the pooling window, the size of the feature map of the input feature and the size of the padding region; and determining the convolution traversal order based on the pooling traversal order and the convolution kernel size.

[0015] The electronic device provided by the embodiment of the present application comprises a processor and a memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any of the above.

[0016] The data scheduling method and the electronic device provided by the embodiment of the present application are suitable for a neural network accelerator, the extended index of the input feature is determined in a padding mask manner, the mapping relationship between the extended index and the original index of the input feature is established, the position of the input feature can be determined in real time through the padding mask, the padding element does not need to be actually stored in the memory, thereby reducing the memory cache pressure. The convolution traversal order is determined based on the pooling traversal order, the activation calculation and the pooling calculation are sequentially performed on each convolution result obtained, and the corresponding pooling result is obtained; the quantization calculation is performed on the pooling result corresponding to the Mth pooling calculation, the quantization result is obtained, and the quantization result is stored in the buffer, wherein M is determined according to the size of the pooling window, the convolution result can be compared in real time, and the address access time is saved. The problem that the data scheduling method for the neural network accelerator has large time and space overheads in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other embodiments according to these drawings without creative labor.

[0018] Figure 1 is a step flow chart of a data scheduling method suitable for a neural network accelerator in the embodiment of the present application.

[0019] Figure 2Fig. 1 is a schematic diagram of a module of an AI accelerator according to an embodiment of the present application.

[0020] Figure 3 Fig. 2 is a schematic diagram of a convolution kernel according to an embodiment of the present application.

[0021] Figure 4 Fig. 3 is a schematic diagram of a pooling window according to an embodiment of the present application.

[0022] Figure 5 Fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present application.

[0023] In the above drawings, the following reference signs are used:

[0024] X represents the width direction of the feature map; and Y represents the height direction of the feature map. DETAILED DESCRIPTION

[0025] Embodiments of the present application will be described in more detail by referring to the accompanying drawings. Although certain embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather these embodiments are provided so as to more completely and thoroughly understand the present application. It is understood that the drawings and embodiments of the present application are for exemplary purposes only and are not intended to limit the scope of the present application.

[0026] In related technologies, when a padding operator is scheduled to perform data padding, a large number of 0 elements are pre-stored in the memory in advance, which will squeeze the valid data space, resulting in waste of memory bandwidth and increase of data interaction delay. Alternatively, the pixel position is determined in real time by hardware logic when the data is loaded, 0 elements are generated and spliced with valid data, but this method is only suitable for fixed padding parameters, has poor flexibility, and the additional judgment and splicing circuit occupies chip area and consumes power, thereby reducing the energy efficiency ratio. When a pooling operator is scheduled to perform pooling calculation in related technologies, the convolution operation needs to be completed, and then the convolution results stored in the cache are taken out to perform pooling calculation, which increases the calculation delay and storage overhead.

[0027] To this end, as shown in Fig. 1, an embodiment of the present application provides a data scheduling method suitable for a neural network accelerator, which includes steps S101-S104. Figure 1

[0028] In step S101, a mapping relationship between an extended index and an original index of input features is determined, wherein the extended index is obtained by using a padding mask on the input features.

[0029] ​In step S102, the convolution operator is scheduled based on the mapping relationship, the size of the convolution kernel, and the convolution traversal order to perform convolution calculation on the input feature to obtain a plurality of convolution results, wherein the convolution traversal order is determined based on the pooling traversal order, and the pooling traversal order is used to schedule the pooling operator to perform pooling calculation.

[0030] In step S103, after each convolution result is obtained, activation calculation and pooling calculation are sequentially performed to obtain a corresponding pooling result, wherein the input of each pooling calculation includes the activation result of the activation calculation corresponding to the current pooling calculation and the pooling result of the previous pooling calculation of the current time.

[0031] In step S104, the pooling result corresponding to the Mth pooling calculation is subjected to quantization calculation to obtain a quantization result, and the quantization result is stored in a buffer, wherein M=a×N, a is a positive integer, and N is the size of the pooling window.

[0032] The above data scheduling method provided by the embodiments of the present application can be applied to an AI accelerator. The neural network accelerator is a subset of the AI accelerator and focuses on the optimization of neural network tasks.

[0033] As shown in the AI accelerator shown in Figure 2 The above data scheduling method based on the scheduler controls the input and output of the padding operator, the convolution operator, the activation operator, the pooling operator, and the quantization operator, including: reading the input feature from the memory storing the original input feature and calling the padding operator to perform 0 element padding; sending the padded input feature to the convolution operator, calling the convolution operator to perform convolution calculation, and receiving the convolution calculation result; sending the convolution calculation result to the activation operator to perform activation, and receiving the activation result; sending the activation result to the pooling operator, calling the pooling operator to perform pooling calculation, and receiving the pooling calculation result; sending the pooling calculation result to the quantization operator, calling the quantization operator to perform quantization, and obtaining a quantization result; and finally storing the quantization result in the memory.

[0034] The memory storing the original input feature and the memory storing the quantization result are usually the same physical memory, but different areas are divided by logical addresses to store the original input feature and the quantization result. The quantization result can also be stored in the buffer as in step S104. In other words, when the above data scheduling method is applied, the quantization result can be stored in the memory or the buffer, which can be selected by those skilled in the art according to the storage capacity and efficiency.

[0035] Compared with the prior art, the data scheduling method provided by the embodiment can reduce the memory cache pressure by designing the padding mask to determine the position of the input feature in real time, and can establish a mapping relationship between the extended index and the original index, and then call the data during the convolution calculation, thereby having the advantages of lightweight and good real-time performance.

[0036] The step of implementing the padding operator in the padding mask mode can be but is not limited to the following: first, defining the up, down, left and right boundaries of the padding region:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] In the formula, represents the left boundary of the padding region, represents the right boundary of the padding region, represents the upper boundary of the padding region, represents the lower boundary of the padding region, represents a feature width count signal, represents an input feature with a width of represents a feature map width, represents an input feature with a height of represents a feature map height, represents the coordinate of the feature map width direction when the convolution window slides, represents the coordinate of the feature map height direction when the convolution window slides.

[0042] Then, taking the convolution kernel size of 3*3 as an example, the indicator is defined to indicate whether the current convolution window index falls in the padding region:

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] In the formula, represents a convolution kernel element count signal. ​​

[0048] Then, the feature index containing padding elements is remapped to the real feature index:

[0049]

[0050]

[0051] wherein, represents the index of the modified input feature in the width direction, represents the index of the modified input feature in the height direction, represents the feature height count signal.

[0052] Specifically, the elements located in the padding area do not need to participate in the convolution calculation, so when actually performing the convolution calculation, the padded input feature needs to be indexed and modified, that is, the index is remapped. At this point, the above-mentioned method provided by the embodiment is different from the related art in that the elements located in the padding area are not actually stored in the memory to occupy space, but the corresponding extended index is directly obtained through the mask method.

[0053] At this time, the address of the feature taken from the memory will become

[0054]

[0055] wherein, represents the width of the input feature.

[0056] Before calling the convolution operator, the feature data required for performing the convolution operation needs to be prepared. According to the size of the convolution kernel, the required feature data is taken from the memory on demand. The size of the convolution kernel can be, but is not limited to, 3x3 and 5x5. In the subsequent embodiment, the size of the convolution kernel is taken as 3x3 for example. There are 9 convolution elements in a 3x3 size convolution kernel. Please refer to the shaded part in FIG. 1B, which represents a convolution kernel. Figure 3 Figure 3

[0057] The size of the convolution window is the same as that of the convolution kernel. The convolution window can slide according to a step size of 1, but is not limited to this. Each convolution result obtained in step S102 corresponds to a specific position of the convolution window.

[0058] The size of the pooling window can be, but is not limited to, 2x2 and 3x3. A 2x2 size pooling window needs to compare 4 convolution results and select the maximum value or the average value among the 4 convolution results. In the embodiment, the size of the pooling window is taken as 2x2 for example. Please refer to the shaded part in FIG. 2B, which represents a pooling window. Figure 4 Figure 4 ​​​​​​​​

[0059] In the related art, all the results obtained by the convolution calculation are cached first, and then four convolution results are taken out each time according to the indexes of the pooling window to perform the pooling calculation, and all the pooling results are re-cached in the memory, which occupies a large amount of cache space and time.

[0060] The data scheduling method provided in the embodiment changes the sliding direction of the convolution window, so that the convolution window slides according to the sliding order of the pooling window, and the convolution results are compared in real time to obtain the corresponding pooling results.

[0061] In addition, for the first convolution result obtained in step S102, the pooling result obtained through step S103 is the activation result obtained after the convolution result is subjected to the activation calculation.

[0062] For the convolution result obtained after the first convolution result, the pooling result obtained through step S103 is the pooling result obtained by performing the pooling calculation on the activation result obtained after the convolution result corresponding to the current pooling calculation is subjected to the activation calculation and the pooling result obtained by the previous pooling calculation.

[0063] Specifically, the signal index of the pooling window in the feature map width direction is set as , the signal index of the pooling window in the feature map height direction is , the pooling counter is , the input of the pooling operator is and , the output of the pooling operator is represented as , and the feature address index containing the padding element will become:

[0064] ;

[0065] ;

[0066] Taking the pooling window size of 2x2 as an example, the scheduling algorithm of the pooling operator is as follows.

[0067] After obtaining the first convolution result, the first pooling is performed

[0068] =0

[0069] =0, =0

[0070] = +1

[0071] = first convolution result

[0072] Return to feature loading, continue with convolution

[0073] # After getting the second convolution result, second pooling

[0074] = 1

[0075] = 1, = 0

[0076] = + 1

[0077] = second convolution result

[0078] =

[0079] Return to feature loading, continue with convolution

[0080] # After getting the third convolution result, third pooling

[0081] = 2

[0082] = 0, = 1

[0083] = + 1

[0084] = third convolution result

[0085] =

[0086] Return to feature loading, continue with convolution

[0087] # After getting the fourth convolution result, fourth pooling

[0088] = 3

[0089] = 1, = 1

[0090] = + 1

[0091] = fourth convolution result

[0092] Will The input data is sent to a quantization layer, quantization is performed, and the quantized result is stored in a cache

[0093] The first, second, third and fourth convolution results are obtained by sequentially sliding a convolution window and performing convolution calculation by calling a convolution operator.

[0094] In summary, the above-mentioned data scheduling method provided by the embodiments of the present application determines the extended index of the input feature by filling the mask, and then establishes the mapping relationship between the extended index and the original index of the input feature.

[0095] The convolution traversal order is determined based on the pooling traversal order, and after obtaining each convolution result, activation calculation and pooling calculation are sequentially performed to obtain the corresponding pooling result.

[0096] The method can solve the problem of large time and space overhead of the data scheduling method for the neural network accelerator in the related art.

[0097] Preferably, in step S101, the mapping relationship between the extended index and the original index of the input feature is determined, including: determining the boundary of the padding region based on the size of the convolution kernel and the size of the feature map of the input feature to obtain the padding region.

[0098] In the case that the convolution element of the current convolution window is located in the padding region, the extended index corresponding to the convolution element has no mapping relationship with the original index, and the padding element is used to replace the convolution element for subsequent calculation.

[0099] In the case that the convolution element is located in the corresponding region of the feature map, the mapping relationship between the extended index corresponding to the convolution element and the original index corresponding to the convolution element is determined.

[0100] This helps to improve the data scheduling efficiency while ensuring the accuracy of data processing.

[0101] Preferably, in step S102, the convolution operator is scheduled based on the mapping relationship, the size of the convolution kernel, and the convolution traversal order to perform convolution calculation on the input feature to obtain a plurality of convolution results, and the method further comprises: determining a pooling traversal order based on the size of the pooling window, the size of the feature map of the input feature, and the size of the padding region; and determining the convolution traversal order based on the pooling traversal order and the size of the convolution kernel.

[0102] The pipeline processing mode of convolution-activation-pooling is facilitated to reduce memory access overhead, and the previous convolution result can enter the activation and pooling stages as soon as it is generated, without waiting for the entire convolution layer to be completed before starting the pooling layer. The above pipeline processing mode enables the convolution operator, the activation operator, and the pooling operator in the accelerator to work in parallel, improves the utilization rate of corresponding hardware resources, and reduces the overall calculation delay.

[0103] In addition, the determination of the pooling traversal order is related to the size of the padding region, and the convolution traversal order is derived based on this, which can naturally maintain the adaptation ability of the convolution traversal order to the padding region.

[0104] Preferably, in step S102, the convolution operator is scheduled based on the mapping relationship, the size of the convolution kernel, and the convolution traversal order to perform convolution calculation on the input feature to obtain a plurality of convolution results, and the method further comprises: taking the convolution kernel as a convolution window, and sliding the convolution window according to the convolution traversal order to index and traverse the input feature. The input feature corresponding to the current convolution window is read from the memory based on the mapping relationship, and the convolution result corresponding to the current convolution window is calculated. When all the input features are indexed and traversed by the convolution window, the plurality of convolution results are obtained, and the convolution calculation is stopped.

[0105] Sliding traversal with the convolution kernel as the window can perform full-coverage scanning on the input feature map, and sliding strictly according to the convolution traversal order can ensure that each valid region of the input feature is covered and calculated by the convolution kernel, avoiding incomplete feature extraction caused by omission.

[0106] Meanwhile, in combination with the mapping relationship determined in step S101, the data read can accurately distinguish between original feature data and padding data. The original feature data needs to access the memory through the mapping relationship, while the padding data can directly use the preset value, ensuring that the calculation of each convolution window meets the mathematical definition and guaranteeing the accuracy of the convolution result from the execution level.

[0107] The core feature of convolution calculation is that there is a large amount of overlapping data between adjacent windows. Sliding the convolution window according to the convolution traversal order can make the overlapping data of adjacent windows be reused in a short time, significantly improving the temporal locality of data. This way of using data reduces repeated access to the memory, and the same data can be shared by multiple windows after being loaded once, reducing the demand of the accelerator on the memory bandwidth, and being particularly suitable for the processing scene of high-resolution feature maps in deep learning.

[0108] The termination condition of stopping the convolution calculation can be achieved by comparing the preset total window number with the current window count after all input features are indexed and traversed by the convolution window, avoiding complex dynamic judgment logic and reducing the size and power consumption of hardware circuit. The preset total window number is calculated based on the size of the feature map to be convolved, the size of the convolution kernel used, and the sliding step.

[0109] Further, the convolution kernel is taken as the convolution window, and the input features are indexed and traversed according to the convolution traversal order, including: setting a feature width increment enable signal , a feature width count signal , a feature width reset signal , a feature height increment enable signal , a feature height count signal , and a feature height reset signal , wherein the initial values of the feature width count signal and the feature height count signal are 0.

[0110] The feature width increment enable signal is set to an enabled value based on the convolution traversal order, and the value of the feature width count signal increases with the sliding of the convolution window in the width direction X of the feature map, wherein the feature map is the feature map of all input features.

[0111] Whenever the value of the feature width count signal is equal to the width of the feature map, the feature width reset signal sets the feature width count signal to 0 and sets the feature height increment enable signal to an enabled value, and the value of the feature height count signal is incremented by 1.

[0112] When the value of the feature height count signal is equal to the height of the feature map, the feature height reset signal sets the feature height count signal to 0, and the indexing and traversal are stopped.

[0113] The sliding logic of the convolution window is converted into simple digital circuit signal interaction by using the signal linkage mechanism of enable-count-reset: the width count is incremented depending on the width enable signal, the width is full to trigger the reset and height enable, and the height count is full to terminate the traversal. This modular signal design of width control and height control separation and linkage can be realized by basic counters, comparators and logic gates without complex state machines or software instructions. Compared with the scheme of dynamically calculating the traversal path, the hardware circuit size is smaller and the delay is lower.

[0114] Further, based on the mapping relationship, the input features corresponding to the current convolution window are read from the memory, and the convolution result corresponding to the current convolution window is calculated, including: determining the input features corresponding to the convolution elements in the current convolution window based on the mapping relationship. The convolution elements are multiplied by the input features corresponding to the convolution elements to obtain the corresponding convolution intermediate value. The convolution intermediate values corresponding to each convolution element in the current convolution window are added to obtain the convolution result corresponding to the current convolution window.

[0115] The essence of convolution calculation is weighted summation. Through the mapping relationship, the original input features or padding values corresponding to each convolution element in the current convolution window, i.e. the convolution kernel weight, are accurately located, so that each weight is matched with the correct input element, and the weighted summation is completed in sequence, from the execution level to ensure that the convolution result completely meets the definition of the mathematical formula, and avoid feature extraction distortion caused by data matching error or calculation sequence deviation.

[0116] Further, in the case that all the input features are indexed and traversed by the convolution window, before obtaining a plurality of convolution results, the above method further comprises: setting a convolution kernel increment enable signal , a convolution kernel element count signal and a convolution kernel reset enable signal .

[0117] When the current convolution window is indexed, the convolution kernel increment enable signal is set to an enabled value, and the convolution kernel element count signal is started, wherein the value of the convolution kernel element count signal increases with the number of convolution elements in the current convolution window that have determined corresponding input features.

[0118] When the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the convolution kernel reset enable signal performs 0 reset on the convolution kernel element count signal , and determines that the convolution result corresponding to the current convolution window is calculated.

[0119] The calculation of the convolution window needs to traverse all elements of the convolution kernel, for example, a 3x3 size convolution kernel needs to process 9 elements, and each element needs to complete a multiplication operation with the corresponding input feature, and then accumulate. Convolution kernel element count signal By recording the number of processed convolution elements in real time, it can be accurately judged whether the current window has completed the calculation of all elements. When the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the current convolution window calculation is confirmed to be completed by the convolution kernel reset enable signal . Ensure that each element of the convolution kernel is effectively involved in the calculation to avoid calculation omission or repeated calculation resulting in deviation of the convolution result.

[0120] Preferably, step S103, after obtaining each convolution result, sequentially performing activation calculation and pooling calculation to obtain the corresponding pooling result, the method further comprises: determining the index information of the pooling result of the current pooling calculation in the corresponding pooling window. Remap the storage address of the input feature corresponding to the current pooling calculation based on the index information to obtain verification information. Update the extended index and the mapping relationship based on the verification information.

[0121] The input of the pooling calculation is the activation result obtained by activating the convolution result, so the pooling result is indirectly related to the original input feature, and in this embodiment, the pooling window and the convolution window have a corresponding relationship, and thus correspond to a specific input feature region. Therefore, by determining the index information of the pooling result in the pooling window, the corresponding original input feature region of the pooling result can be traced back. The verification information obtained by remapping the storage address of the input feature based on the above index information is an actual check of the mapping relationship between the extended index and the original index. If the remapping result is consistent with the original mapping relationship, it means that the mapping is correct; if it is not consistent, the mapping relationship can be corrected through the verification information. This dynamic correction mechanism of the mapping relationship can timely discover and correct the possible deviation in the initial mapping, thereby improving the accuracy of data access from the feedback level.

[0122] Further, determining the index information of the pooling result of the current pooling calculation in the corresponding pooling window comprises: determining the relative coordinates of the pooling result of the current pooling calculation based on the size of the pooling window. Determine the absolute coordinates of the pooling result of the current pooling calculation based on the size of the global feature map, wherein the global feature map is a feature map obtained by filling a mask on the input feature. Determine the index information based on the relative coordinates and the absolute coordinates.

[0123] Taking the global feature map size as 8x8 and the pooling window size as 2x2 as an example, the relative coordinates are, for example, (0, 1), i.e. =0, =1, represents the 0th row and 1st column in the pooling window, which is used to describe the local position of the pooling result in the pooling window, which is the core information of the pooling calculation itself. The absolute coordinate such as (5, 3) represents the absolute position of the corresponding pooling result in the global feature map containing the padding region, which can avoid the local coordinate offset caused by the existence of the padding region. The index information formed by the combination of the two contains both the relative relationship in the local window and the absolute positioning of the global space, and completely describes the spatial attributes of the pooling result, providing an unambiguous position reference for subsequent address remapping and mapping relationship updating.

[0124] The embodiment of the present application also provides the overall flow and part of the algorithm of the above-mentioned data scheduling method, as follows.

[0125] #Load input features in combination with padding operator and pooling operator

[0126] #Set padding boundary

[0127]

[0128]

[0129]

[0130]

[0131] #Set indicator

[0132]

[0133]

[0134]

[0135]

[0136] #Load input features according to convolution window

[0137] #Index convolution kernel window

[0138] ( ==8)?1:0

[0139] =1

[0140] = ? ? +

[0141] if ( ==2)or( ==5):

[0142] = +1

[0143] =0

[0144] else if ( ==8):

[0145] =0

[0146] =0

[0147] else:

[0148] = +1

[0149] #Index feature window

[0150] = ? 1:0

[0151] =( == )?1:0

[0152] = ? 0: ? +2:

[0153] = ? 1:0

[0154] =( == )?1:0

[0155] = ? 0: ? +2:

[0156] #Actual feature position remapping

[0157]

[0158]

[0159] #Memory fetch address update

[0160]

[0161] # execute the convolution operator

[0162] # execute the activation operator

[0163] # execute the pooling operator

[0164] # execute the quantization operator

[0165] # store the quantized features into the cache

[0166] In the above algorithm example, represents a feature width reset signal , an enable signal, represents a feature height reset signal , an enable signal, represents the height of the input feature.

[0167] In the above algorithm example, when the actual feature position remapping is performed, the index information corresponding to the pooling window is introduced on the basis of the mapping relationship between the extended index obtained based on the padding mask and the original index of the input feature.

[0168] In summary, in the accelerator for the neural network, the strategy for scheduling each operator often has the problems of insufficient flexibility and large time and space overhead, and the above scheme provided by the embodiments of the present application greatly reduces the time and space overhead when calling the padding operator and the pooling operator by splitting the execution of each operator, setting the corresponding feature coordinate signal, padding boundary determination signal, and convolution direction control signal, so that the convolutional neural network algorithm can run quickly and orderly.

[0169] The embodiments of the present application also provide a non-transitory machine readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to make the computer execute the method of the embodiments of the present application.

[0170] The embodiments of the present application also provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is used to make the computer execute the method of the embodiments of the present application.

[0171] The embodiments of the present application also provide an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, and the computer program, when executed by the at least one processor, is used to make the electronic device execute the method of the embodiments of the present application.

[0172] Reference Figure 5This is a structural block diagram of an electronic device, either a server or a client, according to an embodiment of the present invention. It is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0173] like Figure 5 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device. The computing unit 501, the ROM 502, and the RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0174] Multiple components in the electronic device are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information into the electronic device. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0175] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, CPUs, graphics processing units (GPUs), various specialized artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, or the like. The computing unit 501 performs various methods and processes described above. For example, in some embodiments, the method embodiments of the present creation can be implemented as a computer program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods described above by way of other any suitable means, such as by way of firmware example.

[0176] Computer programs used to implement embodiments of the present creation methods can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0177] In the context of embodiments of the present creation, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared signals, or any suitable combination thereof. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0178] It should be noted that the term "comprising" and its derivations, as used in the embodiments of the present invention, are intended to be open-ended. The term "comprising" and its derivations are used in the sense of "including but not limited to". The term "based on" is intended to mean "based, at least in part, on" unless explicitly stated otherwise. The term "one embodiment" is intended to mean "at least one embodiment" and the term "another embodiment" is intended to mean "at least one additional embodiment". The term "some embodiments" is intended to mean "at least some embodiments". The terms "a" and "an", as used in the context of the embodiments of the present invention, are to be construed to cover both the singular and the plural, unless otherwise indicated. The term "another" is intended to mean "at least one additional". The term "first", "second", and the like, as used in the context of the embodiments of the present invention, are not intended to denote a relative importance, unless otherwise indicated.

[0179] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or authorized by all parties.

[0180] The steps described in the method embodiments provided by the embodiments of the present invention can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of protection of the present invention is not limited in this respect.

[0181] The word "embodiment" in the specification means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present invention. The presence of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or choice from other embodiments. Each embodiment in the specification is described in a relevant manner, and the same or similar parts between embodiments are cross-referenced. In particular, for device, equipment, system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0182] The above-described embodiments only express several embodiments of the present invention, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of protection. It should be noted that for those skilled in the art, without departing from the concept of the present invention, a number of modifications and improvements can be made, which are within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be subject to the appended claims.

Claims

1. A data scheduling method suitable for a neural network accelerator, characterized in that, The method comprises: determining a mapping relationship between an extended index of an input feature and an original index, wherein the extended index is obtained by padding the input feature with a mask; based on the mapping relationship, the size of the convolution kernel and the convolution traversal order, scheduling a convolution operator to perform convolution calculation on the input feature to obtain a plurality of convolution results, wherein the convolution traversal order is determined based on a pooling traversal order, the pooling traversal order is used to schedule a pooling operator to perform pooling calculation, and the convolution kernel is used as a convolution window, and the convolution window is slid according to the convolution traversal order to index traverse the input feature; each time a convolution result is obtained, activation calculation and pooling calculation are sequentially performed to obtain a corresponding pooling result, wherein the input of each pooling calculation includes the activation result of the current time activation calculation corresponding to the current time pooling calculation, and the pooling result of the previous time pooling calculation corresponding to the current time; quantization calculation is performed on the pooling result corresponding to the Mth pooling calculation to obtain a quantization result, and the quantization result is stored in a buffer, wherein M=a×N, a is a positive integer, and N is the size of the pooling window; determining a mapping relationship between an extended index of an input feature and an original index, comprising: determining the boundary of the padding region based on the size of the convolution kernel and the size of the feature map of the input feature to obtain the padding region; determining the extended index based on the feature map of the input feature and the padding region; in the case that the convolution element of the current convolution window is located in the padding region, the extended index corresponding to the convolution element has no mapping relationship with the original index, and a padding element is used to replace the convolution element for subsequent calculation; in the case that the convolution element is located in the region corresponding to the feature map, the mapping relationship between the extended index corresponding to the convolution element and the original index corresponding to the convolution element is determined; before the method further comprises: based on the size of the pooling window, the size of the feature map of the input feature and the size of the padding region, determining the pooling traversal order; based on the pooling traversal order and the size of the convolution kernel, determining the convolution traversal order; based on the mapping relationship, the size of the convolution kernel and the convolution traversal order, scheduling a convolution operator to perform convolution calculation on the input feature to obtain a plurality of convolution results, wherein the convolution traversal order is determined based on a pooling traversal order, the pooling traversal order is used to schedule a pooling operator to perform pooling calculation, and the convolution kernel is used as a convolution window, and the convolution window is slid according to the convolution traversal order to index traverse the input feature; setting a feature width increment enable signal, a feature width count signal, a feature width reset signal, a feature height increment enable signal, a feature height count signal and a feature height reset signal, wherein the initial values of the feature width count signal and the feature height count signal are 0; based on the convolution traversal order, the feature width increment enable signal is set to an enable value, and the value of the feature width count signal is incremented with the sliding of the convolution window in the feature map width direction, wherein the feature map is the feature map of all the input features; ​ The feature width reset signal resets the feature width count signal to 0 and sets the feature height increment enable signal to an enable value when the value of the feature width count signal is equal to the width of the feature map, and the value of the feature height count signal is incremented by 1; The feature height reset signal resets the feature height count signal to 0 and stops the index traversal when the value of the feature height count signal is equal to the height of the feature map.

2. The method of claim 1, wherein, Based on the mapping relationship, the convolution kernel size and the convolution traversal order are used to schedule a convolution operator to perform convolution calculation on the input features to obtain a plurality of convolution results, including: The input features corresponding to the current convolution window are read from the memory based on the mapping relationship, and the convolution result corresponding to the current convolution window is calculated. In the case that all the input features are indexed and traversed by the convolution window, the plurality of convolution results are obtained, and the convolution calculation is stopped. The input features corresponding to the current convolution window are read from the memory based on the mapping relationship, and the convolution result corresponding to the current convolution window is calculated, including:

3. The method of claim 2, wherein, Based on the mapping relationship, the input features corresponding to the convolution elements in the current convolution window are determined; The convolution elements are multiplied by the input features corresponding to the convolution elements to obtain corresponding convolution intermediate values; The convolution intermediate values corresponding to the convolution elements in the current convolution window are added to obtain the convolution result corresponding to the current convolution window. Before the plurality of convolution results are obtained in the case that all the input features are indexed and traversed by the convolution window, the method further includes:

4. The method of claim 3, wherein, The convolution kernel increment enable signal, the convolution kernel element count signal and the convolution kernel reset enable signal are set; When the current convolution window is indexed, the convolution kernel increment enable signal is set to an enable value, and the convolution kernel element count signal is started, wherein the value of the convolution kernel element count signal increases with the number of convolution elements corresponding to the input features determined in the current convolution window; When the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the convolution kernel reset enable signal resets the convolution kernel element count signal to 0, and determines that the convolution result corresponding to the current convolution window is calculated. After each of the convolution results is obtained, the method further includes:

5. The method of claim 1, wherein, Determine the index information of the pooling result of the current pooling calculation in the corresponding pooling window; Based on the index information, the storage address of the input feature corresponding to the current pooling calculation is remapped to obtain verification information; Based on the verification information, the extended index and the mapping relationship are updated. Determine the index information of the pooling result of the current pooling calculation in the corresponding pooling window, including:

6. The method of claim 5, wherein, Based on the size of the pooling window, the relative coordinates of the pooling result of the current pooling calculation are determined; ​ determine absolute coordinates of a pooling result of the current time pooling calculation based on a size of a global feature map, wherein the global feature map is a feature map obtained by performing padding mask on the input feature; determine the index information based on the relative coordinates and the absolute coordinates.

7. An electronic device comprising: A processor and a memory storing a program, characterized in that the program includes instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images