Data scheduling method suitable for neural network accelerator and electronic equipment

Abstract

The invention relates to the technical field of neural network accelerators, and provides a data scheduling method suitable for a neural network accelerator and electronic equipment, and the method comprises the steps: obtaining an extended index of an input feature based on a filling mask, and determining a mapping relation between the extended index and an original index; performing convolution calculation on the input features based on the mapping relation, the convolution kernel size and a convolution traversal sequence to obtain a plurality of convolution results, and determining the convolution traversal sequence based on the pooling traversal sequence; when each convolution result is obtained, activation and pooling calculation are carried out to obtain a corresponding pooling result, and the input of each pooling calculation comprises an activation result corresponding to the current pooling calculation and a pooling result corresponding to the previous pooling calculation; and calculating a quantization result based on the pooling result corresponding to the Mth pooling calculation, and storing the quantization result in a buffer, M = a * N, a being a positive integer, and N being the size of the pooling window. The problem that the time-space overhead of data scheduling for a neural network accelerator is large is solved.

Description

Technical Field

[0001] This invention relates to the field of neural network accelerator technology, and in particular to a data scheduling method and electronic device suitable for neural network accelerators. Background Technology

[0002] With the widespread adoption of deep learning models in fields such as computer vision and natural language processing, dedicated accelerators for neural networks, such as Neural Processing Units (NPUs) and Tensor Processing Units (TPUs), have become key hardware solutions for addressing the insufficient computing power and low energy efficiency of general-purpose processors. These dedicated accelerators require sequential scheduling of padding operators, convolution operators, activation operators, pooling operators, and quantization operators for related computations, but the scheduling methods incur significant time and space overhead.

[0003] For example, when related technologies schedule padding operators for data padding, they either pre-store a large number of zero elements in memory, which occupies effective data space, leading to wasted memory bandwidth and increased data interaction latency; or they determine the pixel position in real time through hardware logic during data loading, generate zero elements, and concatenate effective data, but this only adapts to fixed padding parameters, has poor flexibility, and the additional judgment and concatenation circuits occupy chip area, consume power, and reduce energy efficiency. Furthermore, when related technologies schedule pooling operators for pooling calculations, they need to wait for all convolution operations to complete before retrieving the convolution results stored in the buffer for pooling calculations, increasing computational latency and storage overhead.

[0004] There is currently no effective solution to the problem of large time and space overhead in data scheduling methods for neural network accelerators in related technologies. Summary of the Invention

[0005] The present invention provides a data scheduling method and electronic device suitable for neural network accelerators, which at least solves the problem of large time and space overhead in data scheduling methods for neural network accelerators in related technologies.

[0006] This invention provides a data scheduling method suitable for neural network accelerators, comprising: determining a mapping relationship between the expanded index and the original index of the input features, wherein the expanded index is obtained by padding the input features; scheduling convolution operators based on the mapping relationship, the kernel size, and the convolution traversal order to perform convolution calculations on the input features, obtaining multiple convolution results, wherein the convolution traversal order is determined based on the pooling traversal order, which is used to schedule pooling operators for pooling calculations; performing activation calculations and pooling calculations sequentially for each convolution result, obtaining a corresponding pooling result, wherein the input for each pooling calculation includes the activation result of the activation calculation corresponding to the current pooling calculation and the pooling result corresponding to the previous pooling calculation; performing quantization calculations on the pooling result corresponding to the Mth pooling calculation, obtaining a quantization result, and storing it in a buffer, wherein M = a × N, a is a positive integer, and N is the size of the pooling window.

[0007] Preferably, determining the mapping relationship between the expanded index and the original index of the input feature includes: determining the boundary of the filling region based on the convolution kernel size and the feature map size of the input feature to obtain the filling region; determining the expanded index based on the feature map of the input feature and the filling region; when the convolution element of the current convolution window is located in the filling region, the expanded index corresponding to the convolution element has no mapping relationship with the original index, and the filling element is used to replace the convolution element for subsequent calculations; when the convolution element is located in the corresponding region of the feature map, determining the mapping relationship between the expanded index corresponding to the convolution element and the original index corresponding to the convolution element.

[0008] Preferably, based on the mapping relationship, the convolutional kernel size and convolutional traversal order are used to schedule the convolutional operators to perform convolutional calculations on the input features, resulting in multiple convolutional results. This includes: using the convolutional kernel as the convolutional window, sliding the convolutional window according to the convolutional traversal order, and indexing and traversing the input features; reading the input features corresponding to the current convolutional window from the memory based on the mapping relationship, and calculating the convolutional result corresponding to the current convolutional window; and stopping the convolutional calculation when all input features have been indexed and traversed by the convolutional window, resulting in multiple convolutional results.

[0009] Preferably, the input features are indexed and traversed by sliding the convolution window with the convolution kernel as the convolution window according to the convolution traversal order. This includes: setting a feature width increment enable signal, a feature width count signal, a feature width reset signal, a feature height increment enable signal, a feature height count signal, and a feature height reset signal, wherein the initial values ​​of the feature width count signal and the feature height count signal are 0; the feature width increment enable signal is set to an enabled value based on the convolution traversal order, and the value of the feature width count signal increases as the convolution window slides in the feature map width direction, wherein the feature map is the feature map of all input features; whenever the value of the feature width count signal is equal to the width of the feature map, the feature width reset signal resets the feature width count signal to 0, sets the feature height increment enable signal to an enabled value, and increments the value of the feature height count signal by 1; when the value of the feature height count signal is equal to the height of the feature map, the feature height reset signal resets the feature height count signal to 0 and stops the index traversal.

[0010] Preferably, the process of reading the input features corresponding to the current convolution window from the memory based on the mapping relationship and calculating the convolution result corresponding to the current convolution window includes: determining the input features corresponding to the convolution elements in the current convolution window based on the mapping relationship; multiplying the convolution elements with the input features corresponding to the convolution elements to obtain the corresponding intermediate convolution values; and summing the intermediate convolution values ​​corresponding to each convolution element in the current convolution window to obtain the convolution result corresponding to the current convolution window.

[0011] Preferably, before obtaining multiple convolution results after all input features have been indexed and traversed by the convolution window, the above method further includes: setting a convolution kernel increment enable signal, a convolution kernel element count signal, and a convolution kernel reset enable signal; when the current convolution window is indexed, the convolution kernel increment enable signal is set to an enabled value, and the convolution kernel element count signal is started, wherein the value of the convolution kernel element count signal increases as the number of convolution elements corresponding to the determined input features in the current convolution window increases; when the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the convolution kernel reset enable signal resets the convolution kernel element count signal to 0, indicating that the convolution result corresponding to the current convolution window has been calculated.

[0012] Preferably, after obtaining each convolution result, activation calculation and pooling calculation are performed sequentially. After obtaining the corresponding pooling result, the above method further includes: determining the index information of the pooling result of the current pooling calculation in the corresponding pooling window; remapping the storage address of the input feature corresponding to the current pooling calculation based on the index information to obtain verification information; and updating the extended index and mapping relationship based on the verification information.

[0013] Preferably, determining the index information of the pooling result of the current pooling calculation within the corresponding pooling window includes: determining the relative coordinates of the pooling result of the current pooling calculation based on the size of the pooling window; determining the absolute coordinates of the pooling result of the current pooling calculation based on the size of the global feature map, wherein the global feature map is a feature map obtained by padding the input features; and determining the index information based on the relative and absolute coordinates.

[0014] Preferably, before scheduling convolution operators based on mapping relationships, kernel size, and convolution traversal order to perform convolution calculations on input features and obtain multiple convolution results, the above method further includes: determining the pooling traversal order based on the pooling window size, feature map size of the input features, and padding region size; and determining the convolution traversal order based on the pooling traversal order and kernel size.

[0015] An electronic device provided by an embodiment of the present invention includes: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform any of the methods described above.

[0016] This invention provides a data scheduling method and electronic device suitable for neural network accelerators. It uses a padding mask to determine the extended index of input features and then establishes a mapping relationship between the extended index and the original index of the input features. This allows for real-time determination of the input feature's position through the padding mask, eliminating the need to actually store the padding elements in memory and reducing memory cache pressure. The convolution traversal order is determined based on the pooling traversal order. For each convolution result, activation and pooling calculations are performed sequentially to obtain the corresponding pooling result. The pooling result corresponding to the Mth pooling calculation is quantized and stored in a cache. M is determined based on the pooling window size, enabling real-time pooling comparison of convolution results and saving address fetching time. This addresses the problem of high time and space overhead in related data scheduling methods for neural network accelerators. Attached Figure Description

[0017] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of a data scheduling method applicable to neural network accelerators in an embodiment of the present invention.

[0019] Figure 2This is a schematic diagram of an AI accelerator module in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the convolution kernel in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of a pooled window in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0023] The above figures include the following reference numerals: X—width direction of the feature map; Y—height direction of the feature map. Detailed Implementation

[0024] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] When related technologies schedule padding operators for data padding, they either pre-store a large number of zero elements in memory, which occupies effective data space, leading to wasted memory bandwidth and increased data interaction latency; or they determine the pixel position in real time through hardware logic during data loading, generate zero elements, and concatenate effective data. However, this only adapts to fixed padding parameters, has poor flexibility, and the additional judgment and concatenation circuits occupy chip area, consume power, and reduce energy efficiency. When related technologies schedule pooling operators for pooling calculations, they need to wait for all convolution operations to be completed before retrieving the convolution results stored in the buffer for pooling calculations, increasing computational latency and storage overhead.

[0026] Therefore, please refer to Figure 1 As shown, the present invention provides a data scheduling method suitable for neural network accelerators, including steps S101 to S104.

[0027] Step S101: Determine the mapping relationship between the extended index and the original index of the input feature, wherein the extended index is obtained by using a padding mask for the input feature.

[0028] Step S102: Based on the mapping relationship, the convolution kernel size and the convolution traversal order, the convolution operators are scheduled to perform convolution calculations on the input features to obtain multiple convolution results. The convolution traversal order is determined based on the pooling traversal order, which is used to schedule the pooling operators to perform pooling calculations.

[0029] In step S103, for each convolution result obtained, activation calculation and pooling calculation are performed sequentially to obtain the corresponding pooling result. The input of each pooling calculation includes the activation result of the activation calculation corresponding to the current pooling calculation and the pooling result corresponding to the previous pooling calculation.

[0030] Step S104: Quantize the pooling result corresponding to the Mth pooling calculation, obtain the quantization result, and store it in the cache, where M=a×N, a is a positive integer, and N is the size of the pooling window.

[0031] The data scheduling method provided in the embodiments of this invention is applicable to AI accelerators. Among them, neural network accelerators are a subset of AI accelerators, focusing on the optimization of neural network tasks.

[0032] by Figure 2 Taking the AI ​​accelerator shown as an example, the above data scheduling method is based on the scheduler controlling the input and output of the padding operator, convolution operator, activation operator, pooling operator, and quantization operator. It includes: reading input features from the memory storing the original input features and calling the padding operator to pad with zero elements; feeding the padded input features into the convolution operator, calling the convolution operator to perform convolution calculation, and receiving the convolution calculation result; feeding the convolution calculation result into the activation operator for activation, and receiving the activation result; feeding the activation result into the pooling operator, calling the pooling operator to perform pooling calculation, and receiving the pooling calculation result; feeding the pooling calculation result into the quantization operator, calling the quantization operator to perform quantization, and obtaining the quantization result; and finally storing the quantization result in the memory.

[0033] The memory storing the raw input features and the memory storing the quantization results are often the same physical memory, but they are divided into different regions using logical addresses to store the raw input features and the quantization results. Alternatively, as in step S104, the quantization results can be stored in a buffer. In other words, when applying the above data scheduling method, the quantization results can be stored in either a memory or a buffer, and those skilled in the art can choose based on storage capacity and efficiency.

[0034] Compared to related technologies that store a large number of zero elements in memory in advance, the data scheduling method provided in this embodiment reduces memory cache pressure by designing a padding mask to determine the location of input features in real time; it establishes a mapping relationship between the extended index and the original index, and then calls the data during convolution calculation, which has the advantages of being lightweight and having good real-time performance.

[0035] The steps for implementing the padding operator using a padding mask can be, but are not limited to: First, define the top, bottom, left, and right boundaries of the padding region: ; ; ; ; In the formula, Indicates the left boundary of the filled region. Indicates the right boundary of the filled region. Indicates the upper boundary of the filled region. Indicates the lower boundary of the filled region. This represents the characteristic width counting signal. Indicates width is Input features, Indicates the width of the feature map. Indicates height as Input features, Indicates the feature map height. This represents the coordinates along the width direction of the feature map as the convolution window slides. This represents the coordinates in the feature map height direction as the convolution window slides.

[0036] Then, taking a convolution kernel size of 3×3 as an example, we define an indicator. Used to indicate whether the current convolution window index falls within the padding region: ; ; ; ; In the formula, This represents the element count signal of the convolution kernel.

[0037] Next, the feature indices containing the padding elements are remapped to the actual feature indices: ; ; In the formula, This represents the index of the corrected input feature in the width direction. This represents the index of the corrected input feature in the height direction. This represents the characteristic height counting signal.

[0038] Specifically, elements located in the padded region do not need to participate in the convolution calculation. Therefore, when actually performing the convolution calculation, the padded input features need to undergo index correction, i.e., index remapping. In this respect, the method provided in this embodiment differs from related technologies in that elements located in the padded region are not actually stored in memory to occupy space, but rather their corresponding extended indices are directly obtained through a mask.

[0039] At this point, the address for fetching the feature from memory will become : ; In the formula, Indicates the width of the input feature.

[0040] Calling a convolution operator requires preparing the feature data needed for the convolution operation. This scheme retrieves the required feature data from memory on demand, according to the size of the convolution kernel. The convolution kernel size can be, but is not limited to, 3×3 or 5×5. This embodiment will use a 3×3 convolution kernel as an example for explanation. A 3×3 convolution kernel has 9 convolution elements. Please refer to... Figure 3 As shown, Figure 3 The shaded area in the diagram represents a convolution kernel.

[0041] The size of the convolution window is the same as the size of the convolution kernel. The convolution window can slide with a stride of 1, but is not limited to. Each convolution result obtained in step S102 corresponds to a convolution window at a specific location.

[0042] The pooling window size can be, but is not limited to, 2×2 or 3×3. A 2×2 pooling window needs to compare four convolution results and select the maximum or average value among them. This example uses a 2×2 pooling window size as an example; please refer to [link / reference]. Figure 4 As shown, Figure 4 The shaded area in the image represents a pooled window.

[0043] In related technologies, all the results obtained from convolution calculations need to be cached first, and then four convolution results are retrieved each time according to the index of the pooling window for pooling calculation. Then all the pooling results are cached back into memory, which will consume a lot of cache space and time.

[0044] The data scheduling method provided in this embodiment changes the sliding direction of the convolution window so that the convolution window slides in the same order as the pooling window, and compares the convolution results in real time to obtain the corresponding pooling results.

[0045] Furthermore, for the first convolution result obtained in step S102, the pooling result obtained in step S103 is the activation result obtained after activation calculation of the convolution result.

[0046] For the convolution result obtained after the first convolution result mentioned above, the pooling result obtained after step S103 is: the activation result obtained after activation calculation of the convolution result corresponding to the current pooling calculation, and the pooling result obtained from the previous pooling calculation, and the pooling result obtained from the previous pooling calculation.

[0047] Specifically, the signal index of the pooling window in the feature map width direction is set to... The signal index of the pooling window in the feature map height direction is The pooling counter is The input to the pooling operator is and The output of the pooling operator is represented as The feature address index containing the padding element will become: ; ; Taking a pooling window size of 2×2 as an example, the scheduling algorithm for the pooling operator is as follows.

[0048] #After obtaining the first convolution result, perform the first pooling.

[0049] =0

[0050] =0, =0

[0051] = +1

[0052] =First convolution result

[0053] Return to feature loading and continue convolution.

[0054] #After obtaining the second convolution result, perform a second pooling operation.

[0055] =1

[0056] =1, =0

[0057] = +1

[0058] =Second convolution result

[0059] =

[0060] Return to feature loading and continue convolution.

[0061] #After obtaining the result of the third convolution, perform the third pooling.

[0062] =2

[0063] =0, =1

[0064] = +1

[0065] =Third convolution result

[0066] =

[0067] Return to feature loading and continue convolution.

[0068] #After obtaining the result of the fourth convolution, perform the fourth pooling.

[0069] =3

[0070] =1, =1

[0071] = +1

[0072] =Fourth convolution result

[0073] Will The data is fed into the quantization layer for quantization, and the quantized result is stored in the cache.

[0074] The first, second, third, and fourth convolution results are obtained by sequentially sliding the convolution window using the convolution operator. The sliding order of the convolution window is determined by the size of the pooling window and its current position. In other words, the convolution traversal order is based on the pooling traversal order.

[0075] In summary, the data scheduling method provided by the embodiments of the present invention uses a padding mask to determine the extended index of the input feature, and then establishes a mapping relationship between the extended index and the original index of the input feature. It can determine the position of the input feature in real time through the padding mask, without actually storing the padding elements in the memory, thereby reducing the memory cache pressure.

[0076] The convolution traversal order is determined based on the pooling traversal order. For each convolution result obtained, activation calculation and pooling calculation are performed sequentially to obtain the corresponding pooling result. The pooling result corresponding to the Mth pooling calculation is quantized to obtain the quantized result and stored in the cache. M is determined according to the size of the pooling window, which enables real-time pooling comparison of convolution results and saves address fetching time.

[0077] It can solve the problem of large time and space overhead in data scheduling methods for neural network accelerators in related technologies.

[0078] Preferably, step S101, determining the mapping relationship between the expanded index and the original index of the input features, includes: determining the boundary of the filling region based on the convolution kernel size and the feature map size of the input features, thereby obtaining the filling region. The expanded index is then determined based on the feature map of the input features and the filling region.

[0079] When the convolution element in the current convolution window is located in the padding region, the extended index corresponding to the convolution element has no mapping relationship with the original index, and the padding element is used to replace the convolution element for subsequent calculations.

[0080] When a convolutional element is located in the corresponding region of the feature map, determine the mapping relationship between the extended index corresponding to the convolutional element and the original index corresponding to the convolutional element.

[0081] This helps to improve data scheduling efficiency while ensuring the accuracy of data processing.

[0082] Preferably, before step S102, which schedules convolution operators based on the mapping relationship, kernel size, and convolution traversal order to perform convolution calculations on the input features and obtain multiple convolution results, the method further includes: determining the pooling traversal order based on the pooling window size, the feature map size of the input features, and the size of the filled region. The convolution traversal order is determined based on the pooling traversal order and the kernel size.

[0083] This helps reduce memory access overhead and supports a pipelined processing mode of convolution-activation-pooling. The activation and pooling stages can begin immediately after the previous convolution result is generated, without waiting for the entire convolutional layer to complete before starting the pooling layer. This pipelined processing mode allows the convolution, activation, and pooling operators in the accelerator to work in parallel, improving the utilization of corresponding hardware resources and reducing overall computational latency.

[0084] In addition, the determination of the pooling traversal order is related to the size of the filling region. Based on this, the convolution traversal order can be derived, which naturally maintains the ability of the convolution traversal order to adapt to the filling region.

[0085] Preferably, step S102 involves scheduling convolution operators based on the mapping relationship, kernel size, and convolution traversal order to perform convolution calculations on the input features, obtaining multiple convolution results. This includes: using the convolution kernel as a convolution window, sliding the convolution window according to the convolution traversal order, and indexing and traversing the input features. Based on the mapping relationship, the input features corresponding to the current convolution window are read from memory, and the convolution result corresponding to the current convolution window is calculated. Once all input features have been indexed and traversed by the convolution window, multiple convolution results are obtained, and the convolution calculation stops.

[0086] By using the convolution kernel as a window for sliding traversal, a full-coverage scan of the input feature map can be performed. By strictly following the convolution traversal order, it can be ensured that every effective region of the input feature is covered and calculated by the convolution kernel, avoiding omissions that would lead to incomplete feature extraction.

[0087] At the same time, by reading data in conjunction with the mapping relationship determined in step S101, the original feature data and the padding data can be accurately distinguished. The original feature data needs to be accessed through the mapping relationship, while the padding data can directly use preset values, ensuring that the calculation of each convolution window conforms to the mathematical definition, and guaranteeing the accuracy of the convolution result from the execution level.

[0088] The core characteristic of convolution computation is the presence of a large amount of overlapping data between adjacent windows. Sliding the convolution window in the order of convolution traversal allows overlapping data between adjacent windows to be reused repeatedly in a short period of time, significantly improving the temporal locality of data. This data utilization method reduces repeated access to memory, and the same data can be shared by multiple windows after being loaded once, reducing the accelerator's memory bandwidth requirements, which is particularly suitable for processing high-resolution feature maps in deep learning scenarios.

[0089] All input features are traversed by the convolution window index. The termination condition for stopping the convolution calculation can be achieved by comparing the preset total number of windows with the current window count, avoiding complex dynamic judgment logic and reducing the size and power consumption of the hardware circuit. The preset total number of windows is calculated based on the size of the feature map to be convolved, the size of the convolution kernel used, and the sliding stride.

[0090] Furthermore, using the convolution kernel as the convolution window, the convolution window is slid according to the convolution traversal order to perform index traversal of the input features, including: setting an enable signal for feature width increment. Feature width counting signal Characteristic width reset signal Feature height increasing enable signal Feature height counting signal and characteristic height reset signal Among them, the characteristic width counting signal and characteristic height counting signal The initial value is 0.

[0091] Enabling the feature width based on the convolution traversal order. Set to enable value, characteristic width counting signal The value of increases as the convolution window slides along the width direction X of the feature map, where the feature map is the feature map of all input features.

[0092] Whenever the feature width counting signal When the value equals the width of the feature map, the feature width reset signal... Feature width counting signal Reset to 0 and increment the feature height enable signal. Set to enable value, characteristic height counting signal The value is increased by 1.

[0093] When the feature height counting signal When the value equals the height of the feature map, the feature height reset signal... Count the feature height signal Set to 0 to reset and stop index traversal.

[0094] A signal linkage mechanism of enable-counter-reset is adopted to transform the sliding logic of the convolution window into simple digital circuit signal interaction: the width count depends on the incrementing width enable signal; when the width is full, a reset and height enable are triggered; when the height count is full, traversal terminates. This modular signal design, which separates and links width and height control, can be implemented using basic counters, comparators, and logic gates, without the need for complex state machines or software instructions. Compared to schemes that dynamically calculate traversal paths, the hardware circuitry is smaller and has lower latency.

[0095] Further, based on the mapping relationship, the input features corresponding to the current convolution window are read from memory, and the convolution result corresponding to the current convolution window is calculated, including: determining the input features corresponding to the convolution elements in the current convolution window based on the mapping relationship; multiplying the convolution element with the corresponding input features to obtain the corresponding intermediate convolution value; and summing the intermediate convolution values ​​corresponding to each convolution element in the current convolution window to obtain the convolution result corresponding to the current convolution window.

[0096] The essence of convolution calculation is weighted summation. By accurately locating the original input features or padding values ​​corresponding to each convolution element, i.e., the weight of the convolution kernel, through mapping relationships, it ensures that each weight matches the correct input element and completes the weighted summation in sequence. From the execution level, it ensures that the convolution result fully conforms to the mathematical formula definition and avoids feature extraction distortion caused by data matching errors or calculation order deviations.

[0097] Furthermore, before obtaining multiple convolution results after all input features have been indexed and traversed by the convolution window, the above method also includes: setting a convolution kernel increment enable signal. Convolution kernel element count signal and convolution kernel reset enable signal .

[0098] When the current convolution window is indexed, the convolution kernel increment enable signal is activated. Set to enable value to start the convolution kernel element counting signal. Among them, the convolution kernel element count signal The value increases as the number of convolutional elements corresponding to the input features in the current convolutional window increases.

[0099] When the convolution kernel element count signal When the value is equal to the size of the convolution kernel, the convolution kernel reset enable signal is activated. Counting signal of convolution kernel elements Perform a reset to 0 to confirm that the convolution result corresponding to the current convolution window has been calculated.

[0100] The computation of a convolution window requires traversing all elements of the convolution kernel. For example, a 3x3 convolution kernel needs to process 9 elements, and each element must be multiplied with its corresponding input feature before being accumulated. (Convolution kernel element count signal) By recording the number of processed convolutional elements in real time, it is possible to accurately determine whether the current window has completed the computation of all elements. This is achieved when the convolutional kernel element count signal... When the value equals the kernel size, the kernel reset enable signal is used. Confirm that the current convolution window has been fully computed. Ensure that each element of the convolution kernel is effectively included in the computation to avoid omissions or duplicate calculations that could lead to deviations in the convolution results.

[0101] Preferably, in step S103, after obtaining each convolution result, activation calculation and pooling calculation are performed sequentially. After obtaining the corresponding pooling result, the method further includes: determining the index information of the pooling result of the current pooling calculation in the corresponding pooling window; remapping the storage address of the input feature corresponding to the current pooling calculation based on the index information to obtain verification information; and updating the extended index and mapping relationship based on the verification information.

[0102] The input to pooling computation is the activation result obtained by activating the convolution result. Therefore, there is an indirect correlation between the pooling result and the original input features. In this embodiment, there is a correspondence between the pooling window and the convolution window, which in turn corresponds to a specific input feature region. Therefore, by determining the index information of the pooling result in the pooling window, the original input feature region corresponding to the pooling result can be traced back. The verification information obtained by remapping the input feature storage address based on the above index information is an actual verification of the mapping relationship between the expanded index and the original index. If the remapping result is consistent with the original mapping relationship, it means that the mapping is correct; if it is inconsistent, the mapping relationship can be corrected through the verification information. This dynamic correction mechanism of the mapping relationship can promptly detect and correct any deviations that may exist in the initial mapping, improving the accuracy of data access from the feedback level.

[0103] Further, the index information of the pooling result of the current pooling calculation within the corresponding pooling window is determined, including: determining the relative coordinates of the pooling result of the current pooling calculation based on the size of the pooling window; determining the absolute coordinates of the pooling result of the current pooling calculation based on the size of the global feature map, where the global feature map is the feature map obtained by padding the input features with a mask; and determining the index information based on the relative and absolute coordinates.

[0104] Taking a global feature map size of 8×8 and a pooling window size of 2x2 as an example, with relative coordinates such as (0,1), that is... =0, =1 indicates the position of the pooling result in row 0 and column 1 within the pooling window, describing its local location within the window—a core piece of information for the pooling calculation itself. Absolute coordinates, such as (5,3), represent the absolute position of the corresponding pooling result in the global feature map, including the filled region, preventing local coordinate shifts caused by the presence of the filled region. The combined index information encompasses both relative relationships within the local window and absolute positioning in global space, fully characterizing the spatial attributes of the pooling result and providing an unambiguous location reference for subsequent address remapping and mapping relationship updates.

[0105] The present invention also provides the overall process and some algorithms of the above data scheduling method, as shown below.

[0106] # Load input features by combining padding and pooling operators

[0107] #Set fill boundaries

[0108]

[0109]

[0110]

[0111]

[0112] #Setting Indicator

[0113]

[0114]

[0115]

[0116]

[0117] #Load input features by convolution window

[0118] #Indexed Convolution Kernel Window

[0119] =( ==8)? 1:0

[0120] =1

[0121] = ? ? +

[0122] if ( ==2)or( ==5):

[0123] = +1

[0124] =0

[0125] else if ( ==8):

[0126] =0

[0127] =0

[0128] else:

[0129] = +1

[0130] #Index Feature Window

[0131] = 1:0

[0132] =( == )?1:0

[0133] = ? 0: ? +2:

[0134] = 1:0

[0135] =( == )?1:0

[0136] = ? 0: ? +2:

[0137] #Remapping of actual feature locations

[0138]

[0139]

[0140] #Fetch and update feature address in memory

[0141]

[0142] #Execute the convolution operator

[0143] #Execute activation operator

[0144] #Execute pooling operator

[0145] #Execute quantization operator

[0146] # Store the quantized features in the cache

[0147] In the above algorithm example, Indicates characteristic width reset signal The enable signal, Indicates characteristic height reset signal The enable signal, This indicates the height of the input feature.

[0148] In the above algorithm example, when performing actual feature location remapping, the index information corresponding to the pooling window is introduced based on the mapping relationship between the extended index obtained from the padding mask and the original index of the input feature.

[0149] In summary, in accelerators designed for neural networks, the scheduling strategies for each operator are often inflexible and have high time and space overhead. However, the solution provided by the embodiments of this invention, by splitting the execution of each operator and establishing corresponding feature coordinate signals, boundary filling judgment signals, and convolution direction control signals, greatly reduces the time and space overhead when calling the filling operator and pooling operator, enabling the convolutional neural network algorithm to run quickly and orderly.

[0150] The present invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0151] The present invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of the embodiments of the present invention.

[0152] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0153] refer to Figure 5 This is a structural block diagram of an electronic device, either a server or a client, according to an embodiment of the present invention. It is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0154] like Figure 5 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device. The computing unit 501, the ROM 502, and the RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0155] Multiple components in the electronic device are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information into the electronic device. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0156] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as computer programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0157] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of embodiments of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more". The descriptions of terms such as "first", "second", etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of indicated technical features.

[0160] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties.

[0161] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0162] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0163] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A data scheduling method suitable for neural network accelerators, characterized in that, include: Determine the mapping relationship between the extended index and the original index of the input feature, wherein the extended index is obtained by padding the input feature using a mask; Based on the mapping relationship, the convolution kernel size and convolution traversal order schedule the convolution operators to perform convolution calculations on the input features, resulting in multiple convolution results. The convolution traversal order is determined based on the pooling traversal order, which is used to schedule the pooling operators for pooling calculations. For each convolution result obtained, activation calculation and pooling calculation are performed sequentially to obtain the corresponding pooling result. The input of each pooling calculation includes the activation result of the activation calculation corresponding to the current pooling calculation and the pooling result corresponding to the previous pooling calculation. The pooling result corresponding to the Mth pooling calculation is quantized to obtain the quantized result, which is then stored in the cache. Here, M = a × N, where a is a positive integer and N is the size of the pooling window.

2. The method according to claim 1, characterized in that, Determine the mapping relationship between the expanded index and the original index of the input features, including: Based on the convolution kernel size and the feature map size of the input features, the boundary of the filling region is determined, and the filling region is obtained; The extended index is determined based on the feature map of the input features and the filled region; When the convolution element in the current convolution window is located in the padding region, the extended index corresponding to the convolution element has no mapping relationship with the original index, and the padding element is used to replace the convolution element for subsequent calculations. When the convolutional element is located in the region corresponding to the feature map, determine the mapping relationship between the extended index corresponding to the convolutional element and the original index corresponding to the convolutional element.

3. The method according to claim 1, characterized in that, Based on the mapping relationship, the convolutional operators are scheduled according to the kernel size and convolutional traversal order to perform convolution calculations on the input features, resulting in multiple convolutional results, including: Using the convolution kernel as a convolution window, the convolution window is slid according to the convolution traversal order to perform index traversal of the input features; Based on the mapping relationship, the input features corresponding to the current convolution window are read from the memory, and the convolution result corresponding to the current convolution window is calculated. Once all the input features have been indexed and traversed by the convolution window, the multiple convolution results are obtained, and the convolution calculation is stopped.

4. The method according to claim 3, characterized in that, Using the convolution kernel as a convolution window, the convolution window is slid according to the convolution traversal order to perform index traversal on the input features, including: Set a feature width increment enable signal, a feature width count signal, a feature width reset signal, a feature height increment enable signal, a feature height count signal, and a feature height reset signal, wherein the initial values ​​of the feature width count signal and the feature height count signal are 0; Based on the convolution traversal order, the feature width increment enable signal is set to an enable value, and the value of the feature width count signal increases as the convolution window slides in the feature map width direction, wherein the feature map is the feature map of all the input features; Whenever the value of the feature width count signal is equal to the width of the feature map, the feature width reset signal resets the feature width count signal to 0, sets the feature height increment enable signal to the enable value, and increments the value of the feature height count signal by 1; When the value of the feature height count signal is equal to the height of the feature map, the feature height reset signal resets the feature height count signal to 0 and stops the index traversal.

5. The method according to claim 3, characterized in that, Based on the mapping relationship, the input features corresponding to the current convolution window are read from the memory, and the convolution result corresponding to the current convolution window is calculated, including: Based on the mapping relationship, the input features corresponding to the convolutional elements in the current convolutional window are determined; Multiply the convolutional element by the corresponding input feature to obtain the corresponding convolutional intermediate value; The convolution result corresponding to the current convolution window is obtained by summing the convolution intermediate values ​​corresponding to each convolution element in the current convolution window.

6. The method according to claim 5, characterized in that, Before obtaining the multiple convolution results after all the input features have been traversed by the convolution window index, the method further includes: Set the kernel increment enable signal, kernel element count signal, and kernel reset enable signal; When the current convolution window is indexed, the convolution kernel increment enable signal is set to an enable value, and the convolution kernel element counting signal is activated. The value of the convolution kernel element counting signal increases as the number of convolution elements corresponding to the input features in the current convolution window increases. When the value of the convolution kernel element count signal is equal to the size of the convolution kernel, the convolution kernel reset enable signal resets the convolution kernel element count signal to 0, thus confirming that the convolution result corresponding to the current convolution window has been calculated.

7. The method according to claim 1, characterized in that, For each convolution result obtained, activation calculation and pooling calculation are performed sequentially. After obtaining the corresponding pooling result, the method further includes: Determine the index information of the pooling result of the current pooling calculation within the corresponding pooling window; Based on the index information, the storage address of the input feature corresponding to the current pooling calculation is remapped to obtain verification information; The extended index and the mapping relationship are updated based on the verification information.

8. The method according to claim 7, characterized in that, Determine the index information of the pooling result of the current pooling calculation within the corresponding pooling window, including: Based on the size of the pooling window, the relative coordinates of the pooling result of the current pooling calculation are determined; Based on the size of the global feature map, the absolute coordinates of the pooling result of the current pooling calculation are determined, wherein the global feature map is a feature map obtained by padding the input features with a mask; The index information is determined based on the relative coordinates and the absolute coordinates.

9. The method according to claim 1, characterized in that, Based on the mapping relationship, the convolution operators are scheduled according to the kernel size and convolution traversal order to perform convolution calculations on the input features. Before obtaining multiple convolution results, the method further includes: The pooling traversal order is determined based on the size of the pooling window, the size of the feature map of the input feature, and the size of the filled region. The convolution traversal order is determined based on the pooling traversal order and the convolution kernel size.

10. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 9.

Images