Data processing apparatus, method and related product for executing a neural network model

Abstract

The present disclosure discloses a data processing apparatus, method and related products for executing a neural network model. The data processing apparatus can be included in a combined processing apparatus as a computing apparatus, which can further include an interface apparatus and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The combined processing apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively for storing data of the computing apparatus and the other processing apparatuses. The scheme of the present disclosure optimizes the convolution operation on multi-dimensional arrays, improving the operation processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a data processing apparatus, data processing method, chip, and board for executing neural network models. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Convolutional layers are commonly used hidden layers in neural network models, extracting features from input data through convolution operations. Neural network models contain a large number of convolutional operations, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. In convolutional operations, there are both instruction alignment requirements and hardware (e.g., parallel processor) alignment requirements for each dimension of the convolutional layer's filters. Therefore, it is necessary to optimize convolutional operations to improve the computational performance of the neural network model. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes data processing schemes for executing neural network models in several aspects, which can effectively improve the computational performance of convolution operations by transforming the filters of convolutional layers. The neural network models of the embodiments of this disclosure can be applied to various fields, such as image processing, speech processing, text processing, etc., and these processes may include, but are not limited to, recognition and classification.

[0005] In a first aspect, this disclosure provides a data processing apparatus for executing a neural network model, comprising: a storage circuit configured to store folded filters of convolutional layers of the neural network model, the folded filters being generated by folding an original filter in a first dimension and a second dimension, wherein the first dimension folding includes synthesizing multiple expanded filters obtained after multiple convolution stride shifts, and the second dimension folding includes rearranging data in the width and / or height dimensions to the input channel dimension; and a processing circuit configured to: perform the second dimension folding on an input feature map to obtain a folded feature map; perform a convolution operation on the folded feature map using the folded filters to obtain an intermediate result; and perform dimension defolding on the intermediate result to obtain an output feature map.

[0006] In a second aspect, this disclosure provides a chip that includes the data processing apparatus of any of the embodiments of the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip of any of the embodiments of the second aspect above.

[0008] In a fourth aspect, this disclosure provides a method for executing a neural network model implemented by a data processing apparatus, the data processing apparatus including a storage circuit and a processing circuit, wherein the storage circuit stores folded filters of the convolutional layers of the neural network model, the folded filters being generated by folding the original filters in a first dimension and a second dimension, wherein the first dimension folding includes synthesizing multiple expanded filters obtained after multiple convolution stride shifts, and the second dimension folding includes rearranging data in the width dimension and / or height dimension to the input channel dimension; the method includes: the processing circuit performing the second dimension folding on an input feature map to obtain a folded feature map; the processing circuit performing a convolution operation on the folded feature map using the folded filters stored in the storage circuit to obtain an intermediate result; and the processing circuit performing dimension defolding on the intermediate result to obtain an output feature map.

[0009] The disclosed solution optimizes convolution operations by folding filters, utilizing the data processing device, chip, board, and data processing method implemented by the data processing device described above. The embodiments disclosed are particularly suitable for cases where both the output and input channel dimensions of the original filter are small. In conventional convolution operations, when the output channel dimension of the filter is small, significant resource waste occurs due to the limitation on the alignment of the number of parallel computing units. Conversely, when the input channel dimension of the filter is small, excessive redundant computation occurs due to the limitation on the vectorization alignment of the AI ​​chip instruction set. On one hand, the embodiments disclosed here, by folding the original filter in the first dimension, can combine multiple expanded filters obtained after multiple convolution stride shifts into a single folded filter to expand the output channel dimension, thereby fully utilizing available parallel computing units. On the other hand, the embodiments disclosed here, by folding the filter in the second dimension, can fold the data of the convolution kernel width and / or height dimensions onto the input channel dimension to meet instruction alignment requirements, minimizing redundant computation. These two aspects can also be used in combination, thereby most effectively avoiding waste of computing resources and improving the computational performance of convolution operations under hardware acceleration. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This is a structural diagram of a board card according to an embodiment of the present disclosure;

[0012] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram illustrating the internal structure of a single-core computing device according to an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram illustrating the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the internal structure of a processor core according to an embodiment of the present disclosure;

[0016] Figure 6 An exemplary convolution operation example that can be applied to embodiments of this disclosure is shown;

[0017] Figure 7 An exemplary schematic diagram of the first-dimensional folding of this disclosure embodiment is shown;

[0018] Figure 8 A more detailed schematic diagram of the first-dimensional folding according to an embodiment of this disclosure is shown;

[0019] Figure 9 An exemplary comparison diagram of the calculation process before and after the first dimension is folded according to an embodiment of this disclosure is shown;

[0020] Figure 10 An exemplary schematic diagram of dimension transposition according to an embodiment of this disclosure is shown;

[0021] Figure 11 An exemplary schematic diagram of the second-dimensional folding of this disclosure embodiment is shown;

[0022] Figure 12 A more detailed schematic diagram of the second-dimensional folding according to an embodiment of this disclosure is shown;

[0023] Figure 13 This illustration schematically shows a diagram of folding padding of the convolution kernel according to an embodiment of this disclosure;

[0024] Figure 14 A schematic diagram illustrating the effect of convolution stride on effective multiple according to embodiments of this disclosure is provided.

[0025] Figure 15 An exemplary schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure is shown; and

[0026] Figure 16An exemplary flowchart of a data processing method according to an embodiment of this disclosure is shown. Detailed Implementation

[0027] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It should be understood that the terms "first," "second," and "third," etc., that may be used in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0029] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0030] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0031] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0033] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0034] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0035] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0036] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0037] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0038] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 201. Depending on the implementation, the processing device 203 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 201 of this disclosure can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0039] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0040] Figure 3 The diagram shows the internal structure of the single-core computing device 201. The single-core computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The single-core computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0041] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0042] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0043] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between single-core computing device 301 and DRAM 204.

[0044] Figure 4 A schematic diagram of the internal structure of the computing device 201 as a multi-core is shown. The multi-core computing device 41 adopts a hierarchical structure design. As a system-on-a-chip, the multi-core computing device 41 includes at least one cluster, and each cluster includes multiple processor cores. In other words, the multi-core computing device 41 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0045] From the perspective of system-on-a-chip hierarchy, such as Figure 4 As shown, the multi-core computing device 41 includes an external storage controller 401, a peripheral communication module 402, an on-chip interconnect module 403, a synchronization module 404, and multiple clusters 405.

[0046] There can be multiple external storage controllers 401; two are shown as an example in the figure. These controllers are used to access external storage devices, such as those issued by the processor core, in response to access requests from the processor core. Figure 2The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 402 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 403 connects the external storage controller 401, the peripheral communication module 402, and multiple clusters 405 to transmit data and control signals between modules. The synchronization module 404 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 405 are the computing cores of the multi-core computing device 41. Four are shown exemplary in the figure; however, with hardware development, the multi-core computing device 41 of this disclosure can also include 8, 16, 64, or even more clusters 405. The clusters 405 are used to efficiently execute deep learning algorithms.

[0047] From the perspective of cluster hierarchy, such as Figure 4 As shown, each cluster 405 includes multiple processor cores (IPU cores) 406 and one memory core (MEM core) 407.

[0048] Four processor cores 406 are shown in the figure as an example, but this disclosure does not limit the number of processor cores 406. Its internal architecture is as follows: Figure 5 As shown. Each processor core 406 is similar to Figure 3 The single-core computing device 301 also includes three main modules: a control module 51, an arithmetic module 52, and a storage module 53. The functions and structures of the control module 51, arithmetic module 52, and storage module 53 are largely the same as those of the control module 31, arithmetic module 32, and storage module 33, and will not be described again. It should be noted that the storage module 53 includes an input / output direct memory access (IODMA) module 533 and a move direct memory access (MVDMA) module 534. The IODMA 533 controls the memory access of NRAM 531 / WRAM 532 and DRAM 204 via the broadcast bus 409; the MVDMA 534 controls the memory access of NRAM 531 / WRAM 532 and SRAM 408.

[0049] Back Figure 4The storage core 407 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 406, and performing communication between cluster 405 and DRAM 204, communication between clusters 405, and communication between processor cores 406. In other embodiments, the storage core 407 has scalar operation capabilities and is used to perform scalar operations.

[0050] The storage core 407 includes an SRAM 408, a broadcast bus 409, a cluster direct memory access (CDMA) module 410, and a global direct memory access (GDMA) module 411. The SRAM 408 acts as a high-performance data relay station. Data multiplexed between different processor cores 406 within the same cluster 405 does not need to be obtained from the DRAM 204 by each processor core 406. Instead, it is relayed between processor cores 406 via the SRAM 408. The storage core 407 only needs to quickly distribute the multiplexed data from the SRAM 408 to multiple processor cores 406 to improve inter-core communication efficiency and greatly reduce on-chip and off-chip input / output access.

[0051] Broadcast bus 409, CDMA 410, and GDMA 411 are used to perform communication between processor cores 406, communication between clusters 405, and data transfer between cluster 405 and DRAM 204, respectively. These will be explained below.

[0052] The broadcast bus 409 is used to complete high-speed communication between the processor cores 406 within the cluster 405. In this embodiment, the broadcast bus 409 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 408 to several specific processor cores 406. Broadcast is a communication method that transmits a piece of data from SRAM 408 to all processor cores 406, and is a special case of multicast.

[0053] CDMA 410 is used to control SRAM 408 access between different clusters 405 within the same computing device 201.

[0054] GDMA 411 works in conjunction with external memory controller 401 to control memory access from SRAM 408 to DRAM 204 in cluster 405, or to read data from DRAM 204 into SRAM 408. As previously described, communication between DRAM 204 and NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between DRAM 204 and NRAM 431 or WRAM 432 via IODAM 433; the second channel involves first transmitting data between DRAM 204 and SRAM 408 via GDMA 411, and then transmitting data between SRAM 408 and NRAM 431 or WRAM 432 via MVDMA 534. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments of this disclosure can select the data transmission channel according to their own hardware conditions.

[0055] In other embodiments, the functions of GDMA 411 and IODMA 533 can be integrated into the same component. For ease of description, this disclosure treats GDMA 411 and IODMA 533 as different components. For those skilled in the art, any component that implements functions and achieves technical effects similar to this disclosure falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 411, IODMA 533, CDMA 410, and MVDMA 534 can also be implemented by the same component.

[0056] Neural network models typically include input layers, convolutional layers, activation functions, pooling layers, fully connected layers, etc., ranging from a few layers to hundreds of layers. Each layer executes an operator; for example, a convolutional layer executes a convolution operator. The number of operators executed depends on the number of layers.

[0057] Training a neural network model involves adjusting the parameters of each layer by inputting training samples, aiming to make the model's calculations as close as possible to the actual results. Neural network training includes forward propagation and backpropagation. Forward propagation, based on the existing model, uses input training samples and the neural network's layers to progressively extract abstract features from the input feature maps. Backpropagation, on the other hand, calculates a loss function based on the forward propagation results and the actual values. Using gradient descent and the chain rule, it calculates the partial derivative of the loss function with respect to each parameter to update the parameters. The updated parameters are then used for further training, and this process is repeated multiple times until the forward propagation results meet expectations. Using the trained neural network model to perform forward computation on real-world inputs to complete a given task is called neural network inference.

[0058] Based on the aforementioned hardware environment, this disclosure provides a data processing scheme for executing neural network models, and more specifically, a scheme for optimizing convolution operations in neural network models.

[0059] Figure 6 An exemplary convolution operation example that can be applied to embodiments of this disclosure is shown. As illustrated, convolutional layers in a neural network model can perform feature extraction by applying filters to the input feature map and performing convolution processing.

[0060] The figure exemplifies an input feature map of size 6×6×3, which can represent three 6×6 feature maps (i.e., a 6×6×3 three-dimensional matrix), each representing a different feature. In this example, the width W of the feature map is 6, and the height H is also 6. The number of input feature maps can also be called the number of input channels Ci. For example, the example input in the figure has 3 feature maps, also referred to as 3 feature channels.

[0061] The figure also exemplifies a 2×3×3×3 filter, which can represent two 3×3×3 convolutional kernels (i.e., two 3×3×3 three-dimensional matrices). Each convolutional kernel has three different 3×3 kernels, corresponding to three different input feature maps. The number of stereo convolutional kernels can be called the number of output channels Co, which is 2 in this example. Within each stereo convolutional kernel, the number of two-dimensional convolutional kernels can be called the number of input channels Ci, which is consistent with the number of channels in the input feature map. Each two-dimensional convolutional kernel has a corresponding width Kw and height Kh, both of which are 3 in this example.

[0062] The convolution of the input feature map with the filter outputs two 4×4 feature maps. The convolution of the input feature map with the upper 3D convolution kernel results in the upper 4×4 output feature map, and the convolution of the input feature map with the lower 3D convolution kernel results in the lower 4×4 output feature map. The value at each position in the output feature map is obtained by performing a 2D convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) in the upper output feature map is obtained by performing a 2D convolution operation between the block enclosed by the black cube in the input feature map and the upper 3D convolution kernel, resulting in three values, which are then summed to obtain the final value. To obtain the output at other positions, the position of the convolution kernel can be moved on the input feature map. In the example in the figure, the convolution stride (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0,1) or (1,0) in the upper output feature map can be obtained respectively.

[0063] As described above, a convolutional layer in a neural network contains a set of input feature maps, comprising H × W × Ci pieces of information. H and W represent the height and width of the input feature maps, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the kernels, respectively. The output feature map contains Ho × Wo × Co pieces of information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation involves a stride (Sx, Sy), the size of which affects the size of the output feature map.

[0064] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is typically allocated in contiguous storage space, meaning it can be expanded one-dimensionally and stored sequentially in memory. For example, in this disclosed embodiment, it is stored sequentially according to a low-dimensional priority (Ci is the lowest dimension here). Adjacent dimensions refer to dimensions that are immediately next to each other in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. Adjacent dimensions can also be called consecutive dimensions.

[0065] To accelerate the computation of neural network models, multiple computational units are typically used for parallel processing. For example, Figure 3 The operation module 32 or Figure 5 The computation module 52 can include multiple dedicated convolutional computation units (or convolutional units), each capable of performing complete computations along the (H, W, Ci) dimensions. In other words, computations along the (H, W, Ci) dimensions (Co) can be distributed across Co convolutional units for parallel computation, thereby improving computational speed. Typically, the number of convolutional units is fixed. If the size of the Co dimension is small, there will be idle convolutional units, failing to fully utilize computational resources. In some cases, it may be necessary to align the size of the Co dimension to the number of convolutional units for unified scheduling. However, when the Co dimension is small, this alignment constraint can introduce invalid computations, resulting in significant resource waste.

[0066] On the other hand, to improve memory access speed and fully utilize memory bandwidth, AI chip instruction sets typically require vector alignment. AI chip designs usually use the Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above. Therefore, instruction alignment requires the Ci dimension to be aligned to a specified value, such as the instruction alignment value Aci, so that data access is performed in units of this instruction alignment value Aci. However, when the Ci dimension is small, this alignment constraint can lead to a large amount of redundant computation, resulting in wasted resources.

[0067] In view of this, this disclosure provides a data processing scheme for executing neural network models based on the aforementioned hardware environment. In a first aspect, this disclosure provides a first-dimensional folding scheme that optimizes the Co dimension in convolutional layers, reducing resource waste while meeting the aforementioned hardware alignment constraints. In a second aspect, this disclosure provides a second-dimensional folding scheme that optimizes the Ci dimension in convolutional layers, meeting the aforementioned instruction alignment constraints while reducing resource waste. These two aspects can be implemented individually or in combination. When implemented in combination, the two aspects can complement each other, thereby maximizing the optimization of computational performance. These two aspects are described below.

[0068] Figure 7 An exemplary schematic diagram of the first-dimensional folding scheme of this disclosure embodiment is illustrated by a specific example. Assume the number of convolutional units used to perform the convolution operation is Aco. Depending on the hardware design, Aco can have different values, such as 32, 64, 128, etc. In the example below, Aco = 64 is used as an example. According to the hardware alignment requirements, the Co dimension of the filter needs to be aligned to Aco, that is, aligned to 64.

[0069] The left side of the figure shows the original filter of the convolutional layer, which is represented as, for example, 4×3×3×4, meaning its output channels Co are 4, its input channels Ci are 4, and each convolutional kernel is 3×3. Furthermore, the convolutional stride of the original filter is (Sx, Sy) = (1,1). As can be seen from the figure, the Co dimension of the original filter is much smaller than the hardware alignment requirement (64). According to conventional methods, the Co dimension would be padded with zeros to align to 64. Aligning from 4 to 64 requires a significant amount of redundant computation, resulting in wasted resources.

[0070] The right side of the figure shows the first folded filter according to the embodiment disclosed herein, which is, for example, 64×6×6×4, meaning its output channel number Co' is 64, and the input channel number is the same as the original filter, still 4, with each convolution kernel size being 6×6. Furthermore, the convolution stride of the first folded filter becomes (Sx', Sy') = (4, 4). It can be seen that the output channel number of the first folded filter is aligned to the hardware alignment requirement (64), therefore, the convolution units can be fully utilized for computational processing.

[0071] In the aforementioned first-dimensional folding process, the first folded filter is synthesized from multiple expanded filters obtained by shifting the original filter through multiple convolution strides. This first-dimensional folding process is based on the following consideration: if the original calculation method is followed, a single calculation will waste some convolution units. Therefore, the next calculation, the calculation after that, etc., can be moved to a single calculation, thereby making full use of the convolution units. In other words, the first-dimensional folding scheme of this disclosed embodiment is equivalent to folding the calculations corresponding to multiple convolution strides (or multiple time-based calculations) into a single convolution stride (or a single time period).

[0072] In two-dimensional convolution, the convolution stride includes a horizontal stride Sx in the kernel width direction and a vertical stride Sy in the kernel height direction. Moving the convolution stride once in either direction yields one computation, which corresponds to a filter. This filter is obtained by moving the original filter by one convolution stride. In order to be synthesized with other similarly generated filters later, the width and height of its convolution kernel need to be expanded accordingly based on the convolution stride; this is called an expanded filter.

[0073] Assuming the original filter is shifted by Nw1-1 horizontal convolution strides in the kernel width direction and by Nh1-1 vertical convolution strides in the kernel height direction, then N can be obtained. total1 N extended filters, where N total1 = Nw1 × Nh1, where Nw1 and Nh1 are natural numbers. This N... total1An extended filter can be synthesized into a folded filter along the Co dimension of the output channel, referred to here as the first folded filter.

[0074] Still with Figure 7 In the example above, the Co of the original filter is 4. According to the hardware alignment requirements, it needs to be aligned to 64. Therefore, 64÷4=16 extended filters are needed to synthesize a first folded filter.

[0075] Figure 8 A more detailed schematic diagram of the first-dimensional folding according to an embodiment of this disclosure is shown. The figure illustrates the 16 extended filters required in the aforementioned example. The darker areas in the figure represent the convolution kernels of the original filters. As can be seen from the figure, these 16 extended filters are obtained by shifting the width direction by 3 times (Sx) and the height direction by 3 times (Sy). That is, the width is quadrupled, and the height is also quadrupled, thus resulting in a total of 16 extended filters.

[0076] To combine these 16 extended filters into a single first folded filter along the Co dimension of the output channel, their shapes need to be adjusted. As shown in the figure, zero-padding can be applied to the area excluding the dark portion occupied by the original filters, ensuring that the width and height of the 16 extended filters are consistent. The convolution value calculated by each extended filter (6×6 convolution kernel) is the same as the convolution value calculated by the original filter (3×3 convolution kernel).

[0077] Next, the 16 (4,6,6,4) extended filters can be folded along the output channel Co dimension into a first folded filter of (16*4,6,6,4). At this point, the output channel Co' of the first folded filter is 16*4=64, which is equal to the number of convolutional units, thus meeting the hardware alignment requirements and achieving the goal of fully utilizing the convolutional units.

[0078] from Figure 8 As can be seen during the folding process, the number of output channels of the first folded filter increases exponentially compared to the number of output channels of the original filter. Therefore, the folding scheme of this embodiment is particularly suitable for cases where the number of output channels Co of the original filter is small, for example, Co does not exceed a first threshold A1, and the first threshold A1 is less than a second threshold Aco (i.e., hardware alignment requirement). In some embodiments, the first threshold A1 ≤ Aco / 2. Practical results show that the smaller Co is, the greater the potential for improvement compared to existing algorithms.

[0079] from Figure 8 It can also be seen during the folding process that, based on the parameters of the original filter and the hardware alignment requirements, the parameters of the first folded filter can be determined.

[0080] First, the first total folding factor N can be determined based on the number of output channels Co of the original filter and the hardware alignment requirement Aco. total1 .

[0081] In some embodiments, the total folding factor N can be determined as follows:

[0082] N total1 =Aco / Coa (1-1)

[0083] Where Coa is the nearest Aco / 2 aligned to Co. n The value of , where n is a natural number.

[0084] As previously described, the embodiments disclosed herein align Co to a specified value Aco through multiple folds. When Aco is 2 times Co... n When the factor is multiple, dividing the two directly yields the total number of folds required. For example, in the previous example, Co = 4, therefore N total1 =64 / 4 = 16 times. When Aco is not 2 of Co. n When multiplying by 2, Co needs to be aligned to the nearest Aco / 2 first. n The value. For example, if Aco is 64, then Aco / 2 n Including 32, 16, 8, 4, and 2, Co needs to be aligned to the nearest value among these numbers. For example, if Co = 3, it needs to be aligned to 4; if Co = 5, it needs to be aligned to 8, and so on. After alignment, removing Aco yields the first total folding multiple N. total1 .

[0085] Next, after determining the first total folding multiple N... total1 Then, it can be split into the first width dimension folding factor Nw1 in the kernel width direction and the first height dimension folding factor Nh1 in the kernel height direction. Various different rules can be used for this folding to achieve different advantages.

[0086] In one embodiment, the folding factor can be preferentially split along the kernel width direction. As described later, the convolution operation using the filter folded in the first dimension yields an intermediate result, which still needs to undergo dimension defolding or dimension rearrangement to obtain the final result. When the first dimension folding occurs along the kernel width direction, dimension rearrangement is simple; it only requires adjusting the dimension representation of the intermediate result, or dimension reorganization, without any other processing. Therefore, preferentially splitting the folding factor along the kernel width direction simplifies subsequent processing. For example, in the example above, the first total folding factor of 16 can be split into a width of 16 times and a height of 1 times, thereby simplifying subsequent result processing. In this case, the kernel size of the first folded filter is 3×18. In another implementation, the first total folding factor of 16 in the example above is split into a width of 8 times and a height of 2 times, in which case the kernel size of the first folded filter is 4×10. It can be seen that different splitting methods will affect the kernel size of the first folded filter.

[0087] In another embodiment, the folding factor can be preferentially split into directions with smaller convolution strides. As seen from the preceding folding process, the kernel size of the first folded filter is related to the convolution stride of the original filter; the larger the convolution stride, the larger the kernel of the first folded filter expands. Therefore, preferentially splitting the folding factor into directions with smaller convolution strides helps reduce kernel expansion, thereby reducing computational complexity. For example, assuming the kernel of the original filter is 3×3, (Sx, Sy) = (1,3), a first total folding factor of 16 can be split into a width of 16 times and a height of 1 times, resulting in a kernel size of 3×18 for the first folded filter. If the first total folding factor of 16 is split into a width of 8 times and a height of 2 times, the kernel size of the first folded filter is 6×10. If the first total folding factor of 16 is split into a width of 4 times and a height of 4 times, the kernel size of the first folded filter is 12×6.

[0088] In another embodiment, the folding factor can be evenly distributed across the kernel width and kernel height directions. For example, see the preceding reference... Figure 7 and Figure 8 In the example, the first total fold factor of 16 is divided equally into a width of 4 times and a height of 4 times.

[0089] In another embodiment, the folding factor can be split to minimize the kernel size of the first folded filter, thereby minimizing the added redundant computation.

[0090] Based on the above rules, in some embodiments, the first total folding multiple N can be split as follows: total1 :

[0091]

[0092] Nh1=N total1 / Nw1 (1-3)

[0093] After determining the first width dimension folding factor Nw1 in the width direction and the first height dimension folding factor Nh1 in the height direction, the convolution stride of the first folded filter can be determined based on Nw1, Nh1 and the convolution stride of the original filter; and the convolution kernel size of the first folded filter can be determined based on Nw1, Nh1 and the convolution kernel and convolution stride of the original filter.

[0094] In some embodiments, the convolution stride of the folded filter can be determined as follows:

[0095] S′ x =S x *Nw1 (1-4)

[0096] S y ′=S y *Nh1 (1-5)

[0097] Among them, S x S y These are the horizontal and vertical convolution strides of the original filter, S′, respectively. x S y ′ represents the horizontal convolution stride and the vertical convolution stride of the first folded filter, respectively.

[0098] In some embodiments, the kernel size of the first folded filter can be determined as follows:

[0099] k w ′=k w +(Nw1-1)*S x (1-6)

[0100] k h ′=k h +(Nh1-1)*S y (1-7)

[0101] Where, k w k h These represent the kernel width and kernel height of the original filter, respectively, k w ′、k h ′ represents the kernel width and kernel height of the first folded filter, respectively.

[0102] Therefore, the above describes how to design the parameters of the first folded filter according to the hardware alignment requirements.

[0103] Figure 9An exemplary comparison diagram of the calculation process before and after the first dimension is folded according to an embodiment of this disclosure is shown.

[0104] As can be seen from the first dimension folding process above, the input channel dimension of the first folded filter is the same as that of the original filter. Therefore, no processing is required on the input feature map before convolution operation can be performed directly with the first folded filter.

[0105] As shown in the figure, assume the input feature map is (1, 16, 16, 4), which is four 16×16 feature maps. The upper part of the figure shows the result after performing a convolution operation using the original filter (4, 3, 3, 4) and a convolution stride (Sx, Sy) = (1, 1). The output feature map is (1, 14, 14, 4), which is also four 14×14 feature maps. The area is represented by dark squares in the figure. Those skilled in the art will understand that, for clarity, the data in the Ci dimension is not shown, and only numerical values ​​are used for illustration.

[0106] In contrast, the lower part of the figure shows the result obtained after performing convolution operation using the first folded filter (64,6,6,4) and convolution stride (Sx', Sy') = (4,4), with the output feature map being (1,4,4,64), which is 64 4×4 feature maps.

[0107] As can be seen from the comparison in the figure, although the values ​​calculated by each expanded filter (6×6 convolution kernel) are the same as those calculated by the original filter (3×3 convolution kernel), the positions of the data values ​​change because the calculation of multiple convolution strides is performed in one operation; in other words, the data values ​​are rearranged in dimensions. The intermediate result obtained by performing convolution operations on the input feature map using the first folded filter can be rearranged in dimensions again, i.e., dimensionally defolded, to obtain the final output feature map, thus making it completely consistent with the result of the original filter. The dimensions of the intermediate result can be represented, for example, as (N, ceil(Ho / Nh1), ceil(Wo / Nw1), Nh1*Nw1*Coa), where ceil() represents the floor function, and Coa is the nearest Aco / 2 aligned to Co. nThe values ​​are as follows. For example, in the example in the figure, Coa = Co = 4, Nh1 = Nw1 = 4, ceil(Ho / Nh1) = ceil(14 / 4) = 4, ceil(Wo / Nw1) = ceil(14 / 4) = 4. Due to the convolution kernel dilation and increased convolution stride caused by the first dimension folding of the filter, and depending on the size of the input feature map, there may be some redundant calculations in the folded convolution operation. When the intermediate results are rearranged in dimensions, these redundant calculations will appear on the outermost side of the truly desired result, specifically the rightmost and bottommost sides. For example, in the example in the figure, the intermediate result (1,4,4,64) can be rearranged in dimensions to obtain (1,16,16,4), which has some redundant areas compared to the original result (1,14,14,4), as represented by the white square area in the inverted "L" shape in the figure. At this time, it is only necessary to remove these redundant areas.

[0108] As described above, the first-dimensional fold can be distributed along the width and / or height of the convolutional kernel. Folding in different directions will result in different dimensional rearrangement results, thus requiring different processing methods. In the following description, for simplicity, the redundant regions mentioned above are not considered. However, those skilled in the art will understand from the description herein that even in cases where redundant regions exist, dimensional rearrangement can be performed as described below, followed by redundancy removal.

[0109] In one embodiment, when only the kernel width W is folded during the first dimension folding, that is, when the extended filter only contains the extended filter obtained by moving the original filter in the kernel width W direction, the intermediate result can be directly re-dimensioned to obtain the output feature map.

[0110] As mentioned earlier in the embodiments disclosed herein, the dimensions of the multidimensional data are represented as (Co, H, W, Ci), meaning the data is stored in memory in the order Co, H, W, Ci, with Ci being the lowest dimension. Those skilled in the art will also understand that splitting or merging adjacent dimensions in multidimensional data does not affect the position of each data element in memory.

[0111] In this disclosed embodiment, based on the characteristics of convolution operations, the dimensions W and Ci are either adjacent or continuous. Therefore, if only the W direction is folded, the intermediate result (N, Ho, Wo / Nw, Nw*Co) can be directly regarded as (N, Ho, Wo, Co). That is, the dimensions W and Ci can be reassembled from (Wo / Nw, Nw*Co) to (Wo, Co). This reassembly does not affect the storage location of each data element of the intermediate result in memory.

[0112] In another embodiment, when only the kernel height H is folded during the first dimension folding—that is, when the expanded filter only contains the expanded filter obtained by moving the original filter along the kernel height H—it is necessary to perform a dimension transpose on the intermediate result to obtain the output feature map. Dimension transpose changes the position of each data element of the multidimensional data in memory.

[0113] Specifically, the features of the intermediate result (N,Ho / Nh,Wo,Nh*Co) along the Co dimension from (Co,(Nh-1)*Co] can be transposed back to the H dimension to obtain (N,Ho,Wo,Co).

[0114] Figure 10 An exemplary schematic diagram of dimension transposition according to an embodiment of this disclosure is shown.

[0115] like Figure 10 As shown, assuming Nh1 = 4, the intermediate result on the left includes four times the features in the H direction along the Co dimension, namely Co0, Co1, Co2, and Co3. In this case, the Co0 feature can be retained along the Co dimension, and the remaining Co1, Co2, and Co3 features can be transferred to the H dimension through dimension transpose, as shown in the final result on the right.

[0116] In another embodiment, when folding is performed simultaneously in the kernel width W direction and the height H direction during the first dimension folding, that is, when the extended filter includes both the extended filter obtained by moving the original filter in the kernel width direction and the extended filter obtained by moving the original filter in the kernel height direction, the intermediate results can be processed by combining the two embodiments described above.

[0117] In one implementation, the intermediate result can be first restructured in terms of dimensions, and then transposed in terms of dimensions to obtain the output feature map. Specifically, the intermediate result (N, Ho / Nh, Wo / Nw, Nh*Nw*Co) can be regarded as (N, Ho / Nh, Wo, Nh*Co), that is, the features in the Co dimension folded from the W direction are regarded as features in the W direction. Then, the dimension transpose operation is performed on it, transposing the features in the Co dimension from (Co, (Nh-1)*Co] to the height H dimension, thus obtaining (N, Ho, Wo, Co).

[0118] In another implementation, the intermediate results can be transposed first, and then the dimensions can be recombined to obtain the output feature map.

[0119] Specifically, the intermediate result (N, Ho / Nh, Wo / Nw, Nh*Nw*Co) can be transposed first. For example, still referring to... Figure 10First, we transpose Co1, Co2, and Co3 along the Co dimension to the H dimension, where Co0 = Co1 = Co2 = Co3 = Nw * Co. After transposing, the intermediate result becomes (N, Ho, Wo / Nw, Nw * Co). Then, we treat each Co0, Co1, Co2, and Co3 as Nw Co, thus obtaining (N, Ho, Wo, Co).

[0120] The above describes a scheme for adjusting a filter by folding a first dimension according to embodiments of this disclosure. In some embodiments, the first folded filter may be generated offline. For example, during inference using a neural network model, a pre-arranged, offline-generated first folded filter can be used to perform convolution operations with the input feature map for the inference process. In other embodiments, the first folded filter may be generated online. For example, during training a neural network model, the filters of the convolutional layers can be folded online and then convolved with the training data for the training process.

[0121] Regardless of which process utilizes the first folded filter of this disclosure embodiment, achieving Co-dimensional alignment through first-dimensional folding can significantly optimize the computational cost of convolution operations. The performance of the first-dimensional folding scheme of this disclosure embodiment compared with existing convolution operations is discussed below regarding computational cost.

[0122] Let P represent the computational complexity of convolution, and A ci Indicates the value after Ci alignment, A CO This represents the value after Co alignment.

[0123] Then P = n * A ci *A CO *H o *W o *k w *k h (1-8)

[0124] Hardware employing the NHWC dimension arrangement order often needs to be aligned to A because the Ci dimension is the lowest dimension and vector instruction alignment requirements exist. ci , so A ci This addresses the requirements for vector instruction alignment; AI computing acceleration hardware typically has multiple parallel, high-performance convolutional computation units, hence A CO This addresses the requirement for alignment of the Co dimension of the convolution kernel, where the value represents the number of high-performance parallel computing units.

[0125] Before optimization, the computational cost of existing convolution operations is:

[0126] P before1 =n*A ci *ACO *H o *W o *k w *k h (1-9)

[0127] After optimization using the first-dimensional folding scheme of this disclosure embodiment, the computational cost of convolution operations is:

[0128]

[0129] The performance optimization rate after the first-dimensional folding is:

[0130]

[0131] References above Figures 7-8 For example, as described,

[0132] Before optimization, P before1 =1*64*64*ho*wo*3*3 (Sx,Sy=1,1)

[0133] After optimization, P after1 =1*64*64*(ho / 4)*(wo / 4)*6*6 (Sx,Sy=4,4)

[0134] = 1*64*64*ho*wo*3*3*(1 / 4)

[0135] The amount of convolution computation is reduced by 75% by the number of convolution computation units.

[0136] As can be seen from the above comparison of computational complexity, the first folded filter scheme provided in this embodiment can effectively save the computational complexity of convolution units, thereby improving the computational performance of convolution operations.

[0137] The first-dimensional folding scheme described above can be implemented alone to adjust the output channel dimension of the filter, for example, when the output channel dimension of the filter is small. However, the first-dimensional folding scheme does not change the input channel dimension of the filter. In the second-dimensional folding scheme disclosed herein, the input channel dimension of the filter can be adjusted. The second-dimensional folding scheme can be implemented alone (e.g., when the input channel dimension of the filter is small), or it can be implemented in combination with the first-dimensional folding scheme (e.g., when both the input channel dimension and the output channel dimension of the filter are small). The second-dimensional folding scheme of the embodiments of this disclosure is described below. Those skilled in the art will understand that the second-dimensional folding can be performed based on the original filter, or it can be performed based on the first folded filter obtained by first-dimensional folding of the original filter. Therefore, based on the context, the filter to be folded mentioned herein can be either the original filter or the first folded filter.

[0138] Figure 11 An exemplary schematic diagram of the second-dimensional folding scheme of this disclosure embodiment is illustrated by a specific example. Assume the alignment value required by the instruction alignment is Aci. Depending on the instruction set design, Aci can have different values, such as 32, 64, 128, etc. In the following example, Aci = 64 is used as an example. According to the instruction alignment requirements, the Ci dimension of the filter needs to be aligned to Aci, that is, aligned to 64.

[0139] The left side of the figure shows the filter to be folded in a convolutional layer, for example, represented as 64×4×4×4, meaning its output channel Co is 64, its input channel Ci is 4, and each convolutional kernel is 4×4 in size. As can be seen from the figure, the Ci dimension of the filter to be folded is much smaller than the alignment requirement of the instruction (64). According to conventional methods, the Ci dimension would be padded with zeros to align to 64. Aligning from 4 to 64 requires a significant amount of redundant computation, resulting in a waste of resources.

[0140] The right side of the figure shows the second folded filter after performing the second-dimensional folding according to the embodiment of this disclosure. It is represented, for example, as 64×1×1×64, that is, its input channel number Co' is 64, and its output channel number is the same as that of the filter to be folded, both being 64. Each convolutional kernel size is 1×1. It can be seen that since the data in the width and height dimensions of the convolutional kernel has been transferred to the input channel dimension, the number of input channels of the second folded filter has been aligned to the alignment requirement of the instruction (64). At this time, there is no need to add zeros for Ci dimension alignment, thus avoiding the redundant calculation caused by zero padding in the past.

[0141] In the aforementioned second-dimensional folding process, the second folded filter is obtained by folding the filter to be folded through a second-dimensional folding process. This second-dimensional folding process is based on the following considerations: if the original calculation method is used to achieve Ci alignment by padding with zeros, redundant calculations will occur, wasting computational resources. If data from other dimensions is transferred to the Ci dimension and the Ci dimension is filled to the instruction alignment value, the waste of computational resources can be minimized, improving computational efficiency.

[0142] Figure 12 A more detailed schematic diagram of second-dimensional folding according to an embodiment of this disclosure is shown. The figure illustrates... Figure 11 The example illustrates the second-dimensional folding process of the filter to be folded (64,4,4,4). The Ci of the filter to be folded is 4, and according to the alignment requirements, it needs to be aligned to 64. Therefore, N is required. total2=64÷4=16 times folding. In the example in the figure, the second total folding factor of 16 is allocated as the second width dimension folding factor Nw2 in the kernel width direction (4 times folding in the figure) and the second height dimension folding factor Nh2 in the kernel height direction (4 times folding in the figure).

[0143] As shown in the figure, preferably, the width W dimension can be folded first. Figure 1201 shows that a single layer of W-dimensional data can be divided into four folds or four segments, and then arranged sequentially on the input channel Ci dimension. When the filter to be folded 1200 is completely folded on the W dimension, the intermediate filter shown in 1210 can be obtained, whose dimension can be represented as (64, 4, 1, 16).

[0144] Next, based on the intermediate filter 1210, folding can be performed along the height H dimension. As shown in the figure, a 4x fold is also performed along the H dimension. The H dimension data of the intermediate filter 1210 is divided into four folds or four segments, and then arranged sequentially along the input channel Ci dimension. During this folding process, each of these four segments along the H dimension corresponds to the data previously folded along the single-layer W dimension. After complete folding along the H dimension, the second folded filter shown in 1220 can be obtained, whose dimension can be represented as (64, 1, 1, 64). The input channel Ci dimension of the second folded filter is 64, which satisfies the alignment requirements of the instruction.

[0145] The above describes a folding method where the width (W) dimension is folded first, followed by the width (H) dimension. Those skilled in the art will understand that it is also possible to fold the width (H) dimension first, followed by the width (W) dimension. However, since the width (H) dimension and the height (Ci) dimension are not adjacent, additional processing, such as data transposition, is required compared to folding the width (W) dimension first.

[0146] As seen from the preceding second-dimensional folding process, the input channel Ci dimension of the second-folded filter changes. Therefore, the input feature map also needs to undergo the same type of second-dimensional folding. Since the input feature map and the filter are folded at the same ratio, the output after performing a convolution operation between the second-dimensional folded input feature map and the second-folded filter is the same as the output of the convolution operation before folding, requiring no further processing of the output.

[0147] Furthermore, from Figure 12As can be seen in the second-dimensional folding process, the number of input channels of the second-folded filter increases exponentially relative to the number of input channels of the filter to be folded. Therefore, the second-dimensional folding scheme of this disclosure embodiment is particularly suitable for cases where the number of input channels Ci of the filter is small, for example, Ci does not exceed the third threshold A2, and the third threshold A2 is less than the instruction alignment value Aci. In some embodiments, the third threshold A2 ≤ Aci / 2. Practical results show that the smaller Ci is, the greater the potential for improvement compared to existing algorithms.

[0148] from Figure 12 It can also be seen in the second-dimensional folding process that, based on the parameters of the filter to be folded and the instruction alignment requirements, the parameters of the second folded filter can be determined.

[0149] First, the second total folding factor N can be determined based on the number of input channels Ci of the filter to be folded and the instruction alignment value Aci. total2 .

[0150] In some embodiments, the second total folding multiple N can be determined as follows: total2 :

[0151] N total2 =Aci / Cia (2-1)

[0152] Where Ci is the nearest Aci / 2 aligned to Ci. n The value of , where n is a natural number.

[0153] As previously described, the embodiments disclosed herein align Ci to the instruction alignment value Aci by folding the ci multiple times. When Aci is 2 of Ci... n When the factor is multiple, dividing the two directly yields the total multiple required for folding in the second dimension. For example, in the previous example, Ci = 4, therefore N total2 =64 / 4 = 16 times. When Aci is not a multiple of Ci. n When the value is multiplied, Ci needs to be aligned to the nearest Aci / 2 first. n The value. For example, if Aci is 64, then Aci / 2 n Including 32, 16, 8, 4, and 2, Ci needs to be aligned to the nearest value among these numbers. For example, if Ci = 3, it needs to be aligned to 4; if Ci = 5, it needs to be aligned to 8, and so on. After alignment, removing Aci yields the second total folding factor N. total2 .

[0154] Next, after determining the second total folding multiple N... total2Following the previous considerations, it can then be decomposed into a second width dimension folding factor Nw2 along the kernel width direction and a second height dimension folding factor Nh2 along the kernel height direction. Various different rules can be applied to this folding to achieve different advantages.

[0155] In one embodiment, the folding factor can be evenly split across the kernel width and kernel height directions. For example, see the previous reference. Figure 11 and Figure 12 In the example, the second total fold factor of 16 is divided equally into a width of 4 times and a height of 4 times.

[0156] In another embodiment, the folding factor can be preferentially split along the kernel width W direction. As mentioned earlier, with the data arrangement order NHWC, the W and Ci dimensions are continuous. Therefore, folding along the W dimension is relatively simple to implement, requiring only adjustment of the filter's dimensional representation, or dimensional reorganization, without any other processing. Therefore, preferentially splitting the folding factor along the kernel width W direction simplifies the processing.

[0157] The splitting of the second total fold multiple is also affected by the following two factors.

[0158] On the one hand, depending on the kernel size of the filter to be folded, if the values ​​of each dimension (Kw and Kh) of the kernel are insufficient to achieve the desired folding ratio, other measures need to be taken.

[0159] For example, in the example above, if the principle of prioritizing the width W direction is followed, the second total folding factor of 16 can be split into a width of 16 times and a height of 1. However, since the convolution kernel width Kw of the filter to be folded is only 4, it can only support a maximum of four folds. Therefore, the distribution of folding factors needs to be adjusted. For example, the splitting factor for the width W dimension can be reduced, such as splitting it into a width of 4 times and a height of 4 times.

[0160] For example, if the kernel width dimension and / or kernel height dimension cannot be 2... n Folding is necessary when, for example, Kw or Kh is odd and not divisible by 2, a two-fold fold is not possible; and when not divisible by 4, a four-fold fold is not possible. In such cases, padding is required based on the folding factor to fold the convolution kernel.

[0161] Figure 13 The illustration schematically depicts padding of a convolutional kernel according to an embodiment of this disclosure. The example assumes the convolutional kernel of the filter to be folded is 3×3, but both the width (W) and height (H) dimensions require four folds, necessitating padding in both the W and H directions to multiples of 4.

[0162] Figure 13The padding in the H and W directions is illustrated separately. The single-layer convolutional kernel 1301 to be folded has a size of 3 in the W direction. To perform a four-fold fold, it needs to be padded to 4, as shown in 1311, where the white squares represent padding blocks. Similarly, in the H direction, the single-layer convolutional kernel 1301 to be folded has a size of 3 in the H direction. To perform a four-fold fold, it needs to be padded to 4, as shown in 1312, where the white squares represent padding blocks. When padding is performed simultaneously in both the H and W directions, after final padding from the convolutional kernel 1300 to be folded, convolutional kernel 1310 is obtained, whose kernel size becomes 4×4, and both its width and height can be four-folded.

[0163] from Figure 13 It can be seen that padding the convolution kernel introduces invalid values, resulting in unnecessary redundant calculations.

[0164] Therefore, in some embodiments, to achieve folding, the second total folding factor can be split to minimize the padding caused by folding factor alignment. For example, assuming the size of the convolutional kernel to be folded is 1×6 and the second total folding factor is 16, if it is split into 4 times in the H direction and 4 times in the W direction, the convolutional kernel to be folded needs to be padded to 4×8, with 26 additional zero-padded values ​​per layer; if it is split into 2 times in the H direction and 8 times in the W direction, the convolutional kernel to be folded only needs to be padded to 2×8, with 10 additional zero-padded values ​​per layer; if it is split into 1 times in the H direction and 16 times in the W direction, the convolutional kernel to be folded needs to be padded to 1×16, with 10 additional zero-padded values ​​per layer. With the same padding amount, it is preferable to allocate the folding factor to the W direction, for example, choosing to split it into 1 times in the H direction and 16 times in the W direction.

[0165] On the other hand, the actual effective multiple of the second-dimensional folding scheme in this embodiment is also related to the convolution stride. When the convolution stride cannot be divided evenly by the corresponding folding multiple, the folding method of the convolution kernel remains unchanged; however, the folding of the input feature map will overlap, resulting in some redundant computation in the convolution operation.

[0166] Figure 14 This diagram illustrates the effect of the convolution stride on the effective fold according to embodiments of this disclosure. The figure uses the H-direction as an example to show how the input feature map contains overlapping regions. As shown, when the convolution stride Sy = 4 in the H-direction, a four-fold is performed with two consecutive data blocks α and β in the H-direction, and there is no overlapping region. However, when the convolution stride Sy = 2 in the H-direction, a four-fold is performed with two consecutive data blocks γ and δ in the H-direction, and an overlapping region exists. The size of this overlapping region is the folding factor in that direction minus the convolution stride in that direction.

[0167] At this point, the folding factors in the H and W directions can be flexibly configured. For example, folding factors can be preferentially allocated to higher dimensions to avoid redundant calculations in lower dimensions. For instance, in the NHWC data arrangement, the H dimension is higher than the W dimension; if Sx = 2, Sy = 2, N... total If 2 = 16, then it can be allocated as a two-fold division in the W dimension and an eight-fold division in the H dimension. In this way, the redundant calculations caused by overlapping regions are distributed along the H dimension. At this time, when performing these redundant calculations, since the H dimension is higher, more data needs to be read in each operation, which helps to improve data access I / O efficiency.

[0168] In summary, in the embodiments disclosed herein, the folding multiples of the H and W dimensions can be flexibly configured based on the various rules described above, so as to avoid redundant calculations as much as possible and maximize the improvement of computational efficiency.

[0169] In some embodiments, the second total folding factor can be split as follows: The folding factor in the W direction is determined preferentially; for example, the second total folding factor N can be split in an average manner. total2 To determine the folding factor Nw2 in the W direction:

[0170]

[0171] For example, with the filter to be folded (64,6,6,4), the convolution stride (S) x ′, S y Let's take (2,2) as an example, with the instruction alignment value Aci = 64. At this time,

[0172] Next, the dimensions in the W direction and the convolution stride can be determined accordingly.

[0173] As needed, the kernel width k of the filter to be folded can be first set. w Alignment with the multiple of the folding factor Nw2 in the W direction, denoted as k. wa Then, calculate the size k of the second-dimensional convolution kernel W after folding. w ":

[0174] k″ w =k wa / Nw2 (2-3)

[0175] Continuing with the example above, since k w ′ is 6, while Nw2 is 4, so a padding operation is needed. At this time, k wa =8, then k″ w =k wa / Nw2=2.

[0176] Next, the convolution stride S in the W direction of the second folded filter can be determined as follows: x ":

[0177]

[0178] Continuing with the example above, S at this point x ′ / Nw2=2 / 4<1, therefore S x " = 1. This means that if a four-fold is performed along the W dimension, there will be overlapping regions, introducing redundant calculations. The actual effective folding ratio along the W dimension is S. x ′ / S x "=2 / 1=2, not the 4 times that corresponds to a four-fold discount."

[0179] At this point, the folding ratio can be adjusted. For example, the maximum effective folding ratio of 2 can be achieved in the W direction without overlapping areas, and the remaining required folding ratio can be allocated to the H direction. Therefore, the folding ratio in the H direction can be calculated based on the effective folding ratio in the W direction as follows:

[0180] Nh2=N total2 / (S x ′ / S x (2-5)

[0181] In the example above, Nh2 = 16 / 2 = 8, so the folding ratio in the H direction is 8 times.

[0182] At this point, the folding ratio in the W direction can be updated accordingly:

[0183] Nw2=N total2 / Nh2 (2-6)

[0184] In the example above, Nw2 = 16 / 8 = 2, which is equal to its maximum effective folding ratio. With this folding ratio allocation method, there will be no overlapping area in the W direction, but there will be more overlap in the H direction.

[0185] Once the second folding factors Nw2 and Nh2 in the W and H directions are determined, the remaining parameters in each dimension, including dimension size and convolution stride, can be calculated accordingly.

[0186] Similar to the method described earlier for calculating the dimension size of W and the convolution stride, the kernel width k of the filter to be folded can be first determined as needed. w Alignment with the multiple of the folding factor Nw2 in the W direction, denoted as k. wa The height k of the convolution kernel of the filter to be folded h The multiple of the folding factor Nh2 aligned to the H direction is denoted as k. haThen, the size k of the convolution kernel W dimension after folding in the second dimension is calculated as follows. w "and the size k of the H dimension h ":

[0187] k w "=k wa / Nw2 (2-7)

[0188] k h "=k ha / Nh2 (2-8)

[0189] Continuing with the example above, since k w Since ' is 6 and Nw2 is 2, both are divisible, so no padding operation is needed. At this point, k... wa =6, then k w " = 6 / 2 = 3. For the H direction, since k h Since k' is 6 and Nh2 is 8, they are not divisible, so a padding operation is needed. ha =8, then k h ″=8 / 8=1.

[0190] Next, the convolution stride S in the W direction of the second folded filter can be determined as follows: x The convolution stride S in the H direction y ":

[0191]

[0192]

[0193] For the example above, S x " = S x ′ / Nw2=2 / 2=1; due to S y ′ / Nh2=2 / 8<1, therefore, S″ y =1.

[0194] The above describes how to design the parameters of the second folded filter according to the instruction alignment requirements.

[0195] The above describes a scheme for adjusting filters by second-dimensional folding according to embodiments of this disclosure. In some embodiments, the second-folded filter may be generated offline. For example, during inference using a neural network model, a pre-arranged, offline-generated second-folded filter can be used to perform convolution operations with an input feature map that has undergone the same second-dimensional folding online for the inference process. In other embodiments, the second-folded filter may be generated online. For example, during training a neural network model, the filters of the convolutional layers can be second-dimensionally folded online, and the training data can be second-dimensionally folded online simultaneously, and then convolution operations can be performed on both to perform the training process.

[0196] Regardless of which process utilizes the second folded filter of this disclosure embodiment, achieving Ci-dimensional alignment through second-dimensional folding can significantly optimize the computational cost of convolution operations. The performance of the scheme in this disclosure embodiment is compared with existing convolution operations in terms of computational cost.

[0197] Let P represent the computational complexity of convolution, and A ci Indicates the value after Ci alignment, A CO This represents the value after Co alignment.

[0198] Then P = n * A ci *A CO *H o *W o *k w *k h (2-11)

[0199] Hardware employing the NHWC dimension arrangement order often needs to be aligned to A because the Ci dimension is the lowest dimension and vector instruction alignment requirements exist. ci , so A ci This addresses the requirements for vector instruction alignment; AI computing acceleration hardware typically has multiple parallel, high-performance convolutional computation units, hence A CO This addresses the requirement for alignment of the Co dimension of the convolution kernel, where the value represents the number of high-performance parallel computing units.

[0200] Before optimization, the computational cost of existing convolution operations is:

[0201] P before2 =n*A ci *A CO *H o *W o *k w ′*k h (2-12)

[0202] After optimization using the second-dimensional folding scheme of this disclosure embodiment, the computational cost of convolution operations is:

[0203] P after2 =n*A ci *A CO *H o *W o *k″ w *k h "

[0204] =n*A ci *A CO *H o *W o *alignTo(k w ′,Nw2)*alignTo(k h ′,Nh2) / (Nw2*Nh2) (2-13)

[0205] The performance optimization rate after the second-dimensional fold is:

[0206]

[0207]

[0208] References above Figures 11-12 For example, as described,

[0209] Before optimization, P before2 =1*64*64*ho*wo*4*4(Sx,Sy=4,4)

[0210] After optimization, P after2 =1*64*64*ho*wo*1*1(Sx',Sy'=1,1)

[0211] = 1*64*64*ho*wo*4*4*(1 / 16)

[0212] The amount of convolution computation reduced by 93.75% for the reduced number of convolution computation units.

[0213] Understandable. Figures 11-12 The example illustrates the ideal scenario of second-dimensional folding, where the convolution stride perfectly satisfies the folding requirement, and the kernel size and the input feature map size do not need to be aligned by folding factor. In reality, depending on the specific parameter values, the actual optimization rate will be lower than the peak optimization rate of 93.75% mentioned above.

[0214] As can be seen from the above comparison of computational complexity, the second folded filter scheme provided in this embodiment can effectively save the computational complexity of convolution units, thereby improving the computational performance of convolution operations.

[0215] The first-dimensional folding and the second-dimensional folding according to embodiments of this disclosure have been described above. As can be seen from the foregoing description, when the first-dimensional folding is performed alone, the convolution kernel of the resulting first-folded filter expands, especially when the convolution stride before folding is large, expanding to a very large size. The convolution stride also expands, leading to redundant computation and increased access time for loading the filter. Conversely, when the second-dimensional folding is performed alone, the convolution kernel of the resulting second-folded filter shrinks, and the convolution stride also shrinks. Therefore, combining the first-dimensional folding and the second-dimensional folding reduces the convolution kernel expansion caused by the first-dimensional folding, thereby reducing the memory access requirements for loading the filter.

[0216] On the other hand, when the second-dimensional fold is implemented alone, its effective folding factor depends on the convolution stride. If the convolution stride is too small, there will be redundant computation when performing convolution operations with the folded input feature map, reducing the benefits gained from the fold. The preceding analysis shows that the first-dimensional fold increases the convolution kernel and convolution stride, and the convolution stride after the first-dimensional fold is necessarily even. Therefore, combining the first-dimensional fold with the second-dimensional fold can increase the effective folding factor of the second-dimensional fold, and avoids the need to change the folding method of the convolution kernel when the convolution stride is odd, thus maximizing the optimization effect of the second-dimensional fold.

[0217] Based on the above analysis, the first-dimensional folding and the second-dimensional folding can be combined to compensate for each other's shortcomings and achieve the maximum optimization effect.

[0218] In one embodiment, it is preferable to first perform a first-dimensional fold, thereby obtaining an enlarged convolution kernel and convolution stride; then, based on the first-dimensional fold, a second-dimensional fold is performed, at which point the convolution kernel and convolution stride are reduced. Thus, while satisfying the instruction alignment requirements of the input channel dimension and the hardware alignment requirements of the output channel dimension, the I / O of the loading filter can be increased, and the redundant computation generated by the folding process can be offset, maximizing the improvement in computational performance.

[0219] For example, suppose the original filter is (4,3,3,4) with a convolution stride of (1,1). Referring to the previous description, after the first-dimensional fold (where Nw1 = Nh1 = 4), the resulting first-folded filter is (64,6,6,4) with a convolution stride of (4,4). Next, applying the second-dimensional fold (where Nw2 = Nh2 = 4) to the first-folded filter yields the second-folded filter (64,2,2,64) with a convolution stride of (1,1). Compared to the original filter, the second-folded filter aligns both its input and output channel dimensions to their respective alignment values ​​(64 in this case), and the kernel size is reduced to 2×2 without expansion. The convolution stride is also not expanded, remaining at (1,1) in this example.

[0220] In another embodiment, the second-dimensional fold can be performed first, followed by the first-dimensional fold. However, this implementation cannot fully utilize the kernel expansion and stride increase brought about by the first-dimensional fold, thus hindering performance optimization.

[0221] When the first dimension is folded first, followed by the second dimension, the performance optimization in terms of convolution computation can be calculated as follows:

[0222] Pa = 1 - (alignTo(k) w ′,Nw2)*alignTo(k h ′,Nh2) / (Nw2*Nh2*k w '*k h ′))*(k w ′*k h ′ / (Nh1*Nw1*k w *k h ))

[0223] =1-alignTo(k) w ′,Nw2)*alignTo(k h ′,Nh2) / (Nw2*Nh2*Nh1*Nw1*k w *k h ))

[0224] Taking the example given above, where the original filter is (4,3,3,4) and the convolution stride is (1,1), and the first and second dimension folds are performed, the performance optimization rate is:

[0225] Pa = 1 - alignTo(k) w ′,Nw2)*alignTo(k h ′,Nh2) / (Nw2*Nh2*Nh1*Nw1*k w *k h )

[0226] =1-alignTo(6,4)*alignTo(6,4) / (4*4*4*4*3*3)=1-(8*8) / (4*4*4*4*3*3)=1-1 / 36=97.22%

[0227] This disclosure also provides a data processing apparatus for executing a neural network model, and a method for executing a neural network model implemented by the data processing apparatus.

[0228] Figure 15 A schematic structural diagram of a data processing apparatus that can implement embodiments of this disclosure is shown as an example. Figure 15 As shown, the data processing device 1500 includes a processing circuit 1510 and a storage circuit 1520.

[0229] The processing circuit 1510 is responsible for handling various functions on the data processing device 1500, including but not limited to control, decoding, and calculation. The processing circuit 1510 may include, for example, […]. Figure 3 The control module 31 and / or the arithmetic module 32 are included.

[0230] In some embodiments, the processing circuit 1510 may be configured to perform a second-dimensional folding on the input feature map to obtain a folded feature map; then perform a convolution operation on the folded feature map using the folding filter of this disclosure embodiment to obtain an intermediate result; and perform dimensional defolding on the intermediate result to obtain an output feature map.

[0231] The storage circuit 1520 can be used to store or transfer relevant data, such as... Figure 3 or Figure 5 The various RAMs shown are referred to as on-chip caches. In some embodiments, the storage circuit 1520 can be configured to store folded filters for the convolutional layers of a neural network model. These folded filters are generated from the original filters through a first-dimensional fold and a second-dimensional fold. The first-dimensional fold includes synthesizing multiple expanded filters obtained after multiple convolutional stride shifts. The second-dimensional fold includes rearranging data in the width and / or height dimensions to the input channel dimensions. For example, for data rearrangement or dimension folding in the width dimension, the processing circuit 1510 can achieve this through dimension recombination. For data rearrangement or dimension folding in the height dimension, the processing circuit 1510 can achieve this through dimension transpose.

[0232] In some embodiments, the data processing apparatus 1500 can be configured to perform a training process for a neural network model. In this case, the processing circuit 1510 can be configured to perform first and second dimension folding processing on the filters of the convolutional layers of the neural network model according to the present disclosure embodiments, and to perform second dimension folding processing on the training data according to the present disclosure embodiments, online during training. Then, convolution operations are performed on the folded training data using the obtained folded filters to perform the training process. The specific folding processing performed by the processing circuit 1510 can be referred to the foregoing description, and will not be repeated here.

[0233] In other embodiments, the data processing device 1500 can be configured to perform the inference process of a neural network model. In this case, the processing circuit 1510 can be configured to first perform a second-dimensional folding process on the input neurons, and then directly use the folded filters already stored in the storage circuit 1520 to perform convolution operations on the folded input neurons for the inference process. The second-dimensional folding method of the input neurons is consistent with the second-dimensional folding method experienced by the stored folded filters.

[0234] The intermediate results obtained from convolution operations using folded filters need to undergo dimensionality rearrangement to obtain the final output feature map. The specific rearrangement process is described above and will not be repeated here.

[0235] Figure 16 An exemplary flowchart of a data processing method according to an embodiment of this disclosure is shown.

[0236] As shown in the figure, the data processing method 1600 includes step 1610, in which the processing circuit performs a second-dimensional folding on the input feature map to obtain a folded feature map.

[0237] Next, in step 1620, the processing circuit performs a convolution operation on the folded feature map using the folded filters stored in the storage circuit to obtain an intermediate result. The folded filters are generated by folding the original filters in a first dimension and then in a second dimension. The first dimension folding includes synthesizing multiple expanded filters obtained after multiple convolution stride shifts. The second dimension folding includes rearranging the data in the width and / or height dimensions to the input channel dimensions.

[0238] Finally, in step 1630, the processing circuit performs dimensional defolding on the intermediate results to obtain the output feature map.

[0239] Those skilled in the art will understand that the filter folding method and the dimensional rearrangement of intermediate results described above in conjunction with the accompanying drawings in the embodiments of this disclosure can also be applied to... Figure 15 Data processing devices and Figure 16 The data processing methods are already described, so they will not be repeated here.

[0240] This disclosure also provides a chip that may include the data processing apparatus of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a circuit board that may include the aforementioned chip.

[0241] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0242] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0243] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0244] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0245] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0246] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0247] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A data processing apparatus for executing a neural network model, comprising: The storage circuit is configured to store folded filters of the convolutional layers of the neural network model. The folded filters are generated by folding the original filters in a first dimension and folding them in a second dimension. The first dimension folding includes synthesizing multiple expanded filters obtained after multiple convolutional stride shifts. The second dimension folding includes rearranging the data in the width dimension and / or height dimension to the input channel dimension. as well as Processing circuitry, configured for: The input feature map is folded in the second dimension to obtain a folded feature map; The folded filter is used to perform a convolution operation on the folded feature map to obtain an intermediate result; as well as The intermediate results are defolded in dimension to obtain the output feature map.

2. The data processing apparatus according to claim 1, wherein the folded filter is generated by first folding the original filter in a first dimension to obtain a first folded filter, and then folding the first folded filter in a second dimension.

3. The data processing apparatus of claim 2, wherein the processing circuitry is configured to perform the first dimension folding as follows: The original filter is shifted Nw1-1 times in the horizontal convolution stride direction along the kernel width direction, and / or Nh1-1 times in the vertical convolution stride direction along the kernel height direction to obtain N total1 N extended filters, where N total1 = Nw1 × Nh1, where Nw1 and Nh1 are natural numbers; and The N total1 The extended filters are combined into the first folded filter along the output channel dimension.

4. The data processing apparatus according to claim 3, wherein the output channel dimension of the original filter does not exceed a first threshold A1, and the output channel dimension of the first folded filter is equal to a second threshold Aco, wherein the first threshold A1 is less than the second threshold Aco.

5. The data processing apparatus according to claim 4, wherein the processing circuit is further configured to: The first total folding factor N is determined based on the number of output channels Co of the original filter and the second threshold Aco. total1 ; The first total folding factor N total1 It is split into the first width dimension folding factor Nw1 and the first height dimension folding factor Nh1; Based on Nw1, Nh1, and the convolution stride of the original filter, the convolution stride of the first folded filter is determined; and The kernel size of the first folded filter is determined based on Nw1, Nh1, the convolution kernel of the original filter, and the convolution stride.

6. The data processing apparatus of claim 5, wherein the processing circuitry is further configured to determine the first total folding factor N as follows: total1 : N total1 =Aco / Coa, where Coa is the nearest Aco / 2 aligned to Co. n The value of , where n is a natural number.

7. The data processing apparatus according to any one of claims 5-6, wherein the processing circuitry is further configured to split the first total folding multiple N according to any one of the following rules. total1 : Split along the width direction of the convolutional kernel; Split into directions with smaller convolution strides; The kernel is split evenly along both the width and height directions of the convolution kernel. The convolution kernel size of the first folded filter is minimized by splitting it into parts.

8. The data processing apparatus of claim 5, wherein the processing circuitry is further configured to determine the convolution stride of the first folded filter as follows: S′ x =S x *Nw1 S y 'S y *Nh1 in, S x S y S represents the horizontal and vertical convolution strides of the original filter, respectively. x ′ S y ′ represents the horizontal convolution stride and the vertical convolution stride of the first folded filter, respectively.

9. The data processing apparatus of claim 5, wherein the processing circuitry is further configured to determine the kernel size of the first folded filter as follows: k w ′=k w +(Nw1-1)*S x k h ′=k h +(Nh1-1)*S y in, S x S y These are the horizontal and vertical convolution strides of the original filter, respectively, k. w k h These represent the kernel width and kernel height of the original filter, respectively, k w ′、k h ′ represents the kernel width and kernel height of the first folded filter, respectively.

10. The data processing apparatus of claim 4, wherein the second threshold Aco is determined based on the number of convolution operation units in the processing circuit, and the first threshold A1 ≤ Aco / 2.

11. The data processing apparatus according to claim 1, wherein the input channel dimension of the original filter does not exceed a third threshold A2, and the input channel dimension of the folded filter is equal to a fourth threshold Aci, wherein the third threshold A2 is less than the fourth threshold Aci.

12. The data processing apparatus of claim 11, wherein the processing circuitry is configured to perform the second dimension folding in such a manner as follows: The second total folding factor N is determined based on the input channel dimension size Ci of the multidimensional data to be folded and the fourth threshold Aci. total2 ; The second total folding factor N total2 It is split into the second width dimension folding factor Nw2 and the second height dimension folding factor Nh2; Based on Nw2, Nh2, and the width and height dimensions of the multidimensional data to be folded, determine the width and height dimensions of the folded multidimensional data; and Based on Nw2, Nh2, and the original convolution stride of the convolution operation, the folded convolution stride of the convolution operation is determined.

13. The data processing apparatus of claim 12, wherein the processing circuitry is further configured to determine the second total folding factor N as follows: total2 ; N total2 =Aci / Cia; where Ci is the nearest Aci / 2 aligned to Ci. n The value of , where n is a natural number.

14. The data processing apparatus according to any one of claims 12-13, wherein the processing circuitry is further configured to split the second total folding multiple N according to any one of the following rules. total2 : Split into the width dimension; Split evenly across the width and height dimensions; Break it down into parts that minimize the amount of padding caused by aligning the folding multiples; The convolution stride is split into components such that the convolution stride in the width dimension based on the first folded filter is divisible by the folding factor in that width dimension.

15. The data processing apparatus of claim 12, wherein the processing circuitry is further configured to determine the width dimension and height dimension of the folded multidimensional data as follows: k w ″=k wa / Nw2 (7) k h ″=k ha / Nh2 (8) in, k w "、k h "" represents the width and height dimensions of the folded multidimensional data, respectively, k wa Let k be the width dimension size of the multidimensional data to be folded. w Align with the nearest width dimension collapse factor Nw2, k ha Let k be the height dimension of the multidimensional data to be folded. h Align with the value of the nearest height dimension folding factor Nh2.

16. The data processing apparatus of claim 12, wherein the processing circuitry is further configured to determine the convolution stride of the folded multidimensional data as follows: in, S x ′、S y ′ represents the convolution stride in the width dimension and the convolution stride in the height dimension of the convolution operation before folding in the second dimension, respectively. x "、S y "These are the convolution stride in the width dimension and the convolution stride in the height dimension, respectively, after the second dimension is folded and the convolution operation is performed.

17. The data processing apparatus according to claim 12, wherein the multidimensional data to be folded is a first folding filter and / or the input feature map.

18. The data processing apparatus of claim 11, wherein the fourth threshold Aci is determined based on instruction alignment requirements, and the third threshold A2 ≤ Aci / 2.

19. The data processing apparatus of claim 1, wherein the processing circuitry is further configured to: When the extended filter only includes the extended filter obtained by moving the kernel width, the intermediate result is dimensionally reorganized to achieve dimensional defolding, resulting in an output feature map.

20. The data processing apparatus of claim 1, wherein the processing circuitry is further configured to: When the extended filter includes an extended filter obtained by moving the convolution kernel in the height direction, the intermediate result is dimensionally transposed to achieve dimensional defolding, resulting in an output feature map.

21. The data processing apparatus of claim 1, wherein the processing circuitry is further configured to: When the extended filter includes both the extended filter obtained by moving the kernel width direction and the extended filter obtained by moving the kernel height direction, the intermediate result is dimensionally reorganized and transposed to achieve the dimensional defolding, resulting in an output feature map.

22. The data processing apparatus according to claim 20 or 21, wherein the processing circuitry is further configured to: The features of the intermediate result in the output channel dimension are transposed to the height dimension.

23. The data processing apparatus of claim 1, wherein the processing circuitry is further configured to: The second dimension folding is achieved through dimension restructuring; and / or The second dimension folding is achieved by transposing the dimension.

24. The data processing apparatus according to claim 1, wherein: The input channel dimension of the original filter is equal to the input channel dimension of the first folded filter; The size of the output channel dimension of the first folded filter is equal to the size of the output channel dimension of the folded filter.

25. The data processing apparatus of claim 1, wherein the folded filter is generated offline or online.

26. A chip, characterized in that, The chip includes the data processing device as described in any one of claims 1-25.

27. A circuit board, characterized in that, The board includes the chip as described in claim 26.

28. A method for executing a neural network model, implemented by a data processing apparatus, said data processing apparatus comprising storage circuitry and processing circuitry, wherein... The storage circuit stores folded filters of the convolutional layer of the neural network model. The folded filters are generated by folding the original filters in a first dimension and folding them in a second dimension. The first dimension folding includes synthesizing multiple extended filters after multiple convolutional stride shifts. The second dimension folding includes rearranging the data in the width dimension and / or height dimension to the input channel dimension. The method includes: The processing circuit performs second-dimensional folding on the input feature map to obtain a folded feature map; The processing circuit uses the folded filter stored in the storage circuit to perform a convolution operation on the folded feature map to obtain an intermediate result; as well as The processing circuit performs dimensional defolding on the intermediate results to obtain an output feature map.

29. The method according to claim 28, wherein the folded filter is generated by first folding the original filter in a first dimension to obtain a first folded filter, and then folding the first folded filter in a second dimension.

30. The method of claim 29, further comprising: The processing circuit performs the first-dimensional folding as follows: The original filter is shifted Nw1-1 times in the horizontal convolution stride direction along the kernel width direction, and / or Nh1-1 times in the vertical convolution stride direction along the kernel height direction to obtain N total1 N extended filters, where N total1 = Nw1 × Nh1, where Nw1 and Nh1 are natural numbers; as well as The N total1 The extended filters are combined into the first folded filter along the output channel dimension.

31. The method of claim 30, wherein the output channel dimension of the original filter does not exceed a first threshold A1, and the output channel dimension of the first folded filter is equal to a second threshold Aco, wherein the first threshold A1 is less than the second threshold Aco.

32. The method of claim 31, further comprising: The processing circuit determines the first total folding factor N based on the number of output channels Co of the original filter and the second threshold Aco. total1 ; The first total folding factor N total1 It is split into the first width dimension folding factor nw1 and the first height dimension folding factor Nh1; Based on Nw1, Nh1, and the convolution stride of the original filter, the convolution stride of the first folded filter is determined; and The kernel size of the first folded filter is determined based on Nw1, Nh1, the convolution kernel of the original filter, and the convolution stride.

33. The method of claim 32, further comprising: The processing circuit determines the first total folding factor N as follows: total1 : N total1 =Aco / Coa, where Coa is the nearest Aco / 2 aligned to Co. n The value of , where n is a natural number.

34. The method according to any one of claims 32-33, further comprising: The processing circuit splits the first total folding multiple N according to any of the following rules. total1 : Split along the width direction of the convolutional kernel; Split into directions with smaller convolution strides; The kernel is split evenly along both the width and height directions of the convolution kernel. The convolution kernel size of the first folded filter is minimized by splitting it into parts.

35. The method of claim 32, further comprising: The processing circuit determines the convolution stride of the first folded filter as follows: S′ x =S x *Nw1 S y 'S y *Nh1 Among them, S x S y These are the horizontal and vertical convolution strides of the original filter, S′, respectively. x S y ′ represents the horizontal convolution stride and the vertical convolution stride of the first folded filter, respectively.

36. The method of claim 32, further comprising: The processing circuit determines the convolution kernel size of the first folded filter as follows: k w ′=k w +(Nw1-1)*S x k h ′=k h +(Nh1-1)*S y Among them, S x S y These are the horizontal and vertical convolution strides of the original filter, respectively, k. w k h These represent the kernel width and kernel height of the original filter, respectively, k w ′、k h ′ represents the kernel width and kernel height of the first folded filter, respectively.

37. The method of claim 31, wherein the second threshold Aco is determined based on the number of convolution operation units in the processing circuit, and the first threshold A1 ≤ Aco / 2.

38. The method of claim 28, wherein the input channel dimension of the original filter does not exceed a third threshold A2, and the input channel dimension of the folded filter is equal to a fourth threshold Aci, wherein the third threshold A2 is less than the fourth threshold Aci.

39. The method of claim 38, further comprising: The processing circuit performs the second-dimensional folding as follows: The second total folding factor N is determined based on the input channel dimension size Ci of the multidimensional data to be folded and the fourth threshold Aci. total2 ; The second total folding factor N total2 It is split into the second width dimension folding factor Nw2 and the second height dimension folding factor Nh2; Based on Nw2, Nh2, and the width and height dimensions of the multidimensional data to be folded, determine the width and height dimensions of the folded multidimensional data; and Based on Nw2, Nh2, and the original convolution stride of the convolution operation, the folded convolution stride of the convolution operation is determined.

40. The method of claim 39, further comprising: The processing circuit determines the second total folding factor N as follows: total2 ; N total2 =Aci / Cia; where Ci is the nearest Aci / 2 aligned to Ci. n The value of , where n is a natural number.

41. The method according to any one of claims 39-40, further comprising: The processing circuit splits the second total folding multiple N according to any of the following rules. total2 : Split into the width dimension; Split evenly across the width and height dimensions; Break it down into parts that minimize the amount of padding caused by aligning the folding multiples; The convolution stride is split into components such that the convolution stride in the width dimension based on the first folded filter is divisible by the folding factor in that width dimension.

42. The method of claim 39, further comprising: The processing circuit determines the width and height dimensions of the folded multidimensional data as follows: k w ″=k wa / Nw2 (7) k h ″=k ha / Nh2 (8) Where, k w "、k h "" represents the width and height dimensions of the folded multidimensional data, respectively, k wa Let k be the width dimension size of the multidimensional data to be folded. w Align with the nearest width dimension collapse factor Nw2, k ha Let k be the height dimension of the multidimensional data to be folded. h Align with the value of the nearest height dimension folding factor Nh2.

43. The method of claim 39, further comprising: The processing circuit determines the convolution stride of the folded multidimensional data as follows: Among them, S x ′、S y ′ represents the convolution stride in the width dimension and the convolution stride in the height dimension of the convolution operation before folding in the second dimension, respectively. x "、S y "These are the convolution stride in the width dimension and the convolution stride in the height dimension, respectively, after the second dimension is folded and the convolution operation is performed.

44. The method of claim 39, wherein the multidimensional data to be folded is a first folding filter and / or the input feature map.

45. The method of claim 38, wherein the fourth threshold Aci is determined based on instruction alignment requirements, and the third threshold A2 ≤ Aci / 2.

46. ​​The method of claim 28, wherein dimensional defolding of the intermediate result comprises: When the extended filter only includes the extended filter obtained by moving the kernel width, the intermediate result is dimensionally reorganized to achieve dimensional defolding, resulting in an output feature map.

47. The method of claim 28, wherein dimensional defolding of the intermediate result comprises: When the extended filter includes an extended filter obtained by moving the convolution kernel in the height direction, the intermediate result is dimensionally transposed to achieve dimensional defolding, resulting in an output feature map.

48. The method of claim 28, wherein dimensional defolding of the intermediate result comprises: When the extended filter includes both the extended filter obtained by moving the kernel width direction and the extended filter obtained by moving the kernel height direction, the intermediate result is dimensionally reorganized and transposed to achieve the dimensional defolding, resulting in an output feature map.

49. The method of claim 47 or 48, wherein transposing the intermediate result comprises: The features of the intermediate result in the output channel dimension are transposed to the height dimension.

50. The method of claim 28, further comprising: The processing circuit achieves the second dimension folding in the width dimension through dimension recombination; and / or The second dimension folding is achieved by transposing the dimension.

51. The method according to claim 28, wherein: The input channel dimension of the original filter is equal to the input channel dimension of the first folded filter; The size of the output channel dimension of the first folded filter is equal to the size of the output channel dimension of the folded filter.

52. The method of claim 28, wherein the folded filter is generated offline or online.

Images