Data processing apparatus, method and related product for executing a neural network model

Abstract

The present disclosure discloses a data processing apparatus, method and related products for executing a neural network model. The data processing apparatus can be included in a combined processing apparatus as a computing apparatus, which can further include an interface apparatus and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The combined processing apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively for storing data of the computing apparatus and the other processing apparatuses. The scheme of the present disclosure optimizes the convolution operation on multi-dimensional arrays, improving the operation processing efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of data processing. More specifically, the present disclosure relates to a data processing apparatus, a data processing method, a chip and a board for executing a neural network model. BACKGROUND

[0002] At present, deep learning has become an important branch in machine learning, and also vigorously promotes the development of artificial intelligence (AI). The core technology of deep learning, deep neural network (DNN), has been widely applied in many industries.

[0003] The convolution layer is one of the commonly used hidden layers in the neural network model, which extracts features from the input data through convolution operation. The neural network model contains a large number of convolution operations, and the calculation performance of the convolution operation greatly affects the calculation performance of the entire neural network model. In the convolution operation, for each dimension of the filter of the convolution layer, there are requirements for instruction alignment and hardware (for example, parallel operator) alignment. Therefore, it is necessary to optimize the convolution operation to improve the calculation performance of executing the neural network model. SUMMARY

[0004] In order to at least solve one or more technical problems as mentioned above, the present disclosure proposes a data processing scheme for executing a neural network model in multiple aspects, which can effectively improve the calculation performance of the convolution operation by transforming the filter of the convolution layer. The neural network model of the embodiments of the present disclosure can be applied to various fields, such as image processing, speech processing, text processing, etc., which may, for example, include but are not limited to recognition and classification.

[0005] In a first aspect, the present disclosure provides a data processing apparatus for executing a neural network model, comprising: a storage circuit configured to store a folded filter of a convolution layer of the neural network model, the folded filter being composed of a plurality of extended filters obtained by moving an original filter by a plurality of convolution steps; and a processing circuit configured to: perform a convolution operation on an input feature map using the folded filter to obtain an intermediate result; and perform dimension rearrangement on the intermediate result to obtain an output feature map.

[0006] In a second aspect, the present disclosure provides a chip comprising the data processing apparatus of any of the preceding first aspect.

[0007] In a third aspect, the present disclosure provides a board comprising the chip of any of the preceding second aspect.

[0008] In a fourth aspect, the present disclosure provides a method for executing a neural network model implemented by a data processing apparatus, the data processing apparatus comprising a storage circuit and a processing circuit, the method comprising: performing, by the processing circuit, a convolution operation on an input feature map using folded filters of a convolution layer of the neural network model stored in the storage circuit to obtain an intermediate result, wherein the folded filters are a composition of a plurality of extended filters obtained by moving an original filter by a plurality of convolution step sizes; and performing, by the processing circuit, a dimension rearrangement on the intermediate result to obtain an output feature map.

[0009] By the data processing apparatus, chip, board card and data processing method implemented by the data processing apparatus as provided above, the scheme of the present disclosure optimizes the convolution operation by using the folded filters. The embodiments of the present disclosure are particularly suitable for the case where the output channel dimension of the original filter is small. In the conventional convolution operation, when the output channel dimension of the filter is small, due to the limitation of the number of parallel operation units alignment, it will cause a large waste of resources. The embodiments of the present disclosure synthesize a plurality of extended filters obtained by moving the original filter by a plurality of convolution step sizes into a folded filter, so that the available parallel operation units can be fully utilized, avoiding the waste of operation resources and improving the calculation performance of the convolution operation in hardware acceleration. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a structural diagram of a board card of an embodiment of the present disclosure;

[0012] Figure 2 is a structural diagram of an integrated circuit device of an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of the internal structure of a single-core computing device of an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of the internal structure of a multi-core computing device of an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of the internal structure of a processor core of an embodiment of the present disclosure;

[0016] Figure 6 shows an example convolution operation example to which the embodiments of the present disclosure can be applied;

[0017] Figure 7 An exemplary schematic diagram showing a data processing scheme according to embodiments of the present disclosure is shown;

[0018] Figure 8 An exemplary schematic diagram showing a more detailed filter folding according to embodiments of the present disclosure is shown;

[0019] Figure 9 An exemplary comparison diagram showing a folding before and after calculation process according to embodiments of the present disclosure is shown;

[0020] Figure 10 An exemplary schematic diagram showing a dimension transposition according to embodiments of the present disclosure is shown;

[0021] Figure 11 An exemplary schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure is shown; and

[0022] Figure 12 An exemplary flowchart of a data processing method according to embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative effort fall within the scope of the present disclosure.

[0024] It should be understood that the terms "first", "second", and "third" and the like in the claims, the specification, and the drawings of the present disclosure are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "contain" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0025] It should also be understood that the terms used in the specification of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used in the specification and claims of the present disclosure, the singular forms "a", "an", and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should be further understood that the term "and / or" used in the specification and claims of the present disclosure means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0029] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0030] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0031] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2As shown in FIG. 1, the combined processing device 20 comprises a computing device 201, an interface device 202, a processing device 203 and a DRAM 204.

[0032] The computing device 201 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor to perform deep learning or machine learning calculations, which can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0033] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it to the storage device on the computing device 201. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them to the control buffer on the computing device 201. Alternatively or optionally, the interface device 202 can also read data from the storage device of the computing device 201 and transmit it to the processing device 203.

[0034] The processing device 203 serves as a general-purpose processing device to perform basic controls including but not limited to data transfer, starting and / or stopping of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned before, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core or homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0035] The DRAM 204 is used to store data to be processed, which is a DDR memory, usually with a size of 16G or more, for saving data of the computing device 201 and / or the processing device 203.

[0036] Figure 3The diagram shows the internal structure of the single-core computing device 201. The single-core computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The single-core computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0037] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0038] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0039] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between single-core computing device 301 and DRAM 204.

[0040] Figure 4 A schematic diagram of the internal structure of the computing device 201 as a multi-core is shown. The multi-core computing device 41 adopts a hierarchical structure design. As a system-on-a-chip, the multi-core computing device 41 includes at least one cluster, and each cluster includes multiple processor cores. In other words, the multi-core computing device 41 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0041] From the perspective of system-on-a-chip hierarchy, such as Figure 4 As shown, the multi-core computing device 41 includes an external storage controller 401, a peripheral communication module 402, an on-chip interconnect module 403, a synchronization module 404, and multiple clusters 405.

[0042] There can be multiple external storage controllers 401; two are shown as an example in the figure. These controllers are used to access external storage devices, such as those issued by the processor core, in response to access requests from the processor core. Figure 2 The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 402 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 403 connects the external storage controller 401, the peripheral communication module 402, and multiple clusters 405 to transmit data and control signals between modules. The synchronization module 404 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 405 are the computing cores of the multi-core computing device 41. Four are shown exemplary in the figure; however, with hardware development, the multi-core computing device 41 of this disclosure can also include 8, 16, 64, or even more clusters 405. The clusters 405 are used to efficiently execute deep learning algorithms.

[0043] From the perspective of cluster hierarchy, such as Figure 4 As shown, each cluster 405 includes multiple processor cores (IPU cores) 406 and one memory core (MEM core) 407.

[0044] Four processor cores 406 are shown in the figure as an example, but this disclosure does not limit the number of processor cores 406. Its internal architecture is as follows: Figure 5 As shown. Each processor core 406 is similar to Figure 3 The single-core computing device 301 also includes three main modules: a control module 51, an arithmetic module 52, and a storage module 53. The functions and structures of the control module 51, arithmetic module 52, and storage module 53 are largely the same as those of the control module 31, arithmetic module 32, and storage module 33, and will not be described again. It should be noted that the storage module 53 includes an input / output direct memory access (IODMA) module 533 and a move direct memory access (MVDMA) module 534. The IODMA 533 controls the memory access of NRAM 531 / WRAM 532 and DRAM 204 via the broadcast bus 409; the MVDMA 534 controls the memory access of NRAM 531 / WRAM 532 and SRAM 408.

[0045] Back Figure 4The storage core 407 is mainly used for storing and communication, i.e. storing shared data or intermediate results among the processor cores 406, performing communication between the execution cluster 405 and the DRAM 204, communication among the clusters 405, communication among the processor cores 406, etc. In other embodiments, the storage core 407 has the capability of scalar operation to perform scalar operation.

[0046] The storage core 407 includes an SRAM 408, a broadcast bus 409, a cluster direct memory access (CDMA) 410 and a global direct memory access (GDMA) 411. The SRAM 408 plays the role of a high-performance data relay station. Data reused among different processor cores 406 within the same cluster 405 does not need to be obtained by the processor cores 406 from the DRAM 204 respectively, but is relayed among the processor cores 406 through the SRAM 408. The storage core 407 only needs to quickly distribute the reused data from the SRAM 408 to the multiple processor cores 406, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0047] The broadcast bus 409, the CDMA 410 and the GDMA 411 are respectively used to perform communication among the processor cores 406, communication among the clusters 405 and data transmission between the cluster 405 and the DRAM 204. The following will be described respectively.

[0048] The broadcast bus 409 is used to complete high-speed communication among the processor cores 406 within the cluster 405. The broadcast bus 409 of the embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (e.g. single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a piece of data from the SRAM 408 to specific processor cores 406, and broadcast is a communication mode of transmitting a piece of data from the SRAM 408 to all processor cores 406, which is a special case of multicast.

[0049] The CDMA 410 is used to control access to the SRAM 408 among different clusters 405 within the same computing device 201.

[0050] The GDMA 411 cooperates with the external memory controller 401 to control the access of the SRAM 408 to the DRAM 204 or the reading of data from the DRAM 204 to the SRAM 408 of the cluster 405. As mentioned above, the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 can be achieved through two channels. The first channel is to directly contact the DRAM 204 and the NRAM 431 or the WRAM 432 through the IODMA 433; the second channel is to first transfer data between the DRAM 204 and the SRAM 408 through the GDMA 411, and then transfer data between the SRAM 408 and the NRAM 431 or the WRAM 432 through the MVDMA 534. Although the second channel seems to need more elements to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much larger than that of the first channel, so that the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 through the second channel can be more efficient. The embodiments of the present disclosure can select the data transmission channel according to the hardware conditions.

[0051] In other embodiments, the functions of the GDMA 411 and the functions of the IODMA 533 can be integrated in the same component. For the convenience of description, the GDMA 411 and the IODMA 533 are regarded as different components in the present disclosure, and for those skilled in the art, as long as the functions achieved and the technical effects achieved are similar to the present disclosure, they belong to the protection scope of the present disclosure. Further, the functions of the GDMA 411, the functions of the IODMA 533, the functions of the CDMA 410, and the functions of the MVDMA 534 can also be achieved by the same component.

[0052] The neural network model usually includes an input layer, a convolution layer, an activation function, a pooling layer, a fully connected layer, etc., and there are few layers, or hundreds of layers, each layer executes an operator, for example, the convolution layer executes the convolution operator, and the number of layers needs to execute the number of operators.

[0053] The training of the neural network model is to adjust the parameters of each layer by inputting the training samples, so that the result calculated by the neural network model is as close to the true result as possible. The neural network model training includes forward propagation and backward propagation. The forward propagation is based on the existing model, and the input training samples are calculated through each layer of the neural network model to gradually extract the input feature map as abstract features. The backward propagation is to calculate the loss function according to the forward propagation result and the true value, and to update the parameters by using the gradient descent method and calculating the partial derivative of the loss function to each parameter through the chain rule. The updated parameters are used for training, and the process is repeated for multiple times, so that the calculation result of the forward propagation meets the expectation. The inference of the neural network model is to use the trained neural network model to perform forward operation on the input of the real environment to complete the set task.

[0054] Based on the foregoing hardware environment, the disclosure embodiments provide a data processing scheme for executing a neural network model, and more specifically, a scheme for optimizing convolution operation in the neural network model.

[0055] Figure 6 An exemplary convolution operation example to which the disclosure embodiments can be applied is shown. As shown in the figure, the convolution layer in the neural network model can perform feature extraction by applying a filter to the input feature map for convolution processing.

[0056] An input feature map with a size of 6x6x3 is exemplarily shown in the figure, which can represent 3 6x6 size feature maps (i.e., 3 three-dimensional matrices of 6x6x3), representing three different features respectively. The width W of the feature map in this example is 6, and the height H is also 6. The number of input feature maps can also be referred to as the number of input channels Ci. For example, the example input in the figure has 3 feature maps, also referred to as 3 feature channels.

[0057] A filter with a size of 2x3x3x3 is also exemplarily shown in the figure, which can represent 2 3x3x3 size convolution kernels (i.e., 2 three-dimensional matrices of 3x3x3), each having 3 different 3x3 size convolution kernels, corresponding to the 3 different feature maps of the input. The number of three-dimensional convolution kernels can be referred to as the number of output channels Co, which is 2 in this example. In each three-dimensional convolution kernel, the number of two-dimensional convolution kernels can be referred to as the number of input channels Ci, which is consistent with the number of channels of the input feature map. Each two-dimensional convolution kernel has a corresponding width Kw and height Kh, both of which are 3 in this example.

[0058] The convolution result of the input feature map and the filter outputs two 4x4 size feature maps. Among them, the convolution result of the input feature map and the upper stereo convolution kernel obtains one 4x4 output feature map above, and the convolution result of the input feature map and the lower stereo convolution kernel obtains one 4x4 output feature map below. The value at each position in the output feature map is obtained by adding the two-dimensional convolution operation of the corresponding block of each input feature map and the corresponding convolution kernel. For example, the value at the (0, 0) position of the output feature map above is obtained by performing two-dimensional convolution operation on the block of the input feature map framed by the black cube and the upper stereo convolution kernel to obtain three values, and then adding them to obtain the final value. In order to obtain the output of other positions, the position of the convolution kernel can be moved on the input feature map. In the example in the figure, the convolution step (Sx, Sy) is (1, 1), and when the convolution operation is performed after moving one grid to the right in the horizontal (width) direction or moving one grid downward in the vertical (height) direction, the values at the (0, 1) or (1, 0) positions of the output feature map above can be obtained, respectively.

[0059] From the above description, in a convolution layer of a neural network, there is a group of input feature maps, which contains HxWxCi information in total, where H and W are the height and width of the input feature map, respectively, and Ci is the number of input channels. The convolution layer has CixCo convolution kernels with a size of KhxKw, where Ci is the number of input channels, Co is the number of output feature maps (or the number of output channels), and Kh and Kw are the height and width of the convolution kernel, respectively. The output feature map contains HoxWoxCo information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. In addition, in the convolution operation, the convolution step (Sx, Sy) will also be involved, and the size of the convolution step will affect the size of the output feature map.

[0060] In order to accelerate the calculation of the neural network model, multiple operation units are usually used for parallel operation. For example, the operation module 32 in Figure 3 or the operation module 52 in Figure 5 may include multiple convolution-specific computing units (or convolution units), and each convolution unit can perform complete (H, W, Ci) dimension calculation, for example. In other words, Co (H, W, Ci) dimension calculations can be distributed on Co convolution units for parallel calculation, thereby improving the calculation speed. Generally, the number of convolution units is fixed, and if the size of the Co dimension is small, there will be idle convolution units, which cannot fully utilize the computing resources. In some cases, it may be required to align the size of the Co dimension to the number of convolution units in order to uniformly schedule. However, when the Co dimension is small, this alignment restriction will introduce invalid calculations, causing a large waste of resources.

[0061] In view of this, the embodiments of the present disclosure provide a data processing scheme for executing a neural network model, which is optimized for the Co dimension in a convolution layer, while meeting the alignment restriction and reducing resource waste.

[0062] In the embodiments of the present disclosure, the dimensions of the multi-dimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in the memory. It can be understood that, although the multi-dimensional data has multiple dimensions, there is a correspondence between the multi-dimensional data and the storage order on the memory because the layout of the memory is always one-dimensional. The multi-dimensional data is usually allocated in a continuous storage space, that is, the multi-dimensional data can be one-dimensionally unfolded and stored in order on the memory. For example, in the embodiments of the present disclosure, the storage is performed in a low-dimension-first manner (Ci is the lowest dimension here). Adjacent dimensions refer to dimensions that are next to each other in the dimension information of the multi-dimensional data, for example, W and Ci are adjacent, and the adjacent dimensions can also be referred to as continuous dimensions.

[0063] Figure 7 An exemplary schematic diagram of the data processing scheme of the embodiments of the present disclosure is shown through a specific example. It is assumed that the number of convolution units used to perform convolution operation is Th2. Based on different hardware designs, Th2 can have different values, such as 32, 64, 128, etc. In the following example, Th2 = 64 is taken as an example for description. According to the alignment requirement of the hardware, the Co dimension of the filter needs to be aligned to Th2, that is, aligned to 64.

[0064] The left side of the figure shows the original filter of the convolution layer, which is represented as 4x3x3x64, for example, that is, the output channel number Co is 4, the input channel number Ci is 64, and the size of each convolution kernel is 3x3. In addition, the convolution step of the original filter is (Sx, Sy) = (1, 1). As can be seen from the figure, the Co dimension of the original filter is far less than the alignment requirement of the hardware (64). According to the conventional processing manner, zero padding will be performed on the Co dimension to align to 64. From 4 to 64, a large number of redundant calculations need to be added, causing resource waste.

[0065] The right side of the figure shows the folded filter after folding according to the embodiments of the present disclosure, which is represented as 64x6x6x64, for example, that is, the output channel number Co' is 64, the input channel number is the same as that of the original filter, both of which are 64, and the size of each convolution kernel is 6x6. In addition, the convolution step of the folded filter becomes (Sx', Sy') = (4, 4). As can be seen, the output channel number of the folded filter has been aligned to the alignment requirement of the hardware (64), and therefore, the convolution units can be fully utilized for operation processing at this time.

[0066] In the above folding process, the folded filter is synthesized from multiple expanded filters obtained by shifting the original filter through multiple convolution strides. This folding process is based on the following consideration: if the original calculation method is followed, a single calculation will waste some convolution units. Therefore, the next calculation, the calculation after that, etc., can be moved to a single calculation, thereby making full use of the convolution units. In other words, the solution of this disclosed embodiment is equivalent to folding the calculations corresponding to multiple convolution strides (or multiple time-based calculations) into a single convolution stride (or a single time period).

[0067] In two-dimensional convolution, the convolution stride includes a horizontal stride Sx in the kernel width direction and a vertical stride Sy in the kernel height direction. Moving the convolution stride once in either direction yields one computation, which corresponds to a filter. This filter is obtained by moving the original filter by one convolution stride. In order to be synthesized with other similarly generated filters later, the width and height of its convolution kernel need to be expanded accordingly based on the convolution stride; this is called an expanded filter.

[0068] Suppose we shift the original filter by Nw-1 horizontal convolution strides in the kernel width direction and Nh-1 vertical convolution strides in the kernel height direction, we can obtain N extended filters, where N = Nw × Nh, and Nw and Nh are natural numbers. These N extended filters can be combined into a folded filter along the Co dimension of the output channel.

[0069] Still with Figure 7 For example, the Co of the original filter is 4. According to the hardware alignment requirements, it needs to be aligned to 64. Therefore, 64÷4=16 extended filters are needed to synthesize a folded filter.

[0070] Figure 8 A more detailed schematic diagram of filter folding according to an embodiment of this disclosure is shown. The figure illustrates the 16 extended filters required in the aforementioned example. The darker areas in the figure represent the convolution kernels of the original filters. As can be seen from the figure, these 16 extended filters are obtained by shifting the width direction by 3 times (Sx) and the height direction by 3 times (Sy). That is, the width is quadrupled, and the height is also quadrupled, thus resulting in a total of 16 extended filters.

[0071] To combine these 16 extended filters into a single folded filter along the Co dimension of the output channel, their shapes need to be adjusted. As shown in the figure, zero-padding is applied to the area excluding the dark portion occupied by the original filters, ensuring that the width and height of the 16 extended filters are consistent. The convolution value of each extended filter (6×6 convolution kernel) is the same as the convolution value of the original filter (3×3 convolution kernel).

[0072] Then, the 16 (4, 6, 6, 64) extended filters can be folded into a 16*4, 6, 6, 64) folded filter in the output channel Co dimension. At this time, the output channel Co' of the folded filter is 16*4=64, which is equal to the number of convolution units, thereby meeting the hardware alignment requirement and achieving the purpose of fully utilizing the convolution units.

[0073] From the folding process of the 16 (4, 6, 6, 64) extended filter into a 16*4, 6, 6, 64) folded filter, it can be seen that the number of output channels of the folded filter is multiplied relative to the number of output channels of the original filter, and therefore the folding scheme of the embodiments of the present disclosure is particularly suitable for the case where the number of output channels Co of the original filter is small, for example, Co does not exceed a first threshold Th1, and the first threshold Th1 is less than a second threshold Th2 (i.e., the alignment requirement). In some embodiments, the first threshold Th1≤Th2 / 2. The practice shows that the smaller Co is, the greater the improvement space is compared with the existing algorithm.

[0074] From the folding process of the 16 (4, 6, 6, 64) extended filter into a 16*4, 6, 6, 64) folded filter, it can be seen that the number of output channels of the folded filter is multiplied relative to the number of output channels of the original filter, and therefore the folding scheme of the embodiments of the present disclosure is particularly suitable for the case where the number of output channels Co of the original filter is small, for example, Co does not exceed a first threshold Th1, and the first threshold Th1 is less than a second threshold Th2 (i.e., the alignment requirement). In some embodiments, the first threshold Th1≤Th2 / 2. The practice shows that the smaller Co is, the greater the improvement space is compared with the existing algorithm. Figure 8 From the folding process of the 16 (4, 6, 6, 64) extended filter into a 16*4, 6, 6, 64) folded filter, it can be seen that the number of output channels of the folded filter is multiplied relative to the number of output channels of the original filter, and therefore the folding scheme of the embodiments of the present disclosure is particularly suitable for the case where the number of output channels Co of the original filter is small, for example, Co does not exceed a first threshold Th1, and the first threshold Th1 is less than a second threshold Th2 (i.e., the alignment requirement). In some embodiments, the first threshold Th1≤Th2 / 2. The practice shows that the smaller Co is, the greater the improvement space is compared with the existing algorithm.

[0075] Firstly, the total folding multiple N can be determined based on the number of output channels Co of the original filter and the alignment requirement Th2.

[0076] In some embodiments, the total folding multiple N can be determined as follows:

[0077] N=Th2 / Ca (1)

[0078] Where Ca is the value closest to Th2 / 2 n of Co, and n is a natural number.

[0079] As mentioned above, the embodiments of the present disclosure fold Co by a multiple to align it to a specified value Th2. When Th2 is 2 n times of Co, the total folding multiple can be obtained by directly dividing them. For example, in the foregoing example, Co=4, so N=64 / 4=16 times. When Th2 is not 2 n times of Co, Co needs to be aligned to the value closest to Th2 / 2 n . For example, if Th2 is 64, Th2 / 2 n includes 32, 16, 8, 4 and 2, and Co needs to be aligned to the closest value among them. For example, Co=3, which needs to be aligned to 4; if Co=5, which needs to be aligned to 8, and so on. After alignment, Th2 is removed to obtain the total folding multiple N.

[0080] ​

[0080] After the total folding factor N is determined, it can be split into a folding factor in the width direction of the convolution kernel Nw and a folding factor in the height direction of the convolution kernel Nh. The splitting of the factor can be performed according to various rules to achieve different advantages.

[0081] In one embodiment, the folding factor can be preferentially split in the width direction of the convolution kernel. As will be described later, the convolution operation with the folded filter results in an intermediate result, which needs to be reshaped in dimension to obtain the final result. When the folding occurs in the width direction of the convolution kernel, the reshaping in dimension is simple, which only needs to adjust the dimension representation of the intermediate result, or called dimension reorganization, without any other processing. Therefore, preferentially splitting the folding factor in the width direction of the convolution kernel can simplify the following processing. For example, in the above example, the total folding factor 16 can be split into 16 times in width and 1 time in height, which can simplify the following result processing. In this case, the convolution kernel size of the folded filter is 3x18. In another implementation, the total folding factor 16 in the above example is split into 8 times in width and 2 times in height, in which case the convolution kernel size of the folded filter is 4x10. It can be seen that different splitting manners will affect the convolution kernel size of the folded filter.

[0082] In another embodiment, the folding factor can be preferentially split in the direction with smaller convolution stride. From the above folding process, it can be seen that the convolution kernel size of the folded filter is related to the convolution stride of the original filter, and the larger the convolution stride, the larger the convolution kernel of the folded filter. Therefore, preferentially splitting the folding factor in the direction with smaller convolution stride is beneficial to reduce the expansion of the convolution kernel, thereby reducing the operation amount. For example, assuming that the convolution kernel of the original filter is 3x3, (Sx, Sy) = (1, 3), and the total folding factor is 16, which can be split into 16 times in width and 1 time in height, then the convolution kernel size of the folded filter is 3x18. If the total folding factor 16 is split into 8 times in width and 2 times in height, then the convolution kernel size of the folded filter is 6x10. If the total folding factor 16 is split into 4 times in width and 4 times in height, then the convolution kernel size of the folded filter is 12x6.

[0083] In yet another embodiment, the folding factor can be equally split in the width direction of the convolution kernel and the height direction of the convolution kernel. For example, in the above examples referring to Figure 7 and Figure 8 , the total folding factor 16 is equally split into 4 times in width and 4 times in height.

[0084] In still another embodiment, the folding factor can be split in such a way that the convolution kernel size of the folded filter is minimized, thereby minimizing the increased redundant operations.

[0085] Based on the above various rules, in some embodiments, the total folding factor N can be split in the following way:

[0086]

[0087] Nh= N / N w (3)

[0088] After determining the folding factor Nw in the width direction of the convolution kernel and the folding factor Nh in the height direction, the convolution step of the folded filter can be determined based on Nw, Nh and the convolution step of the original filter, and the size of the convolution kernel of the folded filter can be determined based on Nw, Nh and the convolution kernel and the convolution step of the original filter.

[0089] In some embodiments, the convolution step of the folded filter can be determined as follows:

[0090] S′ x = S x *Nw (4)

[0091] S y ′= S y *Nh (5)

[0092] where S x , S y are the horizontal and vertical convolution steps of the original filter, and S′ x , S y ′ are the horizontal and vertical convolution steps of the folded filter.

[0093] In some embodiments, the size of the convolution kernel of the folded filter can be determined as follows:

[0094] k w ′= k w +(Nw-1)*S x (6)

[0095] k h ′= k h +(Nh-1)*S y (7)

[0096] where k w , k h are the width and height of the convolution kernel of the original filter, and k′ w , k h ′ are the width and height of the convolution kernel of the folded filter.

[0097] Thus, it is described above how to design the various parameters of the folded filter according to the alignment requirements.

[0098] Thus, it is described above how to design the various parameters of the folded filter according to the alignment requirements.

[0098] Thus, it is described above how to design the various parameters of the folded filter according to the alignment requirements.

[0098] Figure 9 An exemplary comparison diagram of the folding-before and folding-after computation processes is shown.

[0099] As can be seen from the foregoing folding process, the input channel dimension of the folded filter is consistent with that of the original filter, and thus, no processing is needed for the input feature map to directly perform convolution operation with the folded filter.

[0100] As shown in the figure, it is assumed that the input feature map is (1, 16, 16, 64), i.e., 64 feature maps of 16x16. The upper part of the figure shows the result obtained after performing convolution operation with the original filter (4, 3, 3, 64) and convolution step (Sx, Sy) = (1, 1), and the output feature map is (1, 14, 14, 4), i.e., 4 feature maps of 14x14, which is represented by the dark square area in the figure. Those skilled in the art can understand that, for the purpose of clear illustration, no data in the Ci dimension is drawn, but only numerical values are shown.

[0101] In comparison, the lower part of the figure shows the result obtained after performing convolution operation with the folded filter (64, 6, 6, 64) and convolution step (Sx', Sy') = (4, 4), and the output feature map is (1, 4, 4, 64), i.e., 64 feature maps of 4x4.

[0102] As can be seen from the comparison in the figure, although the value of the convolution calculation performed by each extended filter (6x6 convolution kernel) is the same as that of the original filter (3x3 convolution kernel), the position of the data value will change, or in other words, the data value is rearranged in the dimension, because the operation of multiple convolution steps is performed at once. The intermediate result obtained by performing convolution operation on the input feature map by the folded filter can be subjected to dimension rearrangement again to obtain the final output feature map, so as to be completely consistent with the operation result of the original filter. The dimension of the intermediate result can be expressed as (N, ceil(Ho / Nh), ceil(Wo / Nw), Nh*Nw*Ca), where ceil() represents the ceiling function, and Ca is the Co alignment to the nearest Th2 / 2 nFor example, for the example in the figure, Ca= Co= 4, Nh= Nw= 4, ceil(Ho / Nh)= ceil(14 / 4)= 4, ceil(Wo / Nw)= ceil(14 / 4)= 4. Due to the kernel expansion and the increased convolution stride caused by the filter folding, and depending on the size of the input feature map, there can be some redundant computation in the folded convolution operation. When the intermediate result is reshaped, the redundant computation will be at the outermost, i.e., the rightmost and the lowermost, of the actually desired result. For example, for the example in the figure, the intermediate result (1, 4, 4, 64) is reshaped to (1, 16, 16, 4), which has some redundant area compared to the original result (1, 14, 14, 4), as indicated by the white square area in the figure. At this time, only the redundant area needs to be removed.

[0103] From the foregoing description, it can be known that the folding can be distributed in the width direction and / or the height direction of the convolution kernel. The folding in different directions will bring different reshaping results, and thus different processing methods are needed. In the following description, for the sake of simplicity, the above redundant area is not considered in the description. However, those skilled in the art can understand from the description herein that, for the case where the redundant area exists, the reshaping can also be performed according to the following description, and then the redundant removal can be performed.

[0104] In one embodiment, when only the folding in the width W direction of the convolution kernel is performed when folding, i.e., the expanded filter only contains the expanded filter obtained by moving the original filter in the width W direction of the convolution kernel, the intermediate result can be directly reshaped to obtain the output feature map.

[0105] It is mentioned above that in the embodiments of the disclosure, the dimensions of the multi-dimensional data involved are represented as (Co, H, W, Ci), i.e., the storage order of the data in the memory is Co, H, W, Ci, i.e., the Ci dimension is the lowest dimension. Those skilled in the art can also understand that the splitting or merging of adjacent dimensions in the multi-dimensional data does not affect the positions of the data elements of the multi-dimensional data in the memory.

[0106] In the embodiments of the disclosure, based on the characteristics of the convolution operation, the W and Ci dimensions are adjacent or consecutive in dimensions, and thus, when only the folding in the W direction is performed, the intermediate result (N, Ho, Wo / Nw, Nw*Co) can be directly regarded as (N, Ho, Wo, Co) at this time. That is, the W and Ci dimensions can be reshaped from (Wo / Nw, Nw*Co) to (Wo, Co). This reshaping does not affect the storage positions of the data elements of the intermediate result in the memory.

[0107] In another embodiment, when folding is performed only in the height H direction of the convolution kernel when folding, i.e., the extended filter only contains the extended filter obtained by moving the original filter in the height H direction of the convolution kernel, a dimension transposition needs to be performed on the intermediate result to obtain the output feature map. The dimension transposition changes the positions of data elements of multi-dimensional data in the memory.

[0108] Specifically, the features in the Co dimension of the intermediate result (N, Ho / Nh, Wo, Nh*Co) from (Co, (Nh-1)*Co] can be transposed back to the H dimension, thereby obtaining (N, Ho, Wo, Co).

[0109] Figure 10 An exemplary schematic diagram of the dimension transposition according to an embodiment of the present disclosure is shown.

[0110] As shown in Figure 10 , it is assumed that Nh=4, i.e., the intermediate result on the left includes 4 times the features in the H direction in the Co dimension, which are Co0, Co1, Co2, and Co3, respectively. At this time, the Co0 feature can be retained in the Co dimension, and the remaining Co1, Co2, and Co3 features can be transposed to the H dimension by the dimension transposition, as shown in the final result on the right.

[0111] In yet another embodiment, when folding is performed in both the width W direction and the height H direction of the convolution kernel when folding, i.e., the extended filter contains both the extended filter obtained by moving the original filter in the width direction of the convolution kernel and the extended filter obtained by moving the original filter in the height direction of the convolution kernel, the intermediate result can be processed according to the two preceding embodiments.

[0112] In one implementation, the dimension transposition can be performed on the intermediate result first, and then the dimension reorganization can be performed to obtain the output feature map. Specifically, the intermediate result (N, Ho / Nh, Wo / Nw, Nh*Nw*Co) can be first regarded as (N, Ho / Nh, Wo, Nh*Co), i.e., the features in the Co dimension folded from the W direction are regarded as the features in the W direction. Then, the dimension transposition operation is performed again to transpose the features in the Co dimension from (Co, (Nh-1)*Co] to the height H dimension, thereby obtaining (N, Ho, Wo, Co).

[0113] In another implementation, the dimension reorganization can be performed on the intermediate result first, and then the dimension transposition can be performed to obtain the output feature map.

[0114] Specifically, the dimension transposition operation can be first performed on the intermediate result (N, Ho / Nh, Wo / Nw, Nh*Nw*Co). For example, still referring to Figure 10, Co1, Co2 and Co3 in the Co dimension are first transferred to the H dimension by dimension transposition, but here Co0=Co1=Co2=Co3=Nw*Co. After dimension transposition, the intermediate result becomes (N, Ho, Wo / Nw, Nw*Co). Then, each Co0, Co1, Co2 and Co3 is regarded as Nw Co, thereby obtaining (N, Ho, Wo, Co).

[0115] The scheme of generating the folded filter according to the embodiments of the disclosure is described above. In some embodiments, the folded filter can be generated offline. For example, in the process of performing inference by using the neural network model, the offline generated folded filter arranged in advance can be used to perform convolution operation with the input feature map to perform the inference process. In some other embodiments, the folded filter can be generated online. For example, in the process of training the neural network model, the filter of the convolution layer can be folded online, and then convolution operation is performed with the training data to perform the training process.

[0116] No matter in which process the folded filter of the embodiments of the disclosure is used, the alignment of the Co dimension by folding can greatly optimize the calculation amount of the convolution operation. The performance of the scheme of the embodiments of the disclosure and the existing convolution operation are compared in terms of the convolution calculation amount as follows.

[0117] Let P represent the convolution calculation amount, A ci represents the value after Ci alignment, A CO represents the value after Co alignment,

[0118] then P=n*A ci *A CO *H o *W o *k w *k h (8)

[0119] The hardware using NHWC dimension arrangement order needs to be aligned to A ci because the Ci dimension is in the lowest dimension and the vector instruction alignment requirement, A ci is for the requirement of vector instruction alignment; the artificial intelligence computing acceleration hardware usually has multiple parallel high-performance convolution calculation units, so A CO is for the requirement of convolution kernel Co dimension alignment, and its value is the number of high-performance parallel calculation units.

[0120] Before optimization, the calculation amount of the existing convolution operation is:

[0121] P before =n*A ci *A CO *H o *Wo *k w *k h (9)

[0122] After optimization of the scheme of the embodiments of the present disclosure, the calculation amount of the convolution operation is:

[0123]

[0124] The optimization rate of performance after Co folding is:

[0125]

[0126] Taking the example described in the foregoing reference Figure 7- Figure 8 ,

[0127] Before optimization, P before = 1 * 64 * 64 * ho * wo * 3 * 3 (Sx, Sy = 1, 1)

[0128] After optimization, P after = 1 * 64 * 64 * (ho / 4) * (wo / 4) * 6 * 6 (Sx, Sy = 4, 4)

[0129] = 1 * 64 * 64 * ho * wo * 3 * 3 * (1 / 4)

[0130] The calculation amount of the convolution calculation unit is reduced by 75%.

[0131] From the comparison of the calculation amounts above, it can be seen that the folding filter scheme provided by the embodiments of the present disclosure can effectively save the calculation amount of the convolution unit, thereby improving the calculation performance of the convolution operation.

[0132] The embodiments of the present disclosure also provide a data processing apparatus for executing a neural network model, and a method for executing a neural network model implemented by the data processing apparatus.

[0133] Figure 11 An exemplary schematic structural diagram of a data processing apparatus that can implement the embodiments of the present disclosure is shown. As Figure 11 shown, the data processing apparatus 1100 includes processing circuitry 1110 and storage circuitry 1120.

[0134] The processing circuitry 1110 is responsible for processing various functions on the data processing apparatus 1100, including but not limited to control, decoding, operation, etc. The processing circuitry 1110 may, for example, include the control module 31 and / or the operation module 32 in the foregoing. Figure 3

[0135] ​In some embodiments, the processing circuit 1110 can be configured to perform a convolution operation on an input feature map using the folded filter of the convolutional layer of the neural network model stored in the storage circuit 1120 to obtain an intermediate result.

[0136] The storage circuit 1120 can be configured to store or carry relevant data, which can be, for example, various RAMs or on-chip caches as shown. Figure 3 Or Figure 5 In some embodiments, the storage circuit 1120 can be configured to store the folded filter of the convolutional layer of the neural network model. The folded filter is a combination of multiple extended filters obtained by moving the original filter by multiple convolution step sizes.

[0137] In some embodiments, the data processing apparatus 1100 can be configured to perform a training process of the neural network model. At this time, the processing circuit 1110 can be configured to perform the folding process of the filter of the convolutional layer of the neural network model on-line during the training. Then, the obtained folded filter is used to perform a convolution operation on the training data to perform the training process. The specific folding process performed by the processing circuit 1110 can refer to the description above, which will not be repeated here.

[0138] In other embodiments, the data processing apparatus 1100 can be configured to perform an inference process of the neural network model. At this time, the processing circuit 1110 can be configured to directly use the folded filter already stored in the storage circuit 1120 to perform a convolution operation on the input neuron to perform the inference process.

[0139] The intermediate result obtained by performing the convolution operation using the folded filter needs to be processed by a certain dimension rearrangement process to obtain the final output feature map. The specific rearrangement process operation can refer to the description above, which will not be repeated here.

[0140] Figure 12 An exemplary flowchart of a data processing method according to embodiments of the present disclosure is shown.

[0141] As shown, the data processing method 1200 includes a step 1210 of performing a convolution operation on an input feature map using the folded filter of the convolutional layer of the neural network model stored in the storage circuit by the processing circuit to obtain an intermediate result. The folded filter is a combination of multiple extended filters obtained by moving the original filter by multiple convolution step sizes.

[0142] Then, in step 1220, the processing circuit rearranges the dimensions of the intermediate result to obtain an output feature map.

[0143] Those skilled in the art can understand that the filter folding method, the dimension rearrangement of intermediate results and the like of the embodiments of the disclosure described above in combination with the drawings can also be applied to the data processing apparatus of Figure 11 and the data processing method of Figure 12 , and thus repeated description is not performed.

[0144] The disclosure also provides a chip, which can include the data processing apparatus of any of the embodiments described above in combination with the drawings. Further, the disclosure also provides a board card, which can include the aforementioned chip.

[0145] According to different application scenarios, the electronic device or apparatus of the disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle record instrument, a navigation instrument, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an oil smoke exhaust machine; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument and / or an electrocardiograph. The electronic device or apparatus of the disclosure can also be applied to the Internet, the Internet of Things, a data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment and the like. Further, the electronic device or apparatus of the disclosure can also be used in cloud, edge, terminal and the like application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the disclosure scheme can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that according to the hardware information of the terminal device and / or the edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling and cooperative work of end-cloud integration or cloud-edge integration.

[0146] It should be noted that, for the purpose of clarity, the disclosure describes some methods and embodiments thereof as a series of acts and / or combinations thereof, but those skilled in the art will understand that the present disclosure is not limited to the order of the acts described. Those skilled in the art will understand and appreciate that some steps of the methods can be decided to be executed in other orders or at the same time with the disclosure or teachings herein. Further, those skilled in the art will understand and appreciate that some of the embodiments described in the disclosure can be considered optional, i.e., the acts or modules involved therein are not necessarily essential for the implementation of one or more of the aspects of the present disclosure. In addition, the disclosure describes some embodiments with different focuses according to different aspects. Therefore, those skilled in the art will understand and appreciate that some parts not described in detail in one embodiment of the disclosure can be seen from the relevant description of other embodiments.

[0147] In specific implementation aspects, based on the disclosure and teachings herein, those skilled in the art will understand that some of the embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, as to each unit in the electronic device or apparatus embodiments described above, the disclosure divides them based on logical functions, and actual implementation can also have other division manners. For another example, a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As to the connection relationship between different units or components, the connection discussed above in conjunction with the drawings can be direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, where the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0148] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same place or distributed on multiple network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the aspects described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure can be integrated into one unit or physically exist separately.

[0149] In some implementation scenarios, the above-mentioned integrated units can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer-readable memory. Based on this, when the schemes of the present disclosure are embodied in the form of software products (for example, computer-readable storage media), the software products can be stored in the memory, which can include a number of instructions to make a computer device (for example, a personal computer, a server or a network device, etc.) execute part or all of the steps of the method described in the embodiments of the present disclosure. The aforementioned memory can include, but is not limited to, a variety of media capable of storing program codes, such as a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, etc.

[0150] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, that is, specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include, but is not limited to, physical devices, and the physical devices can include, but are not limited to, transistors or memristors, etc. In view of this, various devices described herein (for example, computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs and ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0151] While several embodiments of the disclosure have been shown and described herein, it is to be understood that the embodiments are merely exemplary. Numerous changes, substitutions and equivalents can occur to those skilled in the art without departing from the spirit and scope of the disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein can be employed in practicing the disclosure. It is intended that the following claims define the scope of the disclosure and that methods equivalent to those shown and described herein can be utilized without departing from the spirit and scope of the disclosure.

Claims

1. A data processing apparatus for executing a neural network model, comprising: a storage circuit configured to store a folded filter of a convolution layer of the neural network model, the folded filter being a composition of a plurality of extended filters obtained by moving an original filter by a plurality of convolution steps; and a processing circuit configured to: perform a convolution operation on an input feature map using the folded filter to obtain an intermediate result; and perform a dimension rearrangement on the intermediate result to obtain an output feature map; wherein an output channel number of the original filter is no more than a first threshold Th1, and an output channel number of the folded filter is equal to a second threshold Th2, wherein the first threshold Th1 is less than the second threshold Th2; wherein the processing circuit is further configured to: determine a total folding factor N based on an output channel number Co of the original filter and the second threshold Th2; split the total folding factor N into a folding factor in a kernel width direction Nw and a folding factor in a kernel height direction Nh; determine a convolution step of the folded filter based on Nw, Nh and a convolution step of the original filter; and determine a kernel size of the folded filter based on Nw, Nh and a kernel of the original filter and the convolution step, the kernel size corresponding to a kernel size of the extended filters. 2.The data processing apparatus of claim 1, wherein the folded filter is generated by the processing circuit in the following manner: moving the original filter by (Nw-1) horizontal convolution steps in the kernel width direction and / or by (Nh-1) vertical convolution steps in the kernel height direction to obtain N extended filters, wherein N=Nw×Nh, Nw and Nh are natural numbers; and composing the N extended filters into one folded filter in the output channel dimension. 3.The data processing apparatus of claim 1, wherein the processing circuit is further configured to determine the total folding factor N in the following manner: 4.The data processing apparatus of claim 1, wherein the processing circuit is further configured to split the total folding factor N in any one of the following rules or a combination of the rules: preferentially splitting to the kernel width direction; N = Th2 / Ca, where Ca is the Co aligned to the nearest Th2 / 2 n value, n is a natural number. preferentially splitting to the direction with a smaller convolution step; averagely splitting to the kernel width direction and the kernel height direction; or splitting such that a kernel size of the folded filter is minimized. 5.The data processing apparatus of claim 1, wherein the processing circuit is further configured to determine the convolution step of the folded filter in the following manner: 6.The data processing apparatus of claim 1, wherein the processing circuit is further configured to determine the kernel size of the folded filter in the following manner: 7.The data processing apparatus of claim 1, wherein the processing circuit is further configured to: S' x = S x *Nw S y ′=S y *Nh wherein S x , S y are the horizontal and vertical convolution step sizes of the original filter, respectively, S' x , S y are the horizontal and vertical convolution step sizes of the folded filter, respectively. perform a dimension rearrangement on the intermediate result to obtain an output feature map when the extended filters only include the extended filters obtained by moving the original filter in the kernel width direction. k w ′=k w +(Nw-1)*S x k h ′=k h +(Nh-1)*S y wherein, S x , S y are the horizontal and vertical convolution step sizes of the original filter, respectively, w , k h are the kernel width and kernel height of the original filter, respectively, w , k h are the kernel width and kernel height of the folded filter, respectively. 8.The data processing apparatus of claim 1, wherein the processing circuit is further configured to: ​ ​ When the extended filter only contains the extended filter obtained by moving the original filter in the kernel height direction, the intermediate result is transposed in dimension to obtain the output feature map. 9.The data processing apparatus according to claim 1, wherein the processing circuitry is further configured to: When the extended filter contains both the extended filter obtained by moving the original filter in the kernel width direction and the extended filter obtained by moving the original filter in the kernel height direction, the intermediate result is transposed in dimension and rearranged in dimension to obtain the output feature map. 10.The data processing apparatus according to claim 8 or 9, wherein the processing circuitry is further configured to: transpose the features in the output channel dimension of the intermediate result to the height dimension. 11.The data processing apparatus according to claim 1, wherein the second threshold Th2 is determined based on the number of convolution operation units in the processing circuitry, and the first threshold Th1≤Th2 / 2. 12.The data processing apparatus according to claim 1, wherein the number of input channels of the original filter is equal to the number of input channels of the folded filter. 13.The data processing apparatus according to claim 1, wherein the folded filter is generated offline or online.

14. A chip, characterized by The chip comprises the data processing apparatus according to any one of claims 1-13.

15. A board card, characterized by The board card comprises the chip according to claim 14. 16.A method for executing a neural network model implemented by a data processing apparatus, the data processing apparatus comprising a storage circuitry and a processing circuitry, the method comprising: the processing circuitry performing a convolution operation on an input feature map using a folded filter of a convolution layer of the neural network model stored in the storage circuitry to obtain an intermediate result, wherein the folded filter is a composition of a plurality of extended filters obtained by moving an original filter by multiple convolution step sizes; and the processing circuitry rearranging the intermediate result in dimension to obtain an output feature map; wherein the number of output channels of the original filter does not exceed a first threshold Th1, and the number of output channels of the folded filter is equal to a second threshold Th2, wherein the first threshold Th1 is less than the second threshold Th2; further comprising: the processing circuitry determining a total folding factor N based on the number of output channels Co of the original filter and the second threshold Th2; splitting the total folding factor N into a kernel width direction folding factor Nw and a kernel height direction folding factor Nh; determining a convolution step size of the folded filter based on Nw, Nh, and a convolution step size of the original filter; and determining a kernel size of the folded filter based on Nw, Nh, and a kernel of the original filter and the convolution step size, the kernel size corresponding to a kernel size of the extended filter. 17.The method according to claim 16, wherein the folded filter is generated by the processing circuitry as follows: moving the original filter Nw-1 transverse convolution step sizes in the convolution kernel width direction and / or Nh-1 longitudinal convolution step sizes in the convolution kernel height direction to obtain N extended filters, where N=Nw×Nh, and Nw and Nh are natural numbers; and synthesizing the N extended filters into one folded filter in the output channel dimension.

18. The method of claim 16, further comprising: determining the total folding factor N by the processing circuit as follows: N = Th2 / Ca, where Ca is the Co aligned to the nearest Th2 / 2 n value, n is a natural number.

19. The method of claim 16, further comprising: splitting the total folding factor N by the processing circuit according to any one of the following rules or combination of rules: preferentially splitting into the convolution kernel width direction; preferentially splitting into the direction with smaller convolution step size; averagely splitting into the convolution kernel width direction and the convolution kernel height direction; or splitting into such that the convolution kernel size of the folded filter is minimized.

20. The method of claim 16, further comprising: determining the convolution step size of the folded filter by the processing circuit as follows: S' x = S x *Nw S y ′=S y *Nh where S x , S y are the horizontal and vertical convolution step sizes of the original filter, respectively, and S x ′ , S y are the horizontal and vertical convolution step sizes of the folded filter, respectively.

21. The method of claim 16, further comprising: determining the convolution kernel size of the folded filter by the processing circuit as follows: k w ′= k w +(Nw-1)*S x k h ′= k h +(Nh-1)*S y wherein S x , S y are the horizontal and vertical convolution step size of the original filter respectively, k w , k h are the kernel width and kernel height of the original filter respectively, k w ', k h ' are the kernel width and kernel height of the folded filter respectively.

22. The method of claim 16, wherein the dimension rearranging of the intermediate result comprises: when the extended filters only contain the extended filters obtained by moving the original filter in the convolution kernel width direction, dimension reshaping the intermediate result to obtain the output feature map.

23. The method of claim 16, wherein the dimension rearranging of the intermediate result comprises: when the extended filters only contain the extended filters obtained by moving the original filter in the convolution kernel height direction, dimension transposing the intermediate result to obtain the output feature map.

24. The method of claim 16, wherein the dimension rearranging of the intermediate result comprises: when the extended filters contain both the extended filters obtained by moving the original filter in the convolution kernel width direction and the extended filters obtained by moving the original filter in the convolution kernel height direction, dimension reshaping and dimension transposing the intermediate result to obtain the output feature map.

25. The method of claim 23 or 24, wherein the dimension transposing of the intermediate result comprises: transposing the features in the output channel dimension of the intermediate result to the height dimension.

26. The method of claim 16, wherein the second threshold Th2 is determined based on the number of convolution operation units in the processing circuit, and the first threshold Th1≤Th2 / 2.

27. The method of claim 16, wherein the number of input channels of the original filter is equal to the number of input channels of the folded filter.

28. The method of claim 16, wherein the folded filter is generated offline or online.

Images