Computing Optimization

Abstract

The present disclosure relates to computational optimization. A computer-implemented method for performing a convolution between a subset of an input data array and a kernel to produce a subset of an output data array. The method may include receiving an input data array and determining a subset of the input data array containing at least one non-zero-valued data element using position data indicating positions of elements of the input data array; performing a convolution between the subset of the input data array containing at least one non-zero-valued data element and the kernel to produce an output data array subset; and combining the output data subset with the position data to generate output data indicating a completed output data array.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for managing data in a data processing system. The present disclosure is particularly, but not exclusively, relevant to the management of data within a neural processing unit (NPU). Background Art

[0002] Neural processing units (NPUs) or neural processors are used to accelerate machine learning algorithms. Consisting of specialized electronic circuits designed according to a specified neural network architecture, NPUs use the structure of a neural network to process input data, such as image data, across multiple processing nodes. One such neural network architecture is a convolutional neural network (CNN), which contains convolutional layers, where an input data array is convolved with a kernel or filter to produce an output data array, and other layer types, such as pooling and full connection. A CNN typically contains several convolutional layers, where the output feature map (OFM) is used as the input feature map (IFM) of the next layer.

[0003] It is important that the convolution operations performed by the multiply-accumulate (MAC) unit are performed in an efficient manner to optimize the overall power consumption of the NPU. One such example is to decompose the IFM and kernel into smaller subsets of the data array before performing the convolution with the kernel, resulting in subset OFMs that can be combined to form the complete OFM. Summary of the Invention

[0004] According to a first aspect, a computer-implemented method is provided for performing a convolution between a subset of an input data array and a kernel to produce a subset of an output data array. The method includes receiving the input data array and obtaining position data indicating positions of zero-valued data elements of the input data array. The method also includes determining a subset of the input data array that includes at least one non-zero-valued data element and performing a convolution between the subset of the input data array that includes at least one non-zero-valued data element and the kernel to produce the subset of the output data array. The method combines the output data subset with the position data to generate output data indicating the completion of the output data array.

[0005] According to a second aspect, a computer-implemented method for performing a convolution between an input data array and a kernel to generate an output data array is provided. The method includes decomposing the kernel into subkernels by taking one or more slices, processing the subkernels by determining that one or more rows and / or columns of the original subkernels are to be removed to generate reduced subkernels, storing position data indicating a position of the reduced subkernels within the original subkernels, extracting a subset of the input data array determined by the position data, and performing a convolution of the subset of the input data array and the reduced subkernel to generate a subset of the output data array.

[0006] Further features and advantages will become apparent from the following description of preferred embodiments given by way of example only, made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An example according to the present disclosure shows a flowchart representing a method of transferring initial data or completed OFM data input to a CNN to IFM data at the beginning of a convolutional layer.

[0008] Figure 2 Shown is a schematic block diagram illustrating a data processing system arranged according to a first example of the present disclosure.

[0009] Figure 3 Shown is a schematic block diagram illustrating a data processing system arranged according to a second example of the present disclosure.

[0010] Figure 4 Shown is a schematic block diagram illustrating a data processing system arranged according to a third example of the present disclosure.

[0011] Figure 5 An example of a kernel being decomposed into channel slices is shown.

[0012] Figure 6 Shown Figure 5 An example of a decomposed kernel being further decomposed into sub-kernels.

[0013] Figure 7 An example of a sub-kernel being processed by a weight processor is shown.

[0014] Figure 8 is a schematic block diagram illustrating a data processing system arranged according to examples of the present disclosure.

[0015] Figure 9 Shown Figure 8 A schematic block diagram of a computing engine of a data processing system. DETAILED DESCRIPTION

[0016] Details of the systems and methods according to the examples will become apparent from the following description with reference to the accompanying drawings. In this description, many specific details of certain examples are recorded for illustrative purposes. Reference in the specification to "example" or similar language means that the features, structures, or characteristics described in connection with the example are included in at least one example, but not necessarily in other examples. It should also be noted that some examples are described schematically, in which certain features are omitted and / or necessarily simplified, which is to facilitate the description and understanding of the concepts underlying these examples.

[0017] It is desirable to reduce the power consumption of a neural processing unit (NPU) by reducing the number of computations required to perform convolution. This can be achieved by reducing the number of computations involving zero elements, which would otherwise produce zero results during convolution. Zero elements and at least some of the weights can be identified and removed from the convolution operation for input feature map (IFM) data. Additionally or alternatively, at least some of the zero elements and weights can be identified and removed from the kernel to improve the efficiency of the NPU.

[0018] Allowing the multiply-accumulate (MAC) unit to perform convolutions using a subset of IFMs or kernels consisting of zero-valued data elements results in wasted compute cycles, as the subset output feature map (OFM) will also consist of zero-valued data elements. Bypassing the MAC unit, for example by having the NPU disable computations for zero-valued IFM elements by clock-gating the MAC unit and instead giving them a value of zero, thus skipping these computations, reduces power consumption but results in unused compute cycles with no effect on the data rate through the MAC unit. It would be desirable to increase the data rate through the MAC unit.

[0019] Figure 1A flowchart illustrates an example method for storing initial data or completed OFM data input to the first layer of a convolutional neural network (CNN) from the end of a convolutional layer or other type of layer typically present in the CNN into a memory circuit and subsequently retrieving it as IFM data for the next convolutional layer in the CNN. In one example, for each convolutional layer, an input data array 101 is stored in static random access memory (SRAM) and may be compressed via a compressor 102. Input data array 101 may include initial input data to be processed by the first layer of the CNN or completed OFM data to be processed by the next layer. The initial input data may include image data, such as frames of video data, in which case each completed OFM data will also include (processed) image data. Alternatively, the initial input data may include audio data, in which case each completed OFM data will also include (processed) audio data. Compressor 102 may utilize a compression scheme before the input data array is written to dynamic random access memory (DRAM) 103. The input data array may be compressed selectively or each time the input data array is stored. The input data array 101 may be compressed when the amount of data to be stored exceeds a threshold, indicating that the input data array 101 is too large to be stored in internal SRAM given other storage and / or performance requirements. The compression scheme may generate position data that can be used to identify the location of a data element of a certain value within the input data array 101. In one example, the compression scheme generates a zero-map, which can be generated as multiple data blocks for the input data array. Each block may include a binary data array consisting of NxM elements, which is used to note the location of zero-valued data elements in the input data array 101. This allows the input data array 101 to be compressed to only non-zero-valued data elements by removing zero-valued data elements. The block size is optimized for compression. Too small a block size may result in a large amount of subset data, slowing down calculations. Too large a block size may result in too large a subset of data, slowing down calculations.

[0020] The input data array can be processed by determining a subset of the input data array that includes at least one non-zero valued data element. The subset can be a patch. A patch of the input data array is a subarray of adjacent elements. In some examples, a selected combination of patch size and zero atlas block size produces optimal computational speed. In one example, the zero atlas block can include a 2x2 IFM patch. In one example, the IFM patch size can be 4x4xd elements, where d is the depth of the patch, and in this example d=1. In this example, the zero atlas block size can be 8x8 elements.

[0021] When the data stored in DRAM 103 is retrieved and stored in SRAM to form the IFM data 105 for the next convolutional layer, the data is decompressed by a decompressor 104, which uses the position data from the compressed data stream. In one example, the decompressor 104 recovers the zero map from the compressed stream and uses it to decompress the IFM data 105, recovering all zero-valued data elements that were removed for decompression. In one example, the compressor 102 and the decompressor 104 are pieces of computer software.

[0022] In another example, the SRAM is large enough so that the initial or completed OFM data 101 can be written to the SRAM and read to the next layer as IFM data, eliminating the need for DRAM access. In this case, the compression and decompression steps may or may not be used. In the case where the compression or decompression steps are not used, the position data can still be used in later steps of the data processing system. The SRAM can be a buffer or cache.

[0023] Figure 2 A schematic block diagram of a data processing system is shown, illustrating a first example, in which an IFM data subset is passed from IFM data buffer 204 to MAC unit 207 for convolution with a kernel only if the IFM data subset has at least one non-zero-valued data element, resulting in an OFM data subset stored in OFM data buffer 209. This example is optimized for 1x1 convolution, wherein the example IFM data subset is sized 4x4xd elements, resulting in an OFM data subset of 4x4 elements. In one example, an IFM data subset consists of adjacent elements and is collectively referred to as a patch. In this example, in decompressor 201, position data from the compressed stream is used to decompress the originally compressed data and identify the values of the elements. Decompressor 201 decompresses one IFM data subset at a time. In one example, the position data is a zero map, which can be a binary array or a set of binary arrays. The zero map is used to locate zero-valued data elements in the current IFM data subset. The currently decompressed IFM data subset is then stored in IFM data buffer 204. If the current subset has at least one non-zero-valued data element, the storage location of the subset IFM is pushed to lookup table 202. Decompression of each IFM data subset occurs until the entire IFM data array is stored in the IFM data buffer. At this point, lookup table 202 contains the IFM data buffer 204 storage locations for all IFM data subsets that have at least one non-zero-valued data element. The decompressor also passes the total number of non-zero-valued IFM data subsets to buffer addressing logic 203. In one example, the storage location is the starting address of the IFM data subset in the IFM data buffer.

[0024] Buffer addressing logic 203 performs a lookup in lookup table 202 to collect the starting addresses of IFM data subsets that have at least one non-zero-valued data element. The starting addresses are used to retrieve the IFM data subsets to be passed to MAC unit 207, where they are convolved with the kernel. In one example, the kernel is decomposed into sub-kernels. In another example, the kernel or sub-kernel undergoes pre-processing before being sent to MAC unit 207 to ensure a minimum number of convolution calculations. Buffer addressing logic 203 counts from one to the total number of non-zero-valued IFM data subsets to retrieve each IFM data subset that has at least one non-zero-valued data element. In one example, a MAC array comprising multiple MAC units can be utilized to ensure that convolution calculations occur in parallel. In this example, buffer addressing logic 203 selects each IFM data subset that has at least one non-zero-valued data element in the order in which the starting addresses are entered into the lookup table, thereby utilizing linear addressing. The convolution process generates OFM data subsets. In one example, the IFM data subset consists of a 4x4xd patch, which results in a 4x4 OFM data subset after performing a 1x1 convolution.

[0025] After an OFM data subset is generated by MAC unit 207, buffer addressing logic 208 performs a lookup in lookup table 202 to locate the starting address of the IFM data subset used to generate it. The IFM data buffer 204 starting address is used to generate the OFM data buffer address, which ensures that the OFM data subset is written to the correct location in the completed OFM data array. This process occurs for each OFM data subset generated. Data subsets consisting entirely of zero-valued data elements are also added to OFM data buffer 209. In one example, the zero-valued IFM data subset is sent directly to OFM data buffer 209. In another example, all elements of OFM data buffer 209 are initialized with zero values before any data is sent to it, eliminating the need to write zero values to OFM data buffer 209. The method used is selected based on the highest reduction in power consumed by the NPU performing each process. A combination of these methods produces a completed OFM data array consisting of all generated OFM data subsets. After the OFM data buffer 209 contains the completed OFM data array, the data is flushed to the memory circuit and the OFM data buffer 209 is cleared or initialized to zero, depending on the reduction in power consumed by the NPU when performing this operation. In one example, the completed OFM data array is compressed to DRAM outside the current convolutional layer of the CNN. In another example, a ping-pong buffer configuration can be used to ensure that there is no reduction in throughput while the OFM data array is being transferred. In another example, components included in the data processing system are clock-gated. For IFM data that contains both zero-valued data elements and non-zero-valued data elements, clock gating can be used to skip zero-valued data element calculations to reduce power consumption.

[0026] This example allows for a reduction in NPU power consumption because it reduces the number of convolution calculations performed. Calculations performed using a subset of zero-valued IFM data will produce a subset of zero-valued OFM data. Unlike solutions involving gating MAC units, this example increases throughput because unrequired processing operations are eliminated and functional units are freed up, allowing them to be reused to perform other CNN calculations. In some examples, writes to OFM subsets consisting entirely of zero values are skipped.

[0027] Figure 33 is a schematic block diagram of a data processing system, illustrating a second example in which a subset of IFM data is passed from an IFM data buffer 304 to a MAC unit 308 for convolution with a kernel only if the subset IFM has at least one non-zero-valued data element, resulting in an OFM data subset stored in an OFM data buffer 310. This second example extends support for general NxM convolution. A decompressor 301 uses position data from the compressed stream to decompress the compressed data one IFM data subset at a time and locate the values of the elements contained within the IFM data subset. In one example, the position data is a zero map used to identify zero-valued data elements within the IFM data subset. The IFM data subsets are stored in the IFM data buffer 304, and the storage location of each IFM data subset with at least one non-zero-valued data element within the IFM data buffer 304 is stored in a lookup table 302. Once decompression is complete, all IFM data subsets are stored in IFM data buffer 304 and the storage location of each IFM data subset having at least one non-zero valued data element is stored in lookup table 302. In one example, the storage location is the starting address of the IFM data subset in the IFM data buffer.

[0028] To allow for general NxM convolutions, an IFM receptive field analyzer 305 is utilized. The IFM receptive field analyzer 305 generates an IFM receptive field whose size depends on the convolution requested by the NPU and the size of the OFM data subset. In one example, a 6x6 element receptive field is required for a 4x4 element OFM data subset generated using a 3x3 convolution. The receptive field of the OFM data subset can be constructed by collecting the surrounding IFM data subsets required for the convolution. In one example, the IFM data subset consists of 4x4 elements. To generate the 6x6 element receptive field, elements from adjacent IFM patches are used to complete the receptive field. In the event that an element of the receptive field required to calculate the OFM data subset is outside the boundaries of the IFM data array, a value of zero is used.

[0029] The IFM receive field analyzer 305 collects the IFM data subsets required to generate the OFM data subsets by utilizing buffer addressing logic 303. Buffer addressing logic 303 performs a lookup in lookup table 302 to locate the starting address of the IFM data subset in the IFM data buffer. In this example, the buffer addressing logic can implement any addressing mode required to generate the receive field. Elements from these IFM data subsets are then collected to form the receive field. Only IFM data subsets with at least one non-zero-valued data element have their starting address stored in lookup table 302. If the IFM data subsets required to generate the receive field do not have their starting address included in lookup table 302, the receive field will consist of zero-valued data elements, as these IFM data subsets will consist solely of zero-valued data elements. In this case, the receive field is not passed to the MAC unit 308 for convolution. In one example, the generated zero-valued receive field can be sent to the OFM data buffer 310 or initialized to zero before any data is sent to it, depending on which approach reduces NPU power consumption. If the received field contains at least one IFM data subset containing at least one non-zero valued data element, it is passed to the MAC unit 308 to be convolved with the kernel. In one example, the kernel is decomposed into sub-kernels. In another example, the kernel or sub-kernels undergo pre-processing before being sent to the MAC unit 308 to ensure that a minimum number of convolution calculations occur. In another example, a MAC array containing several MAC units can be utilized to ensure that the convolution calculations occur in parallel.

[0030] The IFM receive domain analyzer 305 indicates to the OFM buffer addressing logic 309 where the resulting OFM data subset will be written in the OFM data buffer 310. It also indicates when a MAC operation has been skipped. When the OFM data buffer 310 contains the completed OFM data array consisting of the generated OFM data subset, it is flushed to the memory circuitry and the contents of the OFM data buffer 310 are either cleared or initialized to zero, depending on which approach results in reduced NPU power consumption during the process. In one example, the completed OFM data array is compressed into DRAM outside the current convolutional layer of the CNN. In another example, a ping-pong buffer configuration can be used to ensure that the NPU throughput is not reduced while the OFM data is being transmitted. In yet another example, the present disclosure is used in conjunction with clock gating of components within a data processing system. For IFM data containing both zero-valued and non-zero-valued data elements, clock gating reduces power consumption by skipping calculations on the zero-valued data elements.

[0031] This example allows for the same reduction in NPU power consumption as the first example, prevents unnecessary convolution of zero-valued IFM data subsets that would otherwise produce zero-valued OFM data subsets, while maintaining computational throughput. In some examples, the need to write zero-valued OFM subsets to OFM data buffer 310 is avoided. However, this example has the advantage of greater flexibility because it allows for general convolution of NxM elements.

[0032] Figure 4 4 is a schematic block diagram of a data processing system, illustrating a third example of the present disclosure, in which a subset of IFM data is passed from IFM data buffer 403 to MAC unit 407 for convolution with a kernel only if the IFM data subset has at least one non-zero-valued data element, resulting in an OFM data subset stored in OFM data buffer 409. This example is optimized for performing 1x1 convolutions. In one example, the IFM data subset is a 4x4xd-element patch, resulting in a 4x4-element OFM data patch. Decompressor 401 similarly uses position data to identify the values of elements in the IFM data subset. In one example, the position data is a zero map, which can be a binary array. This is used to identify zero values in the IFM data subset. In this example, only IFM data subsets with at least one non-zero data element are stored in IFM data buffer 403. When an IFM data subset having at least one non-zero valued data element is stored in the IFM data buffer 403, the decompressor 401 writes a unique subset tag to the non-zero subset tag buffer 404, identifying the location of the IFM subset in the IFM data array. The decompressor 401 also passes the total number of IFM subsets stored in the IFM data buffer 403 to the buffer addressing logic 402. The decompressor 401 decompresses the IFM data subsets one at a time until the entire IFM data array has been viewed.

[0033] Once all IFM data subsets with at least one non-zero value are stored in IFM data buffer 403, buffer addressing logic 402 counts from one to the total number of IFM data subsets stored in IFM data buffer 403, retrieving each subset to pass to MAC unit 407 for convolution with the kernel. In this example, buffer addressing logic 402 reads each IFM data subset in the order in which it was stored, thus using linear addressing. In one example, the kernel is decomposed into sub-kernels. In another example, the kernel or sub-kernel undergoes pre-processing before being sent to the MAC unit to ensure a minimum number of convolution calculations. In yet another example, a MAC array containing multiple MAC units can be utilized to ensure that convolution calculations occur in parallel. Each generated OFM data subset is then written to OFM data buffer 409, where OFM buffer addressing logic 408 uses the non-zero subset tag buffer 404 to locate the correct location in the OFM data array to complete the current OFM subset. In one example, OFM data buffer 409 is initialized to zero before any data is written to it. After OFM data buffer 409 contains a completed OFM data array containing all OFM data subsets, the OFM data is flushed to the memory circuit and the contents of OFM data buffer 409 are initialized to zero values. In one example, the completed OFM data array is compressed to DRAM outside the current convolutional layer of the CNN. In another example, the present disclosure is used in conjunction with clock gating of components included in a data processing system. For IFM data arrays containing both zero-valued data elements and non-zero-valued data elements, the use of clock gating allows for reduced power consumption.

[0034] This example has the advantage of also avoiding convolution of the zero-valued IFM subset with the kernel while maintaining throughput. It does this with a simplified architecture, further reducing NPU power consumption. It also has the advantage of preventing the need to write an IFM data subset consisting entirely of zero values to IFM data buffer 403. In some examples, writing an OFM data subset consisting entirely of zero values to OFM data buffer 409 is prevented.

[0035] Figure 5 A schematic diagram illustrates the decomposition of an example kernel 500 into channel slices 501 and 502. Before convolution occurs in the MAC unit, the kernel can also be decomposed into channel slices to reduce the number of MAC calculations and the overall power consumption of the NPU. Example kernel 500 has a size of 9x9x32, where the depth of 32 matches the depth of the IFM. This depth can be decomposed using channel slices, which can be of equal or different sizes. In this example, a channel slice of 2 is selected, where the kernel depth is decomposed into two equal parts, each of 16. This gives two channel kernels 501 and 502 of 9x9x16.

[0036] Figure 6 Shown from Figure 5 Schematic diagram of channel kernels 501, 502 being further split into 18 sub-kernels 501a, 501b, 501c, 502a, 502b, 502c. In this example, each channel kernel is further decomposed using horizontal and vertical slicing, resulting in a total of 18 sub-kernels of 3x3x16 elements. The sub-kernels generated from the combination of channel, horizontal and vertical slicing will be referred to as original sub-kernels.

[0037] Figure 7 This is to show how the weight processor 702 can be used to further simplify Figure 6 Schematic diagram of three original sub-kernels 701, 704, and 706 consisting of 3x3x16 elements. Each original sub-kernel is passed through a weight processor, which acts as a zero value detector. When one or more columns / rows of a sub-kernel's weights are zero values throughout its depth, a reduced sub-kernel 703, 705, and 707 with fewer elements is generated, thereby reducing the number of MAC calculations during convolution with the IFM. The first original sub-kernel 701 has zero values for the first row and first column throughout its depth. When passed through the weight processor 702, these zero values are removed to produce a reduced sub-kernel 703 of 2x2x16 elements. To ensure that the appropriate elements of the reduced sub-kernel are convolved with the elements of the IFM, horizontal and vertical offsets are stored as position data indicating the position of the reduced sub-kernel within the original sub-kernel. For the original sub-kernel 701, since the first row and column have been removed, the horizontal and vertical offsets are both 1, as this is the position of the first element of the reduced sub-kernel relative to the original sub-kernel 701.

[0038] 706 has zero value on its whole depth in its first and third row and first and third column.When it is passed through weight processor 702, the simplified subkernel 705 obtained is made up of 2x2x16 elements.In this case, the value of horizontal and vertical offset is zero, because the first element of simplified subkernel 705 and original subkernel 704 appear in the same position.The third original subkernel 706 has zero value on its whole depth for its first and third row and first and third column.When it is passed through weight processor 702, the simplified subkernel 707 generated has 1x1x16 elements.Horizontal and vertical offset is 1, because this is the first element of simplified subkernel relative to original subkernel 706.It should be noted that subkernel and simplified subkernel are both kernels.

[0039] In practice, the original sub-kernel may have elements with values close to zero but not zero. To ensure efficient use of the weight processor, a quantizer may be used to zero the weights of the kernel's elements that are less than a set threshold before the sub-kernel is passed to the weight processor. For the purposes of this disclosure, such near-zero values (not zero but set to zero by the quantizer) may be referred to as zero-valued elements. It should be noted that quantizing the weights of the kernel's elements may produce different results than unquantized weight values.

[0040] Once the reduced sub-kernels are generated, they are then convolved with the elements of the IFM with horizontal and vertical offset information. The results from these convolutions can then be used to reconstruct the OFM data that would be generated if the entire kernel were convolved with the IFM data, where the horizontal and vertical offsets are again used to ensure that zero-valued data elements are placed in the correct locations, or the resulting OFM data subset can be sent to the OFM data buffer to be reconstructed there. Figure 5 、 Figure 6 and Figure 7 The combination of the examples shown in has the effect of removing zero-valued data elements from the kernel to avoid having to perform convolution calculations that would only produce zero-valued data elements, thereby reducing power consumption.

[0041] It should be noted that in Figure 2 、 Figure 3 and Figure 4In all three previously discussed examples presented, the data processing system is configured so that subsets of the IFM data passed to the MAC unit can be convolved by sequentially applying kernels or by performing partial convolutions and combining the results. In the case of performing partial convolutions, in one example, there is a series of 3x3x128 element kernels that form a layer of the CNN. Each kernel is split into four channel slices or sub-kernels: 3x3x32 (A0), 3x3x32 (A1), 3x3x32 (A2), 3x3x32 (A3) for the first kernel, 3x3x32 (B0), 3x3x32 (B1), 3x3x32 (B2), 3x3x32 (B3) for the second kernel, and so on. It may be more efficient to convolve the first 3x3x32 element portion of the IFM data subset with the first subkernel of each kernel (A0, B0, etc.), store this partial result in a memory circuit, and only then convolve the second 3x3x32 element portion of the IFM data subset with the second subkernel of each kernel (A1, B1, etc.). The partial results are then used to reconstruct the OFM data subset that would be produced if the IFM data subset were convolved with the kernel in a sequential manner. This example may be more efficient because the same IFM data does not need to be loaded from the memory circuit multiple times, but is instead used in convolutions with multiple subkernels before new IFM data needs to be loaded. It should also be noted that performing partial convolutions can also be used with reduce subkernels.

[0042] The example shown of preventing zero-valued slices contained in the input data array from being sent to the MAC unit can be used alone or in combination with the example shown for decomposing the kernel into sub-kernels and simplifying the sub-kernels to achieve a greater reduction in power consumption. The example shown of decomposing the kernel into sub-kernels and simplifying the sub-kernels can also be used alone or in combination with any of the examples shown for preventing zero-valued slices from the input data array from being sent to the MAC unit, also resulting in a greater reduction in power consumption.

[0043] Figure 8An example of a data processing system 800 arranged according to an example of the present disclosure is shown. In this example, a system bus 802 is connected to a central processing unit (CPU) 504 and dynamic random-access memory (DRAM) 506, although it will be appreciated that the system bus may also be connected to other components such as a graphics processing unit (GPU), a network interface, non-volatile storage, and input / output devices. Data processing system 800 also includes an NPU 808 containing specialized circuitry for a CNN, but may be arranged according to a range of neural network architectures.

[0044] NPU 800 includes a neural control unit (NCU) 810 that generates control data for a compute engine 812. In one example, NCU 810 may include instructions in memory circuits on how to decompose the kernel, as previously described with reference to FIG. Figure 5 and Figure 6 In another example, a user may input instructions on how to decompose the kernel, which instructions are received via the CPU 804. The NCU is also arranged to determine input data to be convolved with the kernel. Figure 8 Three compute engines 812a, 812b, 812c are shown, although it will be appreciated that a different number of compute engines may be utilized in a data processing system without departing from the scope of this disclosure. In one example, the NPU 808 includes sixteen compute engines. Input data is sent from the DRAM 806 and received by the compute engines 812 via direct memory access (DMA) 814 and a main data channel 816. This input data may include image data, IFM data, and weight data associated with kernels to be applied within a given CNN layer. The compute engines are arranged to process the input data in accordance with control data received from the NCU 810 via a control data channel 818. Each compute engine includes SRAM 820 and is configured to process the input data in accordance with the previously referenced Figure 1 、 Figure 2 、 Figure 3 and Figure 4 The various examples discussed retrieve processing circuitry for input data stored by SRAM 820 .

[0045] Figure 9812a. In this example, the other computation engines 812b and 812c include the same components. The computation engines include a MAC computation engine (MCE) 822a that processes data retrieved from the SRAM 820a based on control data from the NCU 810. In one example, the SRAM 820a contains the previously referenced Figure 2 、 Figure 3 and Figure 4 The IFM data buffer, OFM data buffer and kernel discussed above. The MCE 822a passes the IFM data 824a and weight data from the weight processor 826a to the MAC array. In one example, the weight processor is arranged to perform the previously referenced Figure 7 The process discussed above. In another example, the weight processing is performed externally to the NPU using computer software. The MAC array 828a includes a plurality of MAC units and accumulators for performing MAC operations in parallel. According to this example, the MAC array 828a is configured to perform a convolution between a portion of the input data array and the reduce sub-kernel to generate a planar slice of the intermediate data array. In this example, the MAC array 828a includes a 16x16 array of accumulators that can generate a 16x16 planar slice of the intermediate data array by generating the elements of the slice in parallel. In one example, the MAC array 828a can be arranged to sum corresponding elements of multiple intermediate data array slices to generate a slice of the OFM block.

[0046] In another example, the MAC array 828a can be arranged to send slices of the intermediate data array to a programmable layer engine (PLE) 832, which is arranged to perform vector operations on the data to generate slices of the OFM block. The PLE is arranged to perform additional processing operations on the slices of the OFM data, including pooling operations and applying activation functions, and can also be programmed to perform several operations on different layers of the CNN, allowing for the implementation of a variety of CNN architectures. The PLE 832 is arranged to output the processed OFM data to the SRAM 806 of the compute engine. In the context of a CNN, the OFM data becomes IFM data for the next layer in the CNN, which may be, for example, another convolutional layer or a fully connected layer. The processed data can be broadcast to other compute engines 812 for further processing, or can be output to the DRAM 806 of the data processing system 800.

[0047] As mentioned above Figure 1As described, in one example, the initial / IFM data contained in SRAM 820a is compressed using a compression scheme that involves the use of position data while being output to DRAM 806. When the data is retrieved from DRAM 806 as input data for the next layer of the CNN, it is decompressed from the compressed data stream using a zero map and stored in the SRAM 820a of the compute engine. In another example, the initial / IFM data is written to SRAM and read as input data to the next layer and can be processed using compression and decompression steps as required.

[0048] It is to be understood that any feature described in connection with any one example may be used alone or in combination with the other features described, and may also be used in combination with one or more features of any other example or any combination of any other example. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. A computer-implemented method for performing a convolution between a subset of an input data array and a kernel to produce a subset of an output data array, the method comprising: Receive input data array; obtaining position data indicating positions of zero-valued data elements of the input data array; determining, based on the position data, a subset of the input data array containing at least one non-zero-valued data element and a subset of the input data array consisting entirely of zero values, wherein the subset of the input data array corresponds to a subarray of adjacent elements within the input data array; performing a convolution in a multiply-accumulate (MAC) unit between a subset of the input data array containing at least one non-zero-valued data element and a kernel to produce an output data array subset; sending a subset of the input data array consisting entirely of zero values to an output data buffer; and The output data array subset is combined with a subset of the input data array consisting entirely of zero values at the output data buffer to generate output data indicative of a completed output data array.

2. The method of claim 1, wherein: A subset of the input data array containing at least one non-zero valued data element is given an identifier; The subset of the input data array and the identifier are stored; The subset is passed to be convolved with the kernel in the order stored to produce an output data array subset; and The completed output data array is generated by assembling subsets of the output data arrays and analyzing the identifier.

3. The method of claim 1, wherein: The input data array and a memory location of a subset of the input data array containing at least one non-zero valued data element are stored; The subset is passed to be convolved with the kernel in the order stored to produce an output data array subset; and The completed output data array is generated by assembling the output data array subsets and analyzing the storage locations of the input data array subsets.

4. The method of claim 3, wherein: determining the receptive field size of the input data array using the required size of the subset of the output data array and the kernel size; generating an input data array receiving field from a collected subset of contiguous input data arrays determined from said storage locations containing at least one non-zero valued data element; A receptive field containing at least one non-zero valued data element is passed to the convolution; and The memory locations for the receptive fields that are not passed to the convolution are stored for the generation of the completed output data array.

5. The method according to any one of claims 1 to 4, wherein: The compressed data stream is received; Receiving the input data includes decompressing the compressed data stream; and Receiving the position data includes decompressing the compressed data stream.

6. A method as claimed in claim 5, wherein the memory locations of the subset of the input data array containing at least one non-zero valued data element are stored in a lookup table.

7. A method as claimed in claim 6, wherein the position data is a zero map indicating the positions of zero-valued data elements in the input data array.

8. A data processing system comprising: a memory circuit arranged to store the input data array, the kernel, and the output data array; as well as The processing circuit is configured to: Receive input data array; obtaining position data indicating positions of zero-valued data elements of the input data array; determining, based on the position data, a subset of the input data array containing at least one non-zero-valued data element and a subset of the input data array consisting entirely of zero values, wherein the subset of the input data array corresponds to a subarray of adjacent elements within the input data array; performing a convolution in a multiply-accumulate (MAC) unit between a subset of the input data array containing at least one non-zero-valued data element and a kernel to produce an output data array subset; sending a subset of the input data array consisting entirely of zero values to an output data buffer; and The output data array subset is combined with a subset of the input data array consisting entirely of zero values at the output data buffer to generate output data indicative of a completed output data array.

9. A computer-implemented method as claimed in any one of claims 1 to 7 for performing a convolution between an input data array and a kernel to generate an output data array, the method further comprising: decomposing the kernel into sub-kernels by taking one or more slices; processing the sub-kernel by determining that one or more rows and / or columns of an original sub-kernel are to be removed to generate a reduced sub-kernel; storing second position data indicating a position of the reduced sub-kernel within the original sub-kernel; extracting a subset of the input data array determined by the second position data; and A convolution of the subset of the input data array and the reduce sub-kernel is performed to generate a subset of the output data array.

10. The method of claim 9, wherein the processing comprises: identifying data elements having a value of zero to determine the one or more rows and / or columns of the original sub-kernel to be removed to generate the reduced sub-kernel; and The original sub-kernel is passed through a quantizer to identify elements having values below a threshold to thereby allow data elements having a value of zero to be identified.

Images