Hybrid Convolution Operation

Abstract

A device includes one or more processors configured to retrieve a first block of data corresponding to an array of values arranged along at least a first dimension and a second dimension; extract at least a portion of a second block of the data; and perform a first hybrid convolution operation that applies a filter to the first block and at least a portion of the second block to generate output data. The output data includes a first accumulation block and at least a portion of the second accumulation block. The one or more processors are further configured to store the first accumulation block as the first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulation block is adjacent to the first accumulation block along the second dimension.

Description

[0001] Priority claim

[0002] This application claims priority to commonly owned U.S. Provisional Patent Application No. 62 / 969,594, filed on February 3, 2020, and U.S. Non-Provisional Patent Application No. 17 / 165,648, filed on February 2, 2021, the contents of both of which are expressly incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to computing devices and programs for performing convolution. Background Art

[0004] Advances in technology have enabled a wide range of data processing to be performed by portable electronic devices, such as smartphones. For example, conventional portable electronic devices typically have sufficient processing resources to perform data processing, such as filtering image data. Convolution is performed when an input data set (such as a two-dimensional array of image pixel values) is processed by "sliding" a filter over the input data to generate an output value for each input value. Using image data as an example, an output pixel value can be generated for each pixel of the input image based on the pixel value in the input image and also based on the values of the pixel's neighbors in the input image.

[0005] However, because neighboring pixel values are used, multiple input image data blocks are required to generate a single output image data block. For example, generating an output image block may require accessing data from at least four and up to nine input image blocks due to filter overlap between blocks when processing pixels at block edges. As a result, a relatively large number of memory accesses are performed, thereby consuming processor resources and power, introducing complexity associated with calculating addresses for memory accesses, introducing alignment issues that may result in additional memory transfers, or a combination thereof.

[0006] In addition to image processing and similar applications, convolution is increasingly used in machine learning applications, such as convolutional neural networks. In neural network applications, convolutional layer inputs can have a large number of channels (e.g., thousands), thereby limiting the benefits of traditional techniques that can be used to improve efficiency, such as maintaining parts of the input data in an intermediate state so that they can be reused in multiple iterations of the processing loop to reduce memory transfers in single instruction multiple data (SIMD) type processing systems. With the rapid popularity of convolutional neural networks, reducing the amount of memory transfers during convolution and the complexity associated with memory transfers can significantly improve device performance and user experience. Summary of the Invention

[0007] According to one implementation of the present disclosure, a device includes a memory and one or more processors. The memory is configured to store data corresponding to an array of values arranged at least along a first dimension and a second dimension. The one or more processors are configured to retrieve a first block of data and at least a portion of a second block of data from the memory, and to perform a first hybrid convolution operation, which applies a filter on the first block of data and at least a portion of the second block of data to generate output data. The output data includes a first accumulation block and at least a portion of a second accumulation block. The one or more processors are further configured to store the first accumulation block as first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulation block is adjacent to the first accumulation block along the second dimension.

[0008] According to another implementation of the present disclosure, a method for performing convolution includes retrieving, at one or more processors, a first block of data and at least a portion of a second block of data from a memory. The data corresponds to an array of values arranged along at least a first dimension and a second dimension. The method includes performing a first hybrid convolution operation that applies a filter to at least a portion of the first block and the second block to generate output data. The output data includes a first accumulated block and at least a portion of the second accumulated block. The method also includes storing the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

[0009] According to another implementation of the present disclosure, a computer-readable storage device stores instructions that, when executed by one or more processors, cause the one or more processors to retrieve from a memory a first block of data and at least a portion of a second block of data, the data corresponding to an array of values arranged along at least a first dimension and a second dimension. The instructions also cause the one or more processors to perform a first hybrid convolution operation that applies a filter to at least a portion of the first block and the second block to generate output data. The output data includes a first accumulated block and at least a portion of the second accumulated block. The instructions also cause the one or more processors to store the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

[0010] According to another implementation of the present disclosure, an apparatus includes means for retrieving a first block of data and at least a portion of a second block of data, the data corresponding to an array of values arranged along at least a first dimension and a second dimension. The apparatus includes means for performing a hybrid convolution operation that applies a filter on the first block and at least a portion of the second block to generate output data. The output data includes a first accumulated block and at least a portion of the second accumulated block. The apparatus includes means for storing the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

[0011] Other aspects, advantages, and features of the present disclosure will become apparent after reviewing the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of a particular illustrative implementation of a system including a device operable to perform a hybrid convolution operation according to some examples of the present disclosure.

[0013] Figure 2 is a diagram showing a hybrid convolution operation as part of some examples of the present disclosure. Figure 1 A diagram of the operations performed by a device.

[0014] Figure 3 is a diagram showing some examples according to the present disclosure. Figure 1 Diagram of the operations performed by the device in conjunction with hybrid convolution operations.

[0015] Figure 4 is a diagram showing some examples according to the present disclosure. Figure 1 Diagram of the pipelined execution of a sequence of hybrid convolution operations performed by a device.

[0016] Figure 5 is a diagram of another implementation of a device operable to perform a hybrid convolution operation according to some examples of the present disclosure.

[0017] Figure 6 According to some examples of the present disclosure, Figure 1 A flowchart of an implementation of a method for performing convolution performed by a device.

[0018] Figure 7 is a block diagram of a specific illustrative example of a device operable to perform a hybrid convolution operation in accordance with some examples of the present disclosure. DETAILED DESCRIPTION

[0019] When processing data values at the edges of an input data block retrieved from memory to locate neighboring data values of data values located in other input data blocks in memory, conventional convolution operations may require relatively large amounts of memory access and complexity. The memory access and complexity may result in reduced performance of the device, may impair the user experience (such as due to reduced device performance when running such an application), and may increase power consumption (which may reduce useful battery life).

[0020] A hybrid convolution operation is proposed that "slides" input data across one dimension (the "first" dimension) and outputs data across another dimension (the "second" dimension). As a result, in some implementations, each output data block requires only two input memory blocks aligned along the first dimension. Explicit overlapping reads of input data along the first dimension can be performed without requiring overlapping reads of input data along the second dimension. This reduces overall memory accesses and complexity compared to conventional techniques, resulting in enhanced device performance.

[0021] Two accumulators (e.g., registers, dedicated memories, or other circuits) can be used to store local accumulated values, and the size of each accumulator can store a data block. The accumulators are aligned in the second dimension so that when a row of data values (e.g., a sequence of data values extending along the second dimension) is written to the accumulator, when a row of one of the accumulators is filled, the subsequent data values are written to the corresponding row of the other accumulator. In some implementations, the accumulators are physically aligned (e.g., physically adjacent registers). In other implementations, the accumulators are not physically aligned, but logically aligned (e.g., physically non-adjacent registers are accessed for data storage and retrieval as if the registers were physically adjacent). The local accumulated values can be saved until completion, thereby avoiding additional memory access associated with storing the local accumulated values. In some implementations, a ping-pong technique is used to alternately use accumulators between iterations of the hybrid convolution operation.

[0022] As non-limiting examples, the hybrid convolution operation can be applied to two-dimensional convolution layers, can be applied to traditional convolutions with two or more dimensions (including neural network channel methods), and can be applied to other filtering operations such as maximum / average (pooling), absolute difference, image correlation. As illustrative examples, the hybrid convolution techniques described herein can be applied to scalar processing, single instruction multiple data (SIMD) vector processing, and SIMD array processing.

[0023] As used herein, "coupling" may include "communicative coupling," "electrical coupling," or "physical coupling," and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some implementations, two devices (or components) that are communicatively coupled (such as in an electrically communicative manner) may send and receive signals (e.g., digital signals or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without an intermediate component.

[0024] It will also be understood that the terms “comprise; comprises; comprising” can be used interchangeably with “include; includes; including”. Furthermore, it will be understood that the term “wherein” can be used interchangeably with “wherein”. As used herein, “exemplary” can indicate examples, implementations, and / or aspects, and should not be construed as limiting or indicating a preferred or preferred implementation. As used herein, ordinal terms (e.g., “first,” “second,” “third,” etc.) used to modify elements such as structures, components, operations, etc. do not, by themselves, indicate any priority or order of the element relative to another element, but merely distinguish the element from another element having the same name (but using an ordinal term). As used herein, the term “set” refers to one or more specific elements, and the term “plurality” refers to a plurality (e.g., two or more) of specific elements.

[0025] The term "configuration" may be used with reference to a method, an apparatus, a device, a system, or any combination thereof, as indicated by its particular context. Where the term "comprising" is used in this specification and claims, it does not exclude other elements or operations. The term "based on" (as in "A is based on B") is used to indicate any of its ordinary meanings, including (i) "based at least on" (e.g., "A is at least based on B") and (ii) (if appropriate in the particular context) "equal to" (e.g., "A is equal to B"). In the case of "A is based on B" including "at least based on" (i), this may include a configuration where A is coupled to B. Similarly, the term "responsive to" is used to indicate any of its ordinary meanings, including "at least responsive to". The term "at least one" is used to indicate any of its ordinary meanings, including "one or more". The term "at least two" is used to indicate any of its ordinary meanings, including "two or more".

[0026] In this disclosure, terms such as "generate," "calculate," "estimate," "use," "select," "access," and "determine" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or may refer to using, selecting, or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0027] Figure 1 A system 100 is depicted that includes a device 102 configured to perform a first hybrid convolution operation 132 that processes input data 110 using a filter 130. In a particular implementation, the filter 130 is a representative filter among a plurality of filters supported by one or more processors 126, such as an array of programmable weights operable as a convolution kernel. The device 102 includes a memory 108 (e.g., dynamic random access memory (DRAM)) coupled to the one or more processors 126. As illustrative, non-limiting examples, the one or more processors 126 are configured to convolve the input data 110 with the filter 130 according to an image processing operation 150 (e.g., a feature detection operation) or according to a neural network convolution layer operation 152 (e.g., processing the input data using a convolutional neural network).

[0028] The memory 108 stores input data 110. In some implementations, the input data 110 corresponds to image data, such as image data captured by a camera 104 coupled to or integrated into the device 102. In some implementations, the input data 110 corresponds to audio data, such as audio data captured by a microphone 106 coupled to or integrated into the device 102. In some implementations, the input data 110 corresponds to one or more layers of data to be operated on by a convolutional neural network.

[0029] The input data 110 corresponds to an array of values logically arranged in rows and columns, where each column extends along a first dimension 134 and each row extends along a second dimension 136. Although the input data 110 is described herein in terms of rows and columns, the input data 110 need not actually be stored in physical rows and columns of the memory 108 in a manner that replicates the logically arranged array (e.g., where adjacent array values are stored in adjacent physical memory locations). For illustration, the memory 108 can be configured to provide access to the input data 110 to the one or more processors 126 based on a memory access data size 148 supported by the memory 108 for memory access by the one or more processors 126. In some implementations, the memory access data size 148 is a configuration value stored in a register that can be adjusted. In other implementations, the memory access data size 148 is a non-configurable, inherent value based on the layout or design of the one or more processors 126, the memory 108, or a combination thereof. In some implementations, each access to input data 110 retrieves a value read from a sequential memory location in memory 108 , such as a cache line of values, but may represent values corresponding to a two-dimensional segment of an array.

[0030] As further described herein, the input data 110 may be arranged in physical memory locations such that each memory access that retrieves a portion of the input data 110 retrieves a corresponding M×M segment (M being a positive integer) of the array of values. To illustrate, in an implementation where M is 4 and the input data 110 represents a two-dimensional array of pixel values corresponding to an image, each access to read the input data 110 from the memory 108 may correspond to a 4×4 block of pixel values (e.g., 16 pixel values from 4 adjacent rows and 4 adjacent columns of the array of pixel values) arranged as 16 pixel values in consecutive physical locations in the memory 108 so as to be retrievable from the memory 108 as a single cache line.

[0031] The one or more processors 126 are configured to retrieve the first block 114 of input data 110 and at least a portion 144 of the second block 116 of input data 110 from the memory 108 and perform a first hybrid convolution operation 132 that applies the filter 130 to the first block 114 and at least the portion 144 of the second block 116 to generate output data. The output data includes the first accumulation block 128 of data output to the first accumulator 140 and at least a portion 146 of the second accumulation block 138 of data output to the second accumulator 142. In some implementations, each of the first accumulator 140 and the second accumulator 142 corresponds to a register or other data storage circuit configured to store a value that is updated as additional values are accumulated (e.g., added) with the stored value. In some implementations, each of the first accumulator 140 and the second accumulator 142 also includes circuitry (e.g., an adder) configured to receive an input value and accumulate the input value with the stored value. The one or more processors 126 are configured to transfer and store the first accumulation block 128 from the first accumulator 140 to the memory 108 as the first output data 122 .

[0032] As reference Figure 2-Figure 3 As further depicted, portion 144 of second block 116 is adjacent to first block 114 along first dimension 134, and portion 146 of second accumulation block 138 is adjacent to first accumulation block 128 along second dimension 136. In some implementations, a size of each of first block 114, second block 116, first accumulation block 128, and second accumulation block 138 is determined based on a memory access data size 148.

[0033] In some implementations, the first hybrid convolution operation 132 uses zero padding along the second dimension 136 (or performs an operation mathematically equivalent to using zero padding), but does not use zero padding along the first dimension 134, as described in reference to FIG. Figure 2-Figure 3 As further shown, using zero padding along one dimension but not along another dimension enables the first hybrid convolution operation 132 to perform the convolution using at most two memory accesses to retrieve the portion 144 of the first block 114 and the second block 116, and to store the first accumulated block 128 (e.g., as the first output data 122) and the portion 146 of the second accumulated block 138 (e.g., as the second output data 124) using at most two memory accesses. Compared to conventional convolution operations that access data from portions of four input blocks to generate one output block or access one input block to generate output data corresponding to portions of four output blocks, the first hybrid convolution operation 132 uses fewer overall memory accesses to retrieve input data and store output data.

[0034] The one or more processors 126 may also be configured to execute the sequence of mixed convolution operations including the first mixed convolution operation 132 and the second mixed convolution operation in a pipelined manner, wherein the storage of the first accumulation block and the execution of the second mixed convolution operation occur during the same pipeline cycle, as described with reference to FIG. Figure 4 As further described in detail, in one example, the second hybrid convolution operation sequentially follows the first hybrid convolution operation 132 and operates on portions of the third block 118 and the fourth block 120 of input data, using the portion 146 of the second accumulation block 138 as a “seed” value to generate the second output data 124. Pipelining the sequence of hybrid convolution operations reduces overall latency by enabling memory access and computation to be performed in parallel, and the sequential execution of the hybrid convolution operations can reduce the complexity associated with tracking and aligning local convolution data (e.g., the portion 146 of the second accumulation block 138) compared to some conventional techniques.

[0035] Figure 2 A first set of diagrams 200 illustrating operations performed according to a particular implementation of the first hybrid convolution operation 132 is depicted. The first diagram 202 shows an example of the filter 130 as a 3×3 array, the first block 114 as a 4×4 array, the second block 116 as a 4×4 array, the first accumulation block 128 as a 4×4 array, and the second accumulation block 138 as a 4×4 array. Each of the filter 130, the first block 114, the second block 116, the first accumulation block 128, and the second accumulation block 138 corresponds to a data structure, such as a logically arranged value.

[0036] The size and arrangement of the data structures are depicted for ease of explanation, and in other implementations, one or more of the filter 130, the first block 114, the second block 116, the first accumulation block 128, and the second accumulation block 138 have different sizes or arrangements. The size of each of the first block 114, the second block 116, the first accumulation block 128, and the second accumulation block 138 (e.g., the number of elements) matches the memory access data size 148 (e.g., 16 elements), so that the first block 114 can be retrieved from the memory 108 via a first memory access operation (e.g., a single cache line read or load operation), the second block 116 can be retrieved from the memory 108 via a second memory access operation (e.g., another single cache line read or load operation), the first accumulation block 128 can be stored to the memory 108 via a third memory access operation (e.g., a single cache line write or store operation), and the second accumulation block 138 can be stored to the memory 108 via a fourth memory access operation (e.g., another single cache line write or store operation).

[0037] Filter 130 includes nine values, labeled k1, k2, ..., k9, arranged in three rows and three columns. First block 114 includes sixteen values, including a first row with values a1, a2, a3, and a4, a second row with values b1, b2, b3, and b4, a third row with values c1, c2, c3, and c4, and a fourth row with values d1, d2, d3, and d4. Portion 144 of second block 116 includes eight values, arranged in a first row with values e1, e2, e3, and e4, and a second row with values f1, f2, f3, and f4. The fourth row of first block 114 is adjacent to the first row of second block 116 along first dimension 134. First block 114 and portion 144 of second block 116 together correspond to N rows and the first M columns of the array of input data 110, where N and M are positive integers and N is greater than M. Filter 130 includes P rows and P columns of filter values, where P = N - M + 1. Figure 2 In the example of , M=4, N=6, and P=3.

[0038] The one or more processors 126 are configured to access the first block 114 using a first memory access operation 154 and to access a portion 144 of the second block 116 using a second memory access operation 156. Each of the first memory access operation 154 and the second memory access operation 156 is configured to retrieve at most M rows and M columns of the input data 110. The third and fourth rows of the second block 116 are not operated on during the first hybrid convolution operation 132 and are therefore shown as having no values. In some implementations, the entire second block 116 is retrieved from the memory 108 during the second memory access operation 156, even though the values of the third and fourth rows are not operated on during the first hybrid convolution operation 132. In other implementations, the second memory access operation 156 omits retrieving the third and fourth rows of the second block 116 from the memory 108.

[0039] In the first diagram 202, the third column (e.g., the rightmost column) of the filter 130 is shown multiplied with the first three rows (e.g., rows a, b, and c) of the first block 114. The accumulated value A1 generated in the first row and first column of the first accumulation block 128 is: A1 += (k3*a1) + (k6*b1) + (k9*c1), where "*" represents a multiplication operation and "+=" represents a compound addition assignment operator, as further described below. The accumulated value A2 generated in the first row and second column of the first accumulation block 128 is: A2 += (k3*a2) + (k6*b2) + (k9*c2). The accumulated value A3 generated in the first row and third column of the first accumulation block 128 is: A3+=(k3*a3)+(k6*b3)+(k9*c3), and the accumulated value A4 generated in the first row and fourth column of the first accumulation block 128 is: A4+=(k3*a4)+(k6*b4)+(k9*c4).

[0040] The second graph 204 shows the second column (e.g., the middle column) of the filter 130 multiplied by the first three rows of the first block 114. An updated accumulated value A2 is generated in the first row, second column of the first accumulation block 128 by adding the previous value of A2 (as described with reference to the first graph 202) to the value (k2*a1)+(k5*b1)+(k8*c1). An updated accumulated value A3 is generated in the first row, third column of the first accumulation block 128 by adding the previous value of A3 to the value (k2*a2)+(k5*b2)+(k8*c2). An updated accumulated value A4 is generated in the first row, fourth column of the first accumulation block 128 by adding the previous value of A4 to the value (k2*a3)+(k5*b3)+(k8*c3). The accumulated value A5 generated in the first row and first column of the second accumulation block 138 is: A5=(k2*a4)+(k5*b4)+(k8*c4).

[0041] The third graph 206 shows the multiplication of the first column (e.g., the leftmost column) of the filter 130 by the first three rows of the first block 114. An updated accumulated value A3 is generated in the first row, third column of the first accumulation block 128 by adding the previous value of A3 to the value (k1*a1)+(k4*b1)+(k7*c1). An updated accumulated value A4 is generated in the first row, fourth column of the first accumulation block 128 by adding the previous value of A4 to the value (k1*a2)+(k4*b2)+(k7*c2). An updated accumulated value A5 is generated in the first row, first column of the second accumulation block 138 by adding the previous value of A5 to the value (k1*a3)+(k4*b3)+(k7*c3). The accumulated value A6 generated in the first row and second column of the second accumulation block 138 is: A6=(k1*a4)+(k4*b4)+(k7*c4).

[0042] continue Figure 3, the fourth figure 302, the fifth figure 304 and the sixth figure 306 depict the Figure 2 The fourth diagram 302 shows the filter 130 multiplied by the second, third, and fourth rows (rows b, c, and d) of the first block 114 to be similar to Figure 2 The described manner generates the accumulated values B1 , B2 , B3 and B4 in the first accumulation block 128 and the accumulated values B5 and B6 in the second accumulation block 138 .

[0043] The fifth graph 304 shows the filter 130 multiplying the third and fourth rows (rows c and d) of the first block 114 and the first row (row e) of the second block 116 to generate the accumulated values C1, C2, C3, and C4 in the first accumulation block 128 and the accumulated values C5 and C6 in the second accumulation block 138. The sixth graph 306 shows the filter 130 multiplying the fourth row (row d) of the first block 114 and the first two rows (row e and f) of the second block 116 to generate the accumulated values D1, D2, D3, and D4 in the first accumulation block 128 and the accumulated values D5 and D6 in the second accumulation block 138.

[0044] Upon completion of the first hybrid convolution operation 132, the first accumulation block 128 and the second accumulation block 138 have a configuration as depicted in the sixth diagram 306, wherein the first accumulation block 128 has an accumulated value representing the convolution of the filter 130 with the portion 144 of the first block 114 and the second block 116. The second block 116 includes a local accumulated value in the portion 146 that can be used as a "seed" value for the next hybrid convolution operation. The first accumulation block 128 and the second accumulation block 138 together correspond to M rows and N columns of output data. In a specific implementation, the first accumulation block 128 corresponds to M rows and M columns of accumulated values in the first accumulator 140, and the portion 146 of the second accumulation block 138 corresponds to the local accumulated values of the M rows and NM columns in the second accumulator 142.

[0045] The seventh diagram 308 shows a portion of a second hybrid convolution operation following the first hybrid convolution operation 132. The one or more processors 126 perform the second hybrid convolution operation on portions of the third block 118 and the fourth block 120 to update the second accumulation block 138 and generate at least a portion of a third accumulation block 338 adjacent to the second accumulation block 138 along the second dimension 136. The third block 118 and at least a portion of the fourth block 120 together correspond to the Nth row and the second Mth column (e.g., column 5, column 6, column 7, and column 8) of the array that has been retrieved from the memory 108, and the first accumulation block 128 has been stored to the memory 108 as the first output data 122 via the third memory access operation 158. The local accumulated value in the portion 146 of the second accumulation block 138 is used as a seed value, and the filter 130 is applied to the values of the third block 118 and the values of the fourth block 120 in a manner similar to that described with respect to Figures 202-206 and 302-206. For example, the seventh graph 308 depicts the accumulation results after applying the filter 130 to the first three rows of the third block 118 to generate accumulated values A5, A6, A7, and A8 in the second accumulation block 138 and accumulated values A9 and A10 in the third accumulation block 338. In some implementations, the third accumulation block 338 is generated in the third accumulator, and in other implementations, the first accumulator 140 and the second accumulator 142 are accessed according to a ping-pong technique, where the third accumulation block 338 is generated in the first accumulator 140 after the first accumulation block 128 has been transferred to the memory 108.

[0046] The second hybrid convolution operation continues in a similar manner to the first hybrid convolution operation 132 and fills portions of the second accumulation block 138 and the third accumulation block 338 based on the third block 118 and the fourth block 120 of the input data 110. After the second hybrid convolution operation is completed, the one or more processors 126 can store the second accumulation block 138 as the second output data 124 and retrieve additional blocks of the input data 110 for processing during the third hybrid convolution operation, such as with reference to FIG. Figure 4 Further described.

[0047] Will Figure 2 Figures 202-206 and Figure 3 The graphical depiction of the first hybrid convolution operation 132 in Figures 302-306 of FIG. 130 being implemented as a sliding convolution is provided for clarity of understanding and should not be considered limiting, wherein the filter 130 is applied to one or more blocks of input data 110 in a first dimension 134 (e.g., a set of 6×4 values from two 4×4 blocks) to generate accumulated data spanning one or more accumulation blocks in a second dimension 136 (e.g., a set of 4×6 accumulated values from two 4×4 accumulation blocks). Although the first hybrid convolution operation 132 is implemented in FIG. Figure 2-Figure 3134 as “sliding” the filter 130 from left to right in the second dimension 136 and from top to bottom in the first dimension 134, but in other implementations, the first hybrid convolution operation 132 may include sliding the filter 130 from right to left in the second dimension 136, from bottom to top in the first dimension 134, or both. Although zero padding is not depicted, Figure 2-Figure 3 The multiplication described in can be mathematically equivalent to a convolution in which one or more columns of zero padding are added to the left, right, or both (but not to the top or bottom) of the first block 114 and the second block 116. The alignment of blocks of input data 110 with blocks of accumulated data, and the formatting of such blocks for efficiency in storage, retrieval, and arithmetic processing, can deviate from Figure 2-Figure 3 Examples are depicted and may vary depending on the specific implementation.

[0048] Figure 4 14. An implementation is depicted in which one or more processors 126 are configured to perform a sequence of mixed convolution operations during a first pipeline cycle 406, a second pipeline cycle 414 following the first pipeline cycle 406, and a third pipeline cycle 424 following the second pipeline cycle 414. The sequence of mixed convolution operations includes a first mixed convolution operation 132, a second mixed convolution operation 434, and a third mixed convolution operation 436 performed in a pipelined manner, wherein storage by the first accumulation block 128 and execution of the second mixed convolution operation 434 occur in the same pipeline cycle, as further explained below. A portion of the array of input data 110 is illustrated as including a first block 114, a second block 116, a third block 118, a fourth block 120, a fifth block 402 adjacent to the third block 118, and a sixth block 404 adjacent to the fifth block 402 and the fourth block 120. Although six blocks of input data 110 are shown for purposes of example, in other implementations, input data 110 may include any number (eg, thousands) of blocks.

[0049] During a first pipeline cycle 406, data from the first block 114 and at least a portion 144 of the second block 116 are combined via a first multiply and accumulate operation 408 (such as reference Figure 2-Figure 34. The first pipeline cycle 406 is processed by the multiplication and accumulation described above (multiplication and accumulation), and the first multiplication and accumulation operation 408 generates accumulated data 410 and local data 412. The first block 114 and the portion 144 of the second block 116 are retrieved by the first memory access operation 154 and the second memory access operation 156, respectively. In some implementations, the first memory access operation 154 and the second memory access operation 156 are performed during the first pipeline cycle 406, or in other implementations, the first memory access operation 154 and the second memory access operation 156 are performed before the first pipeline cycle 406. In the illustrative example, the accumulated data 410 corresponds to the first accumulation block 128, and the local data 412 corresponds to the portion 146 of the second accumulation block 138 that is retained as a seed value for the second mixed convolution operation 434 (e.g., as described in reference to FIG. Figure 3 308 of FIG. 7).

[0050] During the second pipeline cycle 414, the data of the third block 118 and at least a portion of the data of the fourth block 120 are processed via a second multiply and accumulate operation 418 to generate accumulated data 420 and partial data 422. The third block 118 and the fourth block 120 are retrieved via two memory access operations in a manner similar to the first block 114 and the second block 116. In parallel with the multiply and accumulate operation 418 (i.e., during the second pipeline cycle 414), the accumulated data 410 is stored to the memory 108 as the first accumulated block 128 via an output data storage operation 416 (e.g., the third memory access operation 158).

[0051] During the third pipeline cycle 424, the data of the fifth block 402 and at least a portion of the data of the sixth block 404 are processed via a third multiplication and accumulation operation 428 to generate accumulated data 430 and partial data 432. In parallel with the multiplication and accumulation operation 428, the accumulated data 420 is stored to the memory 108 as the second accumulation block 138 via an output data storage operation 426. After the third pipeline cycle 424 (e.g., during a fourth pipeline cycle), the accumulated data 430 is stored via an output data storage operation 438.

[0052] Although only three pipeline cycles are depicted, the hybrid convolution operation sequence can continue until all block rows and block columns of the input data 110 have been processed. As used herein, a "block row" is a set of adjacent rows that matches multiple rows in a block of the input data 110, and a "block column" is a set of adjacent columns that matches multiple columns in a block of the input data 110. As shown, the first block row of the input data 110 includes the first block 114, the third block 118, and the fifth block 402; the second block row of the input data 110 includes the second block 116, the fourth block 120, and the sixth block 404; the first block column of the input data 110 includes the first block 114 and the second block 116, and the second block column of the input data 110 includes the third block 118 and the fourth block 120. In an implementation where each block of input data 110 is sized as a 4×4 block (eg, 4 rows and 4 columns), each block row spans 4 rows of input data 110 , and each block column spans 4 columns of input data 110 .

[0053] The processing of the input data 110 may be performed according to the pseudo code shown in Table 1.

[0054]

[0055] Table 1

[0056] Figure 5 An implementation of a device 502 is depicted that includes a circuit 516 configured to perform a hybrid convolution operation, such as the first hybrid convolution operation 132. The circuit 516 is integrated into a discrete component, such as an integrated circuit (e.g., an application specific integrated circuit (ASIC)). In some implementations, the device 502 corresponds to the reference Figure 7 In some implementations, the circuitry 516 includes one or more processors 126 configured to execute stored instructions to perform the operations described with respect to the first hybrid convolution operation 132, dedicated circuitry (e.g., hardware) that performs the operations described with respect to the first hybrid convolution operation 132, or any combination thereof.

[0057] The device 502 includes an input 510 (such as a first bus interface) to enable input data 110 (or one or more blocks of input data 110) to be received from one or more storage devices external to the device 502 as input data 504. The device 502 also includes an output 512 (such as a second bus interface) to enable the transmission of computed output data 514, such as output data 112 (or one or more blocks of output data 112), during a hybrid convolution operation. The device 502 enables the convolution processing to be implemented as a component in a system (such as a dedicated neural network processing unit (e.g., including one or more graphics processing units (GPUs), digital signal processors (DSPs), or any combination thereof)) for use in a cloud-based server system to accelerate inference or other machine learning applications.

[0058] refer to Figure 6 , depicts a specific implementation of a method 600 for performing convolution. In some implementations, the method 600 is performed by the device 102 or the device 502, as illustrative and non-limiting examples.

[0059] Method 600 includes, at 602, retrieving, at one or more processors, a first block of data and at least a portion of a second block of data from a memory. The data corresponds to an array of values arranged along at least a first dimension and a second dimension. For example, one or more processors 126 retrieve first block 114 of input data 110 and at least a portion 144 of second block 116 of input data 110.

[0060] Method 600 includes, at 604, performing a first hybrid convolution operation that applies a filter to at least a portion of the first block and the second block to generate output data. The output data includes at least a portion of the first accumulation block and the second accumulation block. For example, one or more processors 126 perform a first hybrid convolution operation 132 that applies a filter 130 to the first block 114 and to a portion 144 of the second block 116 to generate a first accumulation block 128 and a portion 146 of the second accumulation block 138. As an illustrative example, one or more processors 126 are configured to convolve the input data 110 with the filter 130 according to at least one of the image processing operation 150 or the neural network convolution layer operation 152.

[0061] Method 600 includes, at 606, storing the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension. For example, one or more processors 126 store first accumulated block 128 in memory 108 as first output data 122. Portion 144 of second block 116 is adjacent to first block 114 along first dimension 134, and portion 144 of second accumulated block 138 is adjacent to first accumulated block 128 along second dimension 136.

[0062] In some implementations, the first hybrid convolution operation uses zero padding along the second dimension 136 without using zero padding along the first dimension 134 to implement the convolution of the first block 114 and the portion 144 of the second block 116 using at most two memory accesses and the portion 146 of the first accumulation block 128 and the second accumulation block 138 using at most two memory accesses.

[0063] In some implementations, the size of each of the first block, the second block, the first accumulation block, and the second accumulation block is determined according to a memory access data size supported by the memory for memory access by one or more processors, such as the memory access data size 148. In some implementations, retrieving the first block includes using a first memory access operation 154; retrieving a portion 144 of the second block 116 includes using a second memory access operation 156; each of the first memory access operation 154 and the second memory access operation 156 is configured to retrieve at most M rows and M columns of the input data 110 (where M is a positive integer); and each of the first accumulation block 128 and the second accumulation block 138 corresponds to M rows and M columns of the output data 112.

[0064] In some implementations, method 600 also includes: retrieving a third block 118 of input data 110 adjacent to the first block 114 along the first dimension 134 of the array; retrieving at least a portion of a fourth block 120 of input data 110 adjacent to a portion 144 of the second block 116 along the first dimension 134 of the array; performing a second hybrid convolution operation (e.g., second hybrid convolution operation 434) on the third block 118 and a portion of the fourth block 120 to update the second accumulation block 138 and generate at least a portion of a third accumulation block adjacent to the second accumulation block along the second dimension 136 (e.g., local data 422); and storing the second accumulation block as second output data 124.

[0065] As an illustrative example, Figure 4As shown, the first mixed convolution operation 132 and the second mixed convolution operation 434 are part of a sequence of mixed convolution operations performed in a pipelined manner, wherein the storage of the first accumulation block 128 and the execution of the second mixed convolution operation 434 occur during the same pipeline cycle (e.g., the second pipeline cycle 414). In some implementations, the method 600 includes: generating the first accumulation block 128 in the first accumulator 140; generating the second accumulation block 138 in the second accumulator 142; and generating a third accumulation block in the first accumulator 140 after storing the first accumulation block 128, such as in the reference Figure 4 In the ping pong implementation described.

[0066] The method 600 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processor (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, the method 600 may be performed by a processor executing instructions, such as those described herein.

[0067] Figure 7 A block diagram depicts a specific illustrative implementation of a device 700 that includes one or more processors 126, such as in a wireless communication device implementation (e.g., a smartphone) or a digital assistant device implementation. In various implementations, the device 700 may have more than Figure 7 In an illustrative implementation, device 700 may correspond to device 102. In an illustrative implementation, device 700 may execute the reference Figures 1-6 Describes one or more operations.

[0068] In certain implementations, the device 700 includes a processor 706 (e.g., a central processing unit (CPU)) coupled to one or more processors 126 (e.g., one or more DSPs, GPUs, other processors, or any combination thereof). In some implementations, the one or more processors of the device 700 include a speech and music codec-decoder (CODEC), which may include a speech coder (“vocoder”) encoder, a vocoder decoder, or both.

[0069] The device 700 may include a memory 752 and a CODEC 734. The memory 752 may correspond to the memory 108 and may include instructions 756 executable by the one or more processors 126 to implement the functionality described with reference to the first mixed convolution operation 132. For example, the memory 752 may include a computer-readable storage device storing instructions 756 that, when executed by the one or more processors 126, cause the one or more processors 126 to: retrieve from the memory a first block of data and at least a portion of a second block of data, the data corresponding to an array of values arranged along at least a first dimension and a second dimension; perform a first mixed convolution operation that applies a filter on at least a portion of the first block and the second block to generate output data, wherein the output data includes a first accumulated block and at least a portion of the second accumulated block; and store the first accumulated block as the first output data, wherein the portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

[0070] Device 700 can include a wireless controller 738 coupled to one or more antennas 742 via a transceiver 740. In some implementations, one or more antennas 742 include one or more antennas configured to receive data indicative of filter 130, input data 110, or a combination thereof.

[0071] The device 700 may include a display 728 coupled to a display controller 726. The CODEC 734 may be coupled to one or more microphones 746 and one or more speakers 748 and may include a digital-to-analog converter (DAC) 702 and an analog-to-digital converter (ADC) 704. In a particular implementation, the CODEC 734 may receive an analog signal from one or more microphones 746 (e.g., microphone 106), convert the analog signal into a digital signal using the analog-to-digital converter 704, and provide the digital signal for storage as input data 110, such as for speech processing (e.g., convolutional neural network-based speech recognition, speaker recognition, one or more other types of speech processing, or any combination thereof).

[0072] In certain implementations, device 700 includes one or more input devices 730. For example, input device(s) 730 may include one or more cameras (such as camera 104) configured to capture video input for storage as input data 110.

[0073] In a particular implementation, the device 700 may be included in a system-in-package or system-on-chip device 722. In a particular implementation, the memory 752, the processor 706, the processor(s) 126, the display controller 726, the CODEC 734, the wireless controller 738, and the transceiver 740 are included in the system-in-package or system-on-chip device 722. In a particular implementation, an input device 730 and a power supply 744 are coupled to the system-in-package or system-on-chip device 722. Additionally, in a particular implementation, as shown in FIG. Figure 7 As shown, the display 728, input device 730, microphone(s) 746, speaker(s) 748, antenna 742, and power supply 744 are external to the system-in-package or system-on-chip device 722. In a particular implementation, each of the display 728, input device 730, microphone(s) 746, speaker(s) 748, antenna 742, and power supply 744 may be coupled to a component of the system-in-package or system-on-chip device 722, such as an interface or a controller.

[0074] Device 700 may include a voice-activated device, an audio device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a smart speaker, a mobile communication device, a smartphone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a home automation system, a wireless speaker and voice-activated device, a base station, or any combination thereof. In certain aspects, processor 706, processor(s) 126, or a combination thereof are included in an integrated circuit.

[0075] In conjunction with the described implementations, an apparatus for performing a convolution includes means for retrieving a first block of data and at least a portion of a second block of data, the data corresponding to an array of values arranged along at least a first dimension and a second dimension. For example, the means for retrieving the first block may include one or more processors 126, device 502, circuit 516, processor 706, instructions 756 executable by processor(s) 706, 126, one or more other devices, modules, circuits, components, or combinations thereof.

[0076] The apparatus includes means for performing a hybrid convolution operation that applies a filter to at least a portion of a first block and a second block to generate output data, wherein the output data includes at least a portion of the first accumulated block and the second accumulated block. For example, the means for performing the hybrid convolution operation may include one or more processors 126, device 502, circuit 516, processor 706, instructions 756 executable by processor(s) 706, 126, one or more other devices, modules, circuits, components, or a combination thereof.

[0077] The apparatus further includes means for storing the first accumulated block as the first output data, wherein the portion of the second block is adjacent to the first block along the first dimension, and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension. For example, the means for storing may include one or more processors 126, device 502, circuit 516, processor 706, instructions 756 executable by processor(s) 706, 126, one or more other devices, modules, circuits, components, or a combination thereof.

[0078] In some implementations, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to initiate, execute, or control operations to perform some or all of the functions described above. For example, the instructions may be executed to implement Figure 1-Figure 7 In some implementations, Figure 1-Figure 7 Part or all of one or more operations or methods may be implemented by one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs)) that execute instructions, may be implemented by dedicated hardware circuits, or may be implemented by any combination thereof.

[0079] The illustrations of the examples described herein are intended to provide an overall understanding of the structures of various implementations. These illustrations are not intended to serve as a complete description of all elements and features of the apparatus and system utilizing the structures or methods described herein. For those skilled in the art, after reading this disclosure, many other implementations may be apparent. Other implementations can be utilized and derived from this disclosure so that structural and logical replacements and changes can be made without departing from the scope of this disclosure. For example, method operations can be performed in an order different from that shown in the figures, or one or more method operations can be omitted. Accordingly, this disclosure and the accompanying drawings should be considered illustrative and not restrictive.

[0080] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above in general terms according to their functions. Whether such functions are implemented as hardware or as processor-executable instructions depends on the specific application and the design constraints imposed on the entire system. A skilled person can implement the described functions in different ways for each specific application, and such implementation decisions should not be interpreted as resulting in a departure from the scope of this disclosure.

[0081] The steps of the method or algorithm described in conjunction with the implementation disclosed herein can be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium can be integrated into the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a computing device or a user terminal. In an alternative, the processor and the storage medium can reside in a computing device or a user terminal as separate components.

[0082] The foregoing description of the disclosed implementations is provided to enable those skilled in the art to make or use the disclosed implementations. Various modifications to these implementations will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the scope of this disclosure. Submitting the abstract of this disclosure is based on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing specific embodiments, various features may be combined together or described in a single implementation for the purpose of simplifying this disclosure. The examples described above illustrate the disclosure, but do not limit it. It should also be understood that many modifications and variations are possible according to the principles of this disclosure. Therefore, this disclosure is not intended to be limited to the implementations shown herein, but is intended to be consistent with the maximum possible scope consistent with the principles and novel features defined by the appended claims.

Claims

1. A device comprising: Memory, configured to store: data corresponding to an array of input values arranged along at least a first dimension and a second dimension, and output data corresponding to an array of output values arranged along at least the first dimension and the second dimension; as well as One or more processors configured to: retrieving from the memory the first block of data and at least a portion of the second block of data; performing a first hybrid convolution operation configured to apply a filter on the first block and on at least a portion of the second block to generate a first accumulated block and at least a portion of a second accumulated block; as well as The first accumulated block is stored as first output data, wherein: The portion of the second block is adjacent to the first block along the first dimension of the array of input values, and The portion of the second accumulation block is adjacent to the first accumulation block along the second dimension of the array of output values.

2. The apparatus of claim 1 , wherein a size of each of the first block, the second block, the first accumulation block, and the second accumulation block is determined according to a memory access data size supported by the memory for memory access by the one or more processors.

3. The apparatus of claim 2 , wherein the first hybrid convolution operation is configured to use zero padding along the second dimension without using zero padding along the first dimension to enable convolution using at most two memory accesses to retrieve the portions of the first block and the second block and using at most two memory accesses to store the portions of the first accumulated block and the second accumulated block.

4. The apparatus according to claim 1, wherein: The first block and the portion of the second block together correspond to N rows and the first M columns of the array of input values, N and M being positive integers, and N being greater than M; The first accumulation block and the second accumulation block together correspond to M rows and N columns of output data; The first accumulation block corresponds to M rows and M columns of accumulated values in the first accumulator; and The portion of the second accumulation block corresponds to M rows and NM columns of local accumulated values in the second accumulator.

5. The apparatus of claim 4 , wherein the one or more processors are further configured to access the first block using a first memory access operation and to access the portion of the second block using a second memory access operation, and wherein each of the first memory access operation and the second memory access operation is configured to retrieve at most M rows and M columns of the data.

6. The apparatus of claim 4, wherein the filter comprises P rows and P columns of filter values, and wherein P = N - M + 1.

7. The apparatus of claim 4, wherein the one or more processors are further configured to: retrieving at least a portion of a third block and a fourth block of said data that together correspond to N rows and a second M columns of said array of input values; performing a second hybrid convolution operation on the third block and the portion of the fourth block to update the second accumulation block and generate at least a portion of a third accumulation block, the portion of the third accumulation block being adjacent to the second accumulation block along the second dimension of the array of output values; as well as The second accumulated block is stored as second output data.

8. The apparatus of claim 7 , wherein the one or more processors are further configured to execute a sequence of mixed convolution operations comprising the first mixed convolution operation and the second mixed convolution operation in a pipelined manner, wherein storage of the first accumulation block and execution of the second mixed convolution operation occur during the same pipeline cycle.

9. The apparatus of claim 7, wherein the one or more processors are further configured to: generating the first accumulation block in the first accumulator; generating the second accumulation block in the second accumulator; and After storing the first accumulation block, the third accumulation block is generated in the first accumulator.

10. The device of claim 1, wherein the one or more processors are configured to convolve the data with the filter according to an image processing operation.

11. The apparatus of claim 1 , wherein the one or more processors are configured to convolve the data with the filter according to a neural network convolutional layer operation.

12. The apparatus of claim 11, wherein the memory and the one or more processors are integrated into a vehicle.

13. The device of claim 11, wherein the memory and the one or more processors are integrated into a home automation system.

14. The device of claim 1, wherein the memory and the one or more processors are integrated into a mobile device.

15. The apparatus of claim 1, wherein the one or more processors are included in an integrated circuit.

16. A method of performing convolution, comprising: retrieving, at one or more processors, from a memory a first block of data and at least a portion of a second block of data, the data corresponding to an array of input values arranged along at least a first dimension and a second dimension; performing a first hybrid convolution operation that applies a filter on the first block and on at least a portion of the second block to generate a first accumulated block and at least a portion of a second accumulated block; as well as The first accumulated block is stored as first output data, wherein: The portion of the second block is adjacent to the first block along the first dimension of the array of input values, and The portion of the second accumulation block is adjacent to the first accumulation block along the second dimension of an array of output values, wherein the array of output values is arranged along at least the first dimension and the second dimension.

17. The method of claim 16 , wherein a size of each of the first block, the second block, the first accumulation block, and the second accumulation block is determined according to a memory access data size supported by the memory for memory access by the one or more processors; and The memory: is different from the one or more processors; or It is dynamic random access memory (DRAM).

18. The method of claim 16, wherein: The first hybrid convolution operation uses zero padding along the second dimension and no zero padding along the first dimension to enable convolution using at most two memory accesses to retrieve the portions of the first block and the second block and using at most two memory accesses to store the portions of the first accumulated block and the second accumulated block; The array of input values is divided into a plurality of blocks logically arranged along at least the first dimension and the second dimension, the plurality of blocks including the first block and the second block adjacent to the first block along the first dimension; and The array of output values is divided into a plurality of accumulation blocks logically arranged along at least the first dimension and the second dimension, the plurality of accumulation blocks including the first accumulation block and the second accumulation block adjacent to the first accumulation block along the second dimension.

19. The method of claim 16 , wherein retrieving the first block comprises using a first memory access operation, wherein retrieving the portion of the second block comprises using a second memory access operation, wherein each of the first memory access operation and the second memory access operation is configured to extract at most M rows and M columns of the data, where M is a positive integer, and wherein each of the first accumulation block and the second accumulation block corresponds to M rows and M columns of output data.

20. The method according to claim 19, further comprising: retrieving a third block of the data adjacent to the first block along the second dimension of the array of input values; retrieving at least a portion of a fourth block of the data adjacent to the portion of the second block along the second dimension of the array of input values and adjacent to the third block along the first dimension of the array of input values; performing a second hybrid convolution operation on the third block and the portion of the fourth block to update the second accumulation block and generate at least a portion of a third accumulation block, the portion of the third accumulation block being adjacent to the second accumulation block along the second dimension of the array of output values; as well as The second accumulated block is stored as second output data.

21. The method of claim 20, wherein the first mixed convolution operation and the second mixed convolution operation are part of a sequence of mixed convolution operations performed in a pipelined manner, wherein storage of the first accumulation block and execution of the second mixed convolution operation occur during the same pipeline cycle.

22. The method according to claim 20, further comprising: generating the first accumulation block in a first accumulator; generating the second accumulation block in a second accumulator; as well as After storing the first accumulation block, the third accumulation block is generated in the first accumulator.

23. The method of claim 16, wherein the one or more processors are configured to convolve the data with the filter according to at least one of an image processing operation or a neural network convolutional layer operation.

24. A computer-readable storage device storing instructions that, when executed by one or more processors, cause the one or more processors to: retrieving from a memory a first block of data and at least a portion of a second block of data, the data corresponding to an array of input values arranged along at least a first dimension and a second dimension; performing a first hybrid convolution operation that applies a filter on the first block and on at least a portion of the second block to generate a first accumulated block and at least a portion of a second accumulated block; as well as The first accumulated block is stored as first output data, wherein: The portion of the second block is adjacent to the first block along the first dimension of the array of input values, and The portion of the second accumulation block is adjacent to the first accumulation block along the second dimension of an array of output values, wherein the array of output values is arranged along at least the first dimension and the second dimension.

25. The computer-readable storage device of claim 24, wherein a size of each of the first block, the second block, the first accumulation block, and the second accumulation block is determined according to a memory access data size supported by the memory for memory access by the one or more processors.

26. The computer-readable storage device of claim 24, wherein each of the first block, the second block, the first accumulation block, and the second accumulation block have the same size.

27. The computer-readable storage device of claim 24, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: retrieving a third block of the data adjacent to the first block along the second dimension of the array of input values; retrieving at least a portion of a fourth block of the data adjacent to the portion of the second block along the second dimension of the array of input values and adjacent to the third block along the first dimension of the array of input values; performing a second hybrid convolution operation on the portions of the third block and the fourth block to update the second accumulation block and generate at least a portion of a third accumulation block, the portion of the third accumulation block being adjacent to the second accumulation block along the second dimension of the array of output values; as well as The second accumulated block is stored as second output data.

28. An apparatus comprising: means for retrieving a first block of data and at least a portion of a second block of data, the data corresponding to an array of input values arranged along at least a first dimension and a second dimension; means for performing a hybrid convolution operation that applies a filter on the first block and on at least a portion of the second block to generate a first accumulated block and at least a portion of a second accumulated block; as well as means for storing the first accumulated block as first output data, wherein: The portion of the second block is adjacent to the first block along the first dimension of the array of input values, and The portion of the second accumulation block is adjacent to the first accumulation block along the second dimension of an array of output values, wherein the array of output values is arranged along at least the first dimension and the second dimension.

29. The apparatus of claim 28, wherein the means for retrieving, the means for performing the hybrid convolution operation, and the means for storing are integrated into at least one of the following, or a combination thereof: a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, a mobile device.

30. The apparatus of claim 28, wherein the means for performing the hybrid convolution operation is configured to convolve the data with the filter according to at least one of an image processing operation or a neural network convolutional layer operation.

Images